Adaptive resolution changes and scalable codecs for screen content

Abstract

Adaptive resolution change and scalable codecs for screen content are described. One example is a method for video processing, comprising: determining, for conversion between a picture or video and a bitstream representation of the picture or video, whether the picture or video has a native resolution based on an indication included in the bitstream representation; and performing the conversion based on the determination.

Description

[0001] Related cross-references

[0002] This application claims priority to and the benefits of International Patent Application No. PCT / CN2019 / 106984, filed on September 20, 2019, in accordance with the applicable provisions of the Patent Law and / or the Paris Convention. The entire disclosure of International Patent Application No. PCT / CN2019 / 106984 is incorporated herein by reference and made a part of the disclosure of this application. Technical Field

[0003] This patent document relates to video and image encoding and decoding. Background Art

[0004] Despite advances in video compression, digital video still represents the largest use of bandwidth on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, bandwidth demand for digital video usage is expected to continue to grow. Summary of the Invention

[0005] Apparatus, systems and methods related to digital video coding and decoding, and more particularly, video and image coding and decoding in which scalable video coding and decoding are implemented, are disclosed.

[0006] In one exemplary aspect, a video processing method is disclosed that includes performing a conversion between a video having a native resolution and a codec representation of the video having a codec resolution, wherein a field in the codec representation indicates a relationship between the native resolution and the codec resolution.

[0007] In another exemplary aspect, a method of video processing is disclosed. The method includes: for converting between a video segment of a video and a codec representation of the video, determining suitability of a screen content codec for converting the video segment based on usage rules associated with codec mode usage for blocks in the video segment; and performing the conversion based on the determination.

[0008] In another exemplary aspect, a method of video processing is disclosed, comprising: determining, for a conversion between video units of a video and a codec representation of the video using a screen content codec, a characteristic of the conversion caused by a reference picture and a current picture including video units of different sizes; and performing the conversion based on the determination.

[0009] In another exemplary aspect, a method of video processing is disclosed, comprising: determining a constraint for converting between a video unit of a video and a scalable codec representation of the video; and performing the conversion according to the constraint; wherein the scalable codec representation includes a first layer and a second layer, wherein the first layer has a lower quality than the second layer.

[0010] In another exemplary aspect, a method of video processing is disclosed. The method includes: for converting between a video and a codec representation of the video using a lossless codec mode, determining that the codec representation omits syntax elements related to signaling of a quantization parameter or a codec block flag; and performing the conversion based on the determination.

[0011] In another exemplary aspect, a method of video processing is disclosed. The method includes: for converting between a picture or video and a bitstream representation of the picture or video, determining whether the picture or video has a native resolution based on an indication included in the bitstream representation; and performing the conversion based on the determination.

[0012] In another exemplary aspect, a method of video processing is disclosed, comprising: determining, for conversion between a picture or video and a bitstream representation of the picture or video, whether the picture or video is screen content based on the use of one or more codecs or signaling messages; and performing the conversion based on the determination.

[0013] In another exemplary aspect, a method of video processing is disclosed. The method includes: determining one or more constraints associated with a scalable video codec for converting a picture or video to a scalable codec bitstream representation of the picture or video; and performing the conversion based on the determination.

[0014] In another exemplary aspect, a method of video processing is disclosed, comprising: determining, for conversion between a picture or video and a bitstream representation of the picture or video, to skip one or more syntax elements when the conversion uses a lossless codec mode; and performing the conversion based on the determination.

[0015] In yet another representative aspect, the above method is embodied in the form of processor-executable code and stored in a computer-readable program medium.

[0016] In yet another representative aspect, a device configured or operable to perform the above method is disclosed. The device may include a processor programmed to implement the method.

[0017] In yet another representative aspect, a video decoder device may implement the methods described herein.

[0018] In yet another representative aspect, a computer program product stored on a non-transitory computer-readable medium including program code can implement the methods described herein.

[0019] The above and other aspects and features of the disclosed technology are described in more detail in the drawings, the description, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 An example of the current CTU processing order and its available reference samples in the current and left CTUs is shown.

[0021] Figure 2 An example of the residual coding process for transform skip blocks is shown.

[0022] Figure 3 An example of a block encoded and decoded in palette mode is shown.

[0023] Figure 4 Examples of horizontal and vertical traversal scans are shown.

[0024] Figure 5 is a block diagram of an example video processing system in which the disclosed technology may be implemented.

[0025] Figure 6 is a block diagram of an example video processing device.

[0026] Figure 7 is a flow chart of an example method of video processing.

[0027] Figure 8 is a flow chart of an example method of video processing.

[0028] Figure 9 is a flow chart of an example method of video processing.

[0029] Figure 10 is a flow chart of an example method of video processing.

[0030] Figure 11 is a flow chart of an example method of video processing. DETAILED DESCRIPTION

[0031] Embodiments of the disclosed technology can be applied to existing video codec standards (e.g., HEVC, H.265) and future standards to improve compression performance. In this document, section headings are used to improve the readability of the description and do not in any way limit the discussion or embodiments (and / or implementation methods) to the corresponding section.

[0032] 1. Overview

[0033] This document relates to video codec technologies. Specifically, it relates to various techniques for reference picture resampling (or adaptive resolution change) and scalable video codecs for encoding and decoding screen content. It can be applied to existing video codec standards such as HEVC, or to a pending standard (Universal Video Codec). It can also be applied to future video codec standards or video codecs.

[0034] 2. Preliminary Discussion

[0035] Video codec standards have evolved primarily through the development of the well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, ISO / IEC developed MPEG-1 and MPEG-4 Vision, and the two organizations jointly developed H.262 / MPEG-2 Video, H.264 / MPEG-4 Advanced Video Codec (AVC), and H.265 / HEVC standards [1]. Since H.262, video codec standards have been based on a hybrid video codec architecture that uses temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, VCEG and MPEG jointly established the Joint Video Exploration Team (JVET) in 2015. Since then, JVET has adopted many new methods and applied them to reference software called the Joint Exploration Model (JEM). In April 2018, the Joint Video Experts Team (JVET) was created between VCEG (Q6 / 16) and ISO / IEC JTC1SC29 / WG11 (MPEG) to work on the VVC standard with the goal of achieving a 50% bit rate reduction compared to HEVC.

[0036] 2.1 Screen Content Encoding and Decoding Tools

[0037] 2.1.1 Intra-block Copy (IBC)

[0038] Intra-block copying (IBC) is a tool adopted by the HEVC extension for SCC. It is well known that it significantly improves the encoding and decoding efficiency of screen content materials. Since the IBC mode is implemented as a block-level encoding and decoding mode, block matching (BM) is performed at the encoder to find the optimal block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to the reference block that has been reconstructed inside the current picture. The luminance block vector of the CU encoded and decoded by IBC has integer precision. The chrominance block vector is also rounded to integer precision. When combined with AMVR, the IBC mode can switch between 1-pixel and 4-pixel motion vector accuracy. The CU encoded and decoded by IBC is treated as a third prediction mode in addition to the intra or inter prediction mode. The IBC mode is applicable to CUs whose width and height are both less than or equal to 64 luminance samples.

[0039] On the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD checks on blocks with a width or height of no more than 16 luma samples. For non-merge mode, a block vector search is first performed using a hash-based search. If the hash search does not return a valid candidate, a block match based on a local search is performed.

[0040] In a hash-based search, hash key matching (32-bit CRC) between the current block and reference blocks is extended to all allowed block sizes. The hash key calculation for each position in the current picture is based on a 4×4 sub-block. For larger current block sizes, a hash key is determined to match the hash key of a reference block when all hash keys in all 4×4 sub-blocks match the hash key of the corresponding reference position. If multiple reference blocks are found whose hash keys match the hash key of the current block, the block vector cost of each matching reference is calculated and the one with the lowest cost is selected.

[0041] In the block matching search, the search range is set to cover both the previous CTU and the current CTU.

[0042] At the CU level, the IBC mode is signaled using a flag, which can be signaled as IBC AMVP mode or IBC skip / merge mode as follows:

[0043] -IBC skip / merge mode: Use the merge candidate index to indicate which block vector from the adjacent candidate IBC codec block in the list to use to predict the current block. The merge list consists of spatial candidates, HMVP candidates, and pairwise candidates.

[0044] – IBC AMVP mode: Block vector differences are encoded and decoded in the same way as motion vector differences. This block vector prediction method uses two candidates as predictors, one from the left neighbor and one from the upper neighbor (if encoded in IBC). If either neighbor is unavailable, a default block vector is used as the predictor. A flag is signaled to indicate the block vector predictor index.

[0045] 2.1.1.1 IBC Reference Area

[0046] To reduce memory consumption and decoder complexity, IBC in VTM6 only allows the reconstruction of a predefined area including the area of ​​the current CTU and a certain area of ​​the left CTU. Figure 1 The reference area of ​​the IBC mode is shown, where each block represents a 64x64 luma sample unit.

[0047] Depending on the position of the current codec CU within the current CTU, the following applies:

[0048] – If the current block falls within the upper left 64x64 block of the current CTU, in addition to referencing the reconstructed samples in the current CTU, it can also use the CPR mode to reference the reference samples in the lower right 64x64 block of the left CTU. The current block can also use the CPR mode to reference the reference samples in the lower left 64x64 block of the left CTU and the reference samples in the upper right 64x64 block of the left CTU.

[0049] – If the current block falls within the upper right 64x64 block of the current CTU, in addition to referring to the reconstructed samples in the current CTU, if the luma position (0, 64) relative to the current CTU has not been reconstructed, the current block can also use the CPR mode to refer to the reference samples in the lower left 64x64 block and the lower right 64x64 block of the left CTU; otherwise, the current block can also refer to the reference samples in the lower right 64x64 block of the left CTU.

[0050] – If the current block falls within the lower left 0x64 block of the current CTU, then in addition to referencing the reconstructed samples in the current CTU, if the luma position (64,0) relative to the current CTU has not been reconstructed, the current block can also use the CPR mode to reference the reference samples in the upper right 64x64 block and the lower right 64x64 block of the left CTU. Otherwise, the current block can also use the CPR mode to reference the reference samples in the lower right 64x64 block of the left CTU.

[0051] If the current block falls within the lower right 64x64 block of the current CTU, it can use CPR mode to refer only to the reconstructed samples in the current CTU.

[0052] This restriction allows the IBC mode to be implemented using local on-chip memory for hardware implementation.

[0053] 2.1.1.2 Interaction between IBC and other codecs

[0054] The interaction between IBC mode and other inter-frame codec tools in VTM6 (such as pairwise merge candidates, history-based motion vector predictor (HMVP), inter / intra joint prediction mode (CIIP), Merge mode with motion vector difference (MMVD) and triangulation) is as follows:

[0055] – You can use IBC with pairwise merge candidates and HMVP. You can generate a new pairwise IBC merge candidate by averaging two IBC merge candidates. For HMVP,

[0056] Inserts IBC movements into the history buffer for future reference.

[0057] – IBC cannot be used in conjunction with the following interframe tools: Affine Motion, CIIP, MMVD, and Triangulation.

[0058] – When using DUAL_TREE partitioning, IBC is not allowed for chroma codec blocks.

[0059] Unlike the HEVC Screen Content Codec extension, the current picture is no longer included as one of the reference pictures in reference picture list 0 for IBC prediction. The derivation of motion vectors in IBC mode excludes all neighboring blocks in inter mode, and vice versa. The following IBC design aspects apply:

[0060] – IBC shares the same process as in regular MV merge, including adopting pairwise merge candidates and history-based motion predictor, but TMVP and zero vectors are not allowed since they are invalid for IBC mode.

[0061] - Use separate HMVP buffers (5 candidates each) for conventional MV and IBC.

[0062] – Block vector constraints are implemented as bitstream consistency constraints. The encoder must ensure that there are no invalid vectors in the bitstream and must not use merges if the merge candidate is invalid (out of range or zero). Such bitstream consistency constraints are expressed in conjunction with the virtual buffer as described below.

[0063] – For deblocking, IBC is treated as an inter mode.

[0064] If the current block is coded using IBC prediction mode, AMVR does not use quarter pixels; instead, AMVR is signaled to only indicate whether the MV is an integer pixel or 4 integer pixels.

[0065] – The number of IBC merge candidates may be signaled in the slice header independently from the number of regular merge candidates, sub-block merge candidates, and delta merge candidates.

[0066] A virtual buffer concept is used to describe the permissible reference area for IBC prediction modes and valid block vectors. Denoting the CTU size as ctbSize, the virtual buffer ibcBuf has a width of wIbcBuf = 128*128 / ctbSize and a height of hIbcBuf = ctbSize. For example, for a CTU size of 128x128, the size of ibcBuf is also 128x128; for a CTU size of 64x64, the size of ibcBuf is 256x64; and for a CTU size of 32x32, the size of ibcBuf is 512x32.

[0067] The size of the VPDU in each dimension is min(ctbSize, 64), Wv = min(ctbSize, 64).

[0068] The virtual IBC buffer ibcBuf is maintained as follows:

[0069] 1) When starting to decode each CTU row, flush the entire ibcBuf with an invalid value of -1.

[0070] 2) When starting to decode the VPDU (xVPDU, yVPDU) relative to the upper left corner of the picture, set ibcBuf[x][y] = -1, where x = xVPDU% wIbcBuf, ..., xVPDU% wIbcBuf + Wv–1; y = yVPDU% ctbSize, ..., yVPDU% ctbSize + Wv–1.

[0071] 3) After decoding the CU containing (x, y) relative to the upper left corner of the picture, set

[0072] ibcBuf[x%wIbcBuf][y%ctbSize]=recSample[x][y]

[0073] For a block covering coordinates (x,y), it is valid if the following is true for the block vector bv = (bv[0], bv[1]); otherwise, it is invalid:

[0074] ibcBuf[(x+bv[0])%wIbcBuf][(y+bv[1])%ctbSize] must not be equal to -1.

[0075] 2.1.2 Block Differential Pulse Coded Modulation (BDPCM)

[0076] VTM6 supports Block Differential Pulse Coded Modulation (BDPCM) for screen content coding and decoding. At the sequence level, the BDPCM enable flag is signaled in the SPS; this flag is signaled in the SPS only when the transform skip mode (described in the next section) is enabled.

[0077] When BDPCM is enabled, if the CU size in terms of luma samples is less than or equal to MaxTsSize multiplied by MaxTsSize, and if the CU is intra-coded, a flag is transmitted at the CU level, where MaxTsSize is the maximum block size allowed for transform skip mode. This flag indicates whether conventional intra-coding or BDPCM is used. If BDPCM is used, a BDPCM prediction direction flag is transmitted to indicate whether the prediction is horizontal or vertical. The block is then predicted using the conventional horizontal or vertical intra prediction process with the aid of unfiltered reference samples. The residuals are quantized and the difference between each quantized residual and its predictor is encoded, where the predictor refers to the previously encoded residual at the adjacent horizontal or vertical position (depending on the BDPCM prediction direction).

[0078] For a block with size M (height) × N (width), let r i,j ,0≤i≤M-1,0≤j≤N-1 is the prediction residual. Let Q(r i,j ), 0≤i≤M-1,0≤j≤N-1 represents the residual r i,j Applying BDPCM to the quantized residual value yields a quantized version of The modified M×N array in, is predicted by its adjacent quantized residual values. For the vertical BDPCM prediction mode, for 0≤j≤(N-1), the following formula is used to derive

[0079]

[0080] For the horizontal BDPCM prediction mode, for 0≤i≤(M-1), the following formula is used to derive

[0081]

[0082] On the decoder side, reverse the above process to calculate Q(r i,j ), 0≤i≤M-1,0≤j≤N-1, as follows:

[0083] If using vertical BDPCM,

[0084] If using horizontal BDPCM,

[0085] Dequantize the residual Q -1 (Q(r i,j )) is added to the intra block prediction value to generate the reconstructed sample value.

[0086] The predicted quantized residual values ​​are converted to Sent to the decoder. For future intra-mode codecs, if the BDPCM prediction direction is horizontal or vertical, store the horizontal or vertical prediction mode for the BDPCM-coded CU, respectively. For deblocking, if both blocks on either side of a block boundary are coded using BDPCM, then do not deblock that particular block boundary.

[0087] 2.1.3 Residual Codec for Transform Skip Mode

[0088] VTM6 allows the use of transform skip mode for luminance blocks of size up to MaxTsSize multiplied by MaxTsSize, where the value of MaxTsSize is signaled in the PPS and is limited to a maximum of 32 in VTM6. When a CU is encoded or decoded in transform skip mode, its prediction residual is quantized and encoded using the transform skip residual encoding and decoding process. This process is modified from the transform coefficient encoding and decoding process. In transform skip mode, the residual of the TU is also encoded and decoded in units of non-overlapping sub-blocks of size 4x4. In order to obtain higher encoding and decoding efficiency, some modifications are made to the characteristics of the residual signal to customize the residual encoding and decoding process. The following summarizes the differences between transform skip residual encoding and decoding in VTM6 and conventional transform residual encoding and decoding:

[0089] – Apply forward scan order to scan sub-blocks within a transform block and positions within sub-blocks;

[0090] – no signaling of the final (x,y) position;

[0091] – encode and decode coded_sub_block_flag for each sub-block except the last sub-block when all previous flags are equal to 0;

[0092] –sig_coeff_flag context modeling uses a reduced template, and the context model of sig_coeff_flag depends on the neighboring values ​​on the top and left;

[0093] – The context model of the abs_level_gt1 flag also depends on the left and top sig_coeff_flag context modeling uses a reduced template, and the context model of sig_coeff_flag depends on the top and left neighboring values;

[0094] -value.

[0095] –par_level_flag uses only one context model;

[0096] – Signaling additional greater than 3, 5, 7, 9 flags to indicate coefficient magnitudes, one context per flag;

[0097] – derivation of modified rice parameters for binarization of remainder values;

[0098] – Determine the context model for the sign flag based on the left and above neighboring values, and parse the sign flag after sig_coeff_flag so that all context codec bins are kept together;

[0099] For each subblock, if coded_subblock_flag is equal to 1 (ie, there is at least one non-zero quantized residual in the subblock), then the encoding and decoding of the quantized residual magnitude is performed in three passes (see Figure 2 ).

[0100] - First scan pass: Encode and decode the validity flag (sig_coeff_flag), the sign flag (coeff_sign_flag), the absolute magnitude greater than 1 flag (abs_level_gtx_flag[0]), and the parity (par_level_flag). For a given scan position, if sig_coeff_flag is 1, then encode and decode coeff_sign_flag, followed by abs_level_gtx_flag[0] (which specifies whether the absolute magnitude is greater than 1). If abs_level_gtx_flag[0] is 1, then additionally encode and decode par_level_flag to specify the parity of the absolute magnitude.

[0101] - Greater than x scan passes: For each scan position where the absolute magnitude is greater than 1, up to four abs_level_gtx_flag[i] (for i=1...4) are encoded to indicate whether the absolute magnitude at the given position is greater than 3, 5, 7 or 9, respectively.

[0102] - Residue scan pass: For all scan positions where abs_level_gtx_flag[4] is equal to 1 (i.e., the absolute amplitude is greater than 9), the absolute amplitude remainder is encoded and decoded. The absolute amplitude remainder is binarized using the reduced rice parameter derivation template.

[0103] Context coding is performed on the bits in passes #1 and #2 (the first pass and x+ passes) until the maximum number of context coding bits in the TU is exhausted. The maximum number of context coding bits in the residual block is limited to 2*block_width*block_height, or equivalently, an average of 2 context coding bits per sample position. The bits in the last pass (the remainder pass) are bypassed for coding.

[0104] Figure 2 An example of a residual coding pass for a transform skip block is shown.

[0105] In addition, for blocks coded in BDPCM mode, an amplitude mapping mechanism is applied to the transform skip residual codec. Amplitude mapping uses the top and left neighboring coefficient amplitudes to predict the current coefficient amplitude, thereby reducing the signaling cost. For a given residual position, denote absCoeff as the absolute coefficient amplitude before mapping and absCoeffMod as the coefficient amplitude after mapping. Let X0 denote the absolute coefficient amplitude of the left neighboring position, and let X1 denote the absolute coefficient amplitude of the top neighboring position. Amplitude mapping is performed as follows:

[0106]

[0107] Afterwards, the absCoeffMod value is encoded and decoded as described above.

[0108] 2.1.4 Palette Mode

[0109] VTM6 supports palette mode for screen content encoding with 4:4:4 color format. When palette mode is enabled, a flag indicating whether palette mode is used is transmitted at the CU level if the CU size is less than or equal to 64x64. The palette codec unit (CU) is treated as a prediction mode in addition to intra prediction, inter prediction, and intra block copy (IBC) modes.

[0110] If palette mode is used, the sample values ​​in the CU are represented by a small set of representative color values. This set is called the palette. For pixels with values ​​close to the palette colors, the palette index is signaled. It is also possible to specify samples outside the palette by signaling an escape symbol and the subsequent quantized component value. Figure 3 This situation is exemplified in .

[0111] For palette codecs, a palette predictor is maintained. The predictor is initialized to 0 at the beginning of each bar for the non-wavefront case and at the beginning of each CTU row for the wavefront case. For each entry in the palette predictor, a reuse flag is signaled to indicate whether it is part of the current palette in the CU. The reuse signaling is sent using a run length codec of zero. Thereafter, the number of new palette entries and the component values ​​of the new palette entries are signaled. After encoding a palette codec CU, the palette predictor is updated with the current palette and entries from the previous palette predictor that are not reused in the current palette are added to the end of the new palette predictor until the maximum size allowed is reached. An escape flag is signaled for each CU, which indicates whether there is an escape symbol in the current CU. If there is an escape symbol, the palette table is incremented by one and the last index is assigned to the escape symbol.

[0112] The palette index of the samples in the CU forms a palette index map. Figure 4 The index map is encoded and decoded using the horizontal and vertical traversal scans shown in Figure 1. The scanning order is explicitly signaled in the bitstream using palette_transpose_flag.

[0113] The palette index is encoded and decoded using two main palette sampling modes: "INDEX" and "COPY_ABOVE". The mode is signaled using a flag except for the top row when horizontal scanning is used, and except for the first column when vertical scanning is used or when the previous mode was "COPY_ABOVE". In "COPY_ABOVE" mode, the palette index of the samples in the row above is copied. In "INDEX" mode, the palette index is signaled explicitly. For both "INDEX" and "COPY_ABOVE" modes, a run value is signaled that specifies the number of pixels encoded and decoded using the same pattern.

[0114] The encoding order for the index map codec is as follows: First, the number of index values ​​associated with the "INDEX" run is signaled. This is followed by signaling the actual index values ​​associated with the "INDEX" run for the entire CU using the truncated binary codec. After that, the palette mode (INDEX or COPY_ABOVE) and the run length of each run are signaled in an interleaved manner. Finally, the quantized escape pattern colors for the entire CU are gathered together and encoded and decoded using the exponential Golomb codec.

[0115] For stripes using dual luma / chroma trees, the palette is applied separately to luma (Y component) and chroma (Cb and Cr components). For stripes with a single tree, the palette is applied jointly to the Y, Cb, and Cr components, i.e., each entry in the palette contains a Y, Cb, and Cr value.

[0116] For deblocking, palette-coded blocks on the sides of block boundaries are not deblocked.

[0117] 2.2 Reference Image Resampling in VVC

[0118] VVC draft 6 adopts the reference picture resampling (or adaptive resolution change) design. It allows the reference picture and the current picture to have different resolutions. It includes the following aspects:

[0119] Maximum picture size signaling in SPS,

[0120] Moved to PPS actual picture size and consistency window signaling,

[0121] Define the zoom ratio based on the output image size after cropping,

[0122] Resampling is done on a block basis using existing interpolation filters,

[0123] Use TMVP only when the current picture and the co-located picture have the same size,

[0124] • DMVR and BDOF are used only when the current picture and the reference picture (selected for inter prediction in the CU) have the same size.

[0125] 2.2.1 Support for advanced syntax of reference image resampling

[0126] In VVC draft 6, within a sequence, different picture parameter sets can contain parameters defining different picture sizes and a consistency window indicating the output area, as follows:

[0127] pic_width_in_luma_samples specifies the width of each decoded picture that references this PPS in units of luma samples. pic_width_in_luma_samples must not be equal to 0, must be an integer multiple of Max(8,MinCbSizeY), and must be less than or equal to pic_width_max_in_luma_samples.

[0128] When subpics_present_flag is equal to 1, the value of pic_width_in_luma_samples must be equal to pic_width_max_in_luma_samples.

[0129] pic_height_in_luma_samples specifies the height of each decoded picture that references this PPS in units of luma samples. pic_height_in_luma_samples must not be equal to 0, must be an integer multiple of Max(8,MinCbSizeY), and must be less than or equal to pic_height_max_in_luma_samples.

[0130] When subpics_present_flag is equal to 1, the value of pic_height_in_luma_samples must be equal to pic_height_max_in_luma_samples.

[0131] Let refPicWidthInLumaSamples and refPicHeightInLumaSamples be the pic_width_in_luma_samples and pic_height_in_luma_samples, respectively, of the reference picture of the current picture that refers to this PPS. Bitstream conformance requirements are that all of the following conditions are met:

[0132] –pic_width_in_luma_samples*2 must be greater than or equal to

[0133] refPicWidthInLumaSamples.

[0134] –pic_height_in_luma_samples*2 must be greater than or equal to

[0135] refPicHeightInLumaSamples.

[0136] –pic_width_in_luma_samples must be less than or equal to

[0137] refPicWidthInLumaSamples*8.

[0138] –pic_height_in_luma_samples must be less than or equal to

[0139] refPicHeightInLumaSamples*8.

[0140] Derive the following variables: PicWidthInCtbsY, PicHeightInCtbsY, PicSizeInCtbsY, PicWidthInMinCbsY, PicHeightInMinCbsY, PicSizeInMinCbsY, PicSizeInSamplesY, PicWidthInSamplesC, and PicHeightInSamplesC as follows:

[0141] PicWidthInCtbsY = Ceil( pic_width_in_luma_samples ÷ CtbSizeY ) (7-34)

[0142] PicHeightInCtbsY = Ceil( pic_height_in_luma_samples ÷ CtbSizeY )(7-35)

[0143] PicSizeInCtbsY = PicWidthInCtbsY * PicHeightInCtbsY (7-36)

[0144] PicWidthInMinCbsY = pic_width_in_luma_samples / MinCbSizeY (7-37)

[0145] PicHeightInMinCbsY = pic_height_in_luma_samples / MinCbSizeY (7-38)

[0146] PicSizeInMinCbsY = PicWidthInMinCbsY * PicHeightInMinCbsY (7-39)

[0147] PicSizeInSamplesY =

[0148] pic_width_in_luma_samples*pic_height_in_luma_samples(7-40)

[0149] PicWidthInSamplesC = pic_width_in_luma_samples / SubWidthC (7-41)

[0150] PicHeightInSamplesC = pic_height_in_luma_samples / SubHeightC (7-42)

[0151] conformance_window_flag equal to 1 indicates that the conformance cropping window offset parameter is the next one in the SPS. conformance_window_flag equal to 0 indicates that there is no conformance cropping window offset parameter.

[0152] conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset specify the samples of the picture in the CVS output from the decoding process according to the rectangular area for output indicated by the picture coordinates. When conformance_window_flag is equal to 0, the values ​​of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset are inferred to be equal to 0.

[0153] The consistent cropping window contains luma samples with horizontal picture coordinates from SubWidthC*conf_win_left_offset to pic_width_in_luma_samples-(SubWidthC*conf_win_right_offset+1) (inclusive) and vertical picture coordinates from SubHeightC*conf_win_top_offset to pic_height_in_luma_samples-(SubHeightC*conf_win_bottom_offset+1) (inclusive).

[0154] The value of SubWidthC*(conf_win_left_offset+conf_win_right_offset) must be less than pic_width_in_luma_samples, and the value of SubHeightC*(conf_win_top_offset+conf_win_bottom_offset) must be less than pic_height_in_luma_samples.

[0155] The variables PicOutputWidthL and PicOutputHeightL are derived as follows:

[0156] PicOutputWidthL=pic_width_in_luma_samples-SubWidthC*

[0157] (conf_win_right_offset+conf_win_left_offset)(7-43)

[0158] PicOutputHeightL=pic_height_in_pic_size_units-SubHeightC*

[0159] (conf_win_bottom_offset+conf_win_top_offset)(7-44)

[0160] When ChromaArrayType is not equal to 0, the corresponding specified samples of the two chroma arrays are samples with picture coordinates (x / SubWidthC, y / SubHeightC), where (x, y) are the picture coordinates of the specified luma sample.

[0161] NOTE – The consistent crop window offset parameters are only applied on output. All internal decoding processes are applied to the uncropped picture size.

[0162] Let ppsA and ppsB be any two PPSs that reference the same SPS. The bitstream conformance requirement is that when ppsA and ppsB have the same values ​​of pic_width_in_luma_samples and pic_height_in_luma_samples, respectively, then ppsA and ppsB must have the same values ​​of conf_win_left_offset, conf_win_right_offset, conf_win_top_offset, and conf_win_bottom_offset, respectively.

[0163] 2.2.2 Processing Fractional Motion Compensation at Different Resolutions

[0164] In VVC draft 6, the fractional motion compensation process can handle different resolutions. The detailed specifications are as follows.

[0165] 8.5.6.3 Fractional Sample Interpolation Process

[0166] 8.5.6.3.1 Overview

[0167] The inputs to this process are:

[0168] – Luma position (xSb, ySb), specifies the top left sample of the current codec sub-block relative to the top left luma sample of the current picture,

[0169] –Variable sbWidth specifies the width of the current codec sub-block.

[0170] –Variable sbHeight specifies the height of the current codec sub-block.

[0171] – motion vector offset mvOffset,

[0172] – refined motion vector refMvLX,

[0173] – The selected reference picture sample array refPicLX,

[0174] – Half-sample interpolation filter index hpelIfIdx,

[0175] – bidirectional optical flow flag bdofFlag,

[0176] –Variable cIdx specifies the color component index of the current block.

[0177] The output of this process is:

[0178] – predSamplesLX, an array of (sbWidth+brdExtSize)x(sbHeight+brdExtSize) predicted sample values.

[0179] The prediction block boundary extension size brdExtSize is derived as follows:

[0180] brdExtSize=(bdofFlag||(inter_affine_flag[xSb][ySb]&&sps_affine_prof_enabled_flag))? 2:0(8-752)

[0181] Set the variable fRefWidth equal to PicOutputWidthL of the reference picture for the luma samples.

[0182] Set the variable fRefHeight equal to PicOutputHeightL of the reference picture for the luma samples.

[0183] Set the motion vector mvLX equal to (refMvLX-mvOffset).

[0184] – If cIdx is equal to 0, then the following applies:

[0185] – The scaling factor and its fixed-point representation are defined as:

[0186] hori_scale_fp=(fRefWidth<<14)+(PicOutputWidthL>>1)) / PicOutp

[0187] utWidthL(8-753)

[0188] vert_scale_fp=(fRefHeight<<14)+(PicOutputHeightL>>1)) / PicOut

[0189] putHeightL(8-754)

[0190] – Let (xIntL, yIntL) be the luma position given in full sample units, and (xFracL, yFracL) be the offset given in 1 / 16 sample units. These variables are used only in this clause to specify fractional sample positions within the reference sample array refPicLX.

[0191] – Set the upper left coordinate (xSbIntL, ySbIntL) of the boundary block used for reference sample filling to be equal to (xSb+(mvLX[0]>>4), ySb+(mvLX[1]>>4)).

[0192] – For each luma sample position in the predicted luma sample array predSamplesLX (xL = 0..sbWidth-1 + brdExtSize, yL = 0..sbHeight-1 + brdExtSize), the corresponding predicted luma sample value predSamplesLX[xL][yL] is derived as follows:

[0193] - Let (refxSbL, refySbL) and (refxL, refyL) be the luminance positions pointed to by the motion vector (refMvLX[0], refMvLX[1]) given in units of 1 / 16 samples. The variables refxSbL, refxL, refySbL, and refyL are derived as follows:

[0194] refxSbL = ( ( xSb << 4 ) + refMvLX[ 0 ] ) * hori_scale_fp (8-755)

[0195] refxL=((Sign(refxSb)*((Abs(refxSb)+128)>>8) +

[0196] xL * ( ( hori_scale_fp + 8 ) >> 4 ) ) + 32 ) >> 6 (8-756) refySbL=((ySb<<4) + refMvLX[1]) * vert_scale_fp (8-757) refyL=((Sign(refySb)*((Abs(refySb)+128)>>8)+yL*( ( vert_scale_fp + 8) >> 4 ) ) + 32 ) >> 6 (8-758)

[0197] – The variables xIntL, yIntL, xFracL, and yFracL are derived as follows:

[0198] xIntL = refxL >> 4 (8-759)

[0199] yIntL = refyL >> 4 (8-760)

[0200] xFracL = refxL & 15 (8-761)

[0201] yFracL = refyL & 15 (8-762)

[0202] – If bdofFlag is true or (sps_affine_prof_enabled_flag is true and inter_affine_flag[xSb][ySb] is true), and one or more of the following conditions are true, then in the interval between (xIntL+(xFracL>>3)-1), yIntL+(yFracL>>

[0203] 3)-1) and refPicLX are input by calling the luma sample retrieval process specified in clause 8.5.6.3.3 to derive the predicted luma sample value predSamplesLX[xL][yL].

[0204] 1.xL is equal to 0.

[0205] 2.xL equals sbWidth+1.

[0206] 3.yL is equal to 0.

[0207] 4.yL equals sbHeight + 1.

[0208] – Otherwise, in the order of (xIntL-(brdExtSize>0?1:0),

[0209] yIntL-(brdExtSize>0?1:0)), (xFracL,yFracL),

[0210] (xSbIntL,ySbIntL), refPicLX, hpelIfIdx, sbWidth, sbHeight and

[0211] When (xSb, ySb) is input, the predicted luma sample value predSamplesLX[xL][yL] is derived by calling the luma sample 8-tap interpolation filtering process specified in clause 8.5.6.3.2.

[0212] – Otherwise (cIdx is not equal to 0), then the following applies:

[0213] – Let (xIntC, yIntC) be the chroma position given in units of full samples, and

[0214] (xFracC, yFracC) are offsets given in units of 1 / 32 samples. These variables are used only in this clause to specify general fractional sample locations within the reference sample array refPicLX.

[0215] – Set the upper left coordinate (xSbIntC, ySbIntC) of the boundary block used for reference sample filling to be equal to ((xSb / SubWidthC)+(mvLX[0]>>5),

[0216] (ySb / SubHeightC)+(mvLX[1]>>5)).

[0217] – For each chroma sample position in the predicted chroma sample array (xC = 0..sbWidth-1, yC = 0..sbHeight-1), the corresponding predicted chroma sample value predSamplesLX[xC][yC] is derived as follows:

[0218] - Let (refxSbC, refySbC) and (refxC, refyC) be the chroma positions pointed to by the motion vector (mvLX[0], mvLX[1]) given in units of 1 / 32 samples. The variables refxSbC, refySbC, refxC, and refyC are derived as follows:

[0219] refxSbC=((xSb / SubWidthC<<5)+mvLX[0])*hori_scale_fp

[0220] (8-763)

[0221] refxC=((Sign(refxSbC)*((Abs(refxSbC)+256)>>9)

[0222] + xC * ( ( hori_scale_fp + 8 ) >> 4 ) ) + 16 ) >> 5 (8-764)

[0223] refySbC=((ySb / SubHeightC<<5)+mvLX[1])*vert_scale_fp

[0224] (8-765)

[0225] refyC=((Sign(refySbC)*((Abs(refySbC)+256)>>9)

[0226] +yC*((vert_scale_fp+8)>>4))+16)>>5

[0227] (8-766)–The variables xIntC, yIntC, xFracC, and yFracC are derived as follows:

[0228] xIntC = refxC >> 5 (8-767)

[0229] yIntC = refyC >> 5 (8-768)

[0230] xFracC = refyC & 31 (8-769)

[0231] yFracC = refyC & 31 (8-770)

[0232] Derives the predicted sample values ​​predSamplesLX[xC][yC] by invoking the procedure specified in clause 8.5.6.3.4 with (xIntC,yIntC), (xFracC,yFracC), (xSbIntC,ySbIntC), sbWidth, sbHeight and refPicLX as input.

[0233] 2.3 Scalable Video Codec in VVC

[0234] In VVC draft 6, inter-layer reference pictures are introduced to enable spatial and SNR scalability.

[0235] Inter-layer reference picture (ILRP): A picture that has the same access unit as the current picture, has a nuh_layer_id smaller than that of the current picture, and is marked as "used for long-term reference".

[0236] Additionally, inter-layer references to the current picture are supported using the following syntax.

[0237] vps_all_independent_layers_flag equal to 1 specifies that all layers within the CVS can be independently coded without using inter-layer prediction. vps_all_independent_layers_flag equal to 0 specifies that one or more layers in the CVS can use inter-layer prediction. When vps_all_independent_layers_flag is not present, its value is inferred to be equal to 1. When vps_all_independent_layers_flag is equal to 1, the value of vps_independent_layer_flag[i] is inferred to be equal to 1. When vps_all_independent_layers_flag is equal to 0, the value of vps_independent_layer_flag[0] is inferred to be equal to 1.

[0238] vps_layer_id[i] specifies the nuh_layer_id value of layer i. For any two non-negative integer values ​​m and n, if m is less than n, the value of vps_layer_id[m] must be less than vps_layer_id[n].

[0239] vps_independent_layer_flag[i] equal to 1 specifies that the layer with index i does not use inter-layer prediction. vps_independent_layer_flag[i] equal to 0 specifies that the layer with index i can use inter-layer prediction and vps_layer_dependency_flag[i] is present in the VPS.

[0240] vps_direct_dependency_flag[i][j] equal to 0 specifies that the layer with index j is not a direct reference layer for the layer with index i. vps_direct_dependency_flag[i][j] equal to 1 specifies that the layer with index j is a direct reference layer for the layer with index i. For i and j in the range of 0 to vps_max_layers_minus1 (inclusive), when vps_direct_dependency_flag[i][j] is not present, it is inferred to be equal to 0.

[0241] The j-th directly related layer variable DirectDependentLayerIdx[i][j] that specifies the i-th layer is derived as follows:

[0242]

[0243] The variable GeneralLayerIdx[i] that specifies the layer index of the layer with nuh_layer_id equal to vps_layer_id[i] is derived as follows:

[0244] for(i=0;i<=vps_max_layers_minus1;i++)

[0245] GeneralLayerIdx[vps_layer_id[i]]=i

[0246] ilrp_idc[listIdx][rplsIdx][i], for the list of directly related layers, specifies the index of the ILRP of the i-th entry in the ref_pic_list_struct(listIdx, rplsIdx) syntax structure to the list of directly related layers. The value of ilrp_idc[listIdx][rplsIdx][i] must be in the range of 0 to GeneralLayerIdx[nuh_layer_id]-1, inclusive.

[0247] The reference picture lists RefPicList[0] and RefPicList[1] are constructed as follows:

[0248]

[0249]

[0250] Decoded pictures in the DPB can be marked as "not used for reference", "used for short-term reference", or "used for long-term reference", but can only be marked as one of these three at any given moment during the operation of the decoding process. Assigning one of these markings to a picture implicitly removes the other of these markings, where applicable. When a picture is referred to as marked as "used for reference", it collectively refers to pictures marked as either "used for short-term reference" or "used for long-term reference" (but not both).

[0251] STRP and ILRP are identified by their nuh_layer_id value and PicOrderCntVal value. LTRP is identified by its nuh_layer_id value and Log2(MaxLtPicOrderCntLsb)LSB of its PicOrderCntVal value.

[0252] If the current picture is a CLVSS picture, all reference pictures (if any) currently in the DPB that have the same nuh_layer_id as the current picture are marked as "unused for reference".

[0253] Otherwise, the following applies:

[0254] For each LTRP entry in RefPicList[0] or RefPicList[1], mark the picture as “used for long-term reference” when the referenced picture is a STRP with the same nuh_layer_id as the current picture.

[0255] – Mark every reference picture in the DPB with the same nuh_layer_id as the current picture as “unused for reference”, which is not referenced by any entry in RefPicList[0] or RefPicList[1].

[0256] – For each ILRP entry in RefPicList[0] or RefPicList[1], mark the referenced picture as “used for long-term reference”.

[0257] After all slices of the current picture have been decoded, the current decoded picture is marked as "used for short-term reference" and each ILRP entry in RefPicList[0] or RefPicList[1] is marked as "used for short-term reference".

[0258] 2.4 Hypothetical Reference Decoder (HRD)

[0259] VVC follows previous standards such as HEVC and H.264, which have the concept of a hypothetical reference decoder (HRD). The HRD is used to set constraints on the bitstream that do not induce overflow or underflow of the hypothetical reference decoder.

[0260] 3. Technical Problems Solved by the Technical Solutions Disclosed in This Article

[0261] 1. Screen content codecs are sensitive to native resolution. It can happen that when resolution is reduced by downscaling the video, the bitrate increases significantly. Therefore, adaptive resolution scaling and scalable video codecs may not work well with video content.

[0262] 2. Scalability support in VVC may require certain constraints.

[0263] 3. There is no need to send QP related information in lossless mode.

[0264] 4. Examples of Embodiments and Technologies

[0265] Issues related to screen content encoding and decoding

[0266] 1. It is possible to indicate in the bitstream whether the picture / video has native resolution.

[0267] a. In one example, such indication may be signaled in the DPS / VPS / SPS / PPS / slice / slice header.

[0268] b. In one example, such an indication may be signaled in a SEI message.

[0269] 2. Whether an image / video has screen content may depend on the frequency of use of a certain codec tool.

[0270] a. In one example, a picture / video may be classified as screen content when the number / ratio / area of ​​regions in the transition skip codec mode exceeds a threshold.

[0271] b. In one example, the picture / video may be classified as screen content when the number / ratio / area of ​​regions in the BDPCM codec mode exceeds a threshold.

[0272] c. In one example, the picture / video may be classified as screen content when the number / ratio / area of ​​regions in the intra block copy codec mode exceeds a threshold.

[0273] d. In one example, the picture / video may be classified as screen content when the number / ratio / area of ​​regions in the palette codec mode exceeds a threshold.

[0274] 3. Whether the picture / video has screen content may depend on the information in the DPS / VPS / SPS / PPS / slice / strip header.

[0275] 4. Resolution changes may not be allowed for screen content.

[0276] a. In one example, whether a picture / video has screen content can be based on the method described above.

[0277] 5. When the reference picture and the current picture have different sizes, an alternative set of interpolation filters can be used for the screen content.

[0278] a. In one example, whether a picture / video has screen content can be based on the method described above.

[0279] b. In one example, the filter may be equivalent to obtaining the nearest integer pixel in the reference picture. c. In one example, for each interpolation filter in the substitution filter set, only one entry has a non-zero coefficient.

[0280] d. In one example, a bilinear interpolation filter may be used.

[0281] 6. A constraint may be set such that when the reference picture and the current picture have different sizes, the PicOutputWidthL and PicOutputHeightL of the current picture must be equal to those of the reference picture.

[0282] a. In one example, the consistency window of the reference picture and the consistency window of the current picture may have the same width.

[0283] b. In one example, the consistency window of the reference picture and the consistency window of the current picture may have the same height.

[0284] Scalability-related issues

[0285] 7. A constraint can be set such that low-quality layers cannot be used to predict high-quality layers.

[0286] a. In one example, if layer j is used to predict layer i, then the QP in layer i is not allowed to be greater than the QP in layer j.

[0287] b. In one example, if layer j is used to predict layer i, the maximum allowed QP of a block / TU / PU / CU / CTU / CTB in layer i may be capped by the QP of the corresponding block / TU / PU / CU / CTU / CTB in layer j.

[0288] c. In one example, if layer j is used to predict layer i, then the resolution of layer i is not allowed to be smaller than the resolution of layer j.

[0289] d. In one example, this constraint may be applied to pictures with the same picture order count number (ie, PicOrderCntVal).

[0290] 8. A constraint can be set such that the consistency window of the low-quality layer cannot be larger than the consistency window of the high-quality layer.

[0291] a. In one example, the width of the consistency window of the lower resolution picture may not be allowed to be larger than the width of the consistency window of the higher resolution picture.

[0292] b. In one example, the height of the consistency window of the lower resolution picture may not be allowed to be greater than the height of the consistency window of the higher resolution picture.

[0293] c. In one example, this constraint may be applied to pictures with the same picture order count number (ie, PicOrderCntVal).

[0294] 9. A constraint on the maximum number of layers with the same picture order count can be set.

[0295] a. In one example, the allowed number of layers with the same picture order count may be between 1 and some number T, inclusive.

[0296] b. The maximum number of layers with the same picture order count can be indicated in the DPS / VPS / SPS / PPS / slice / slice header.

[0297] 10. HRD parameters can be based on pictures with the same picture order count.

[0298] a. In one example, HRD parameters may be based on pictures in a decoded picture buffer (DPB) with the same picture order count.

[0299] b. In one example, all pictures with the same picture order count may be considered as whole pictures to derive HRD parameters.

[0300] c. In one example, each layer with the same picture order count can have a specific HRD

[0301] to derive the buffer parameters.

[0302] Lossless codec related issues

[0303] 11. In lossless codec mode, Qp-related syntax can be skipped.

[0304] a. In one example, the lossless codec mode can be indicated by cu_transquant_bypass_flag.

[0305] b. In one example, for a region / picture / video, the lossless codec mode may be indicated by a message in the DPS / VPS / SPS / PPS / slice / brick / strip header.

[0306] c. In one example, the syntax for Qp differences can be skipped.

[0307] d. In one example, the syntax regarding chroma Qp may be skipped.

[0308] 12. In lossless codec mode, cbf related syntax can be skipped.

[0309] a. In one example, the lossless codec mode can be indicated by cu_transquant_bypass_flag.

[0310] b. In one example, for regions / pictures / videos, you can use DPS / VPS / SPS / PPS /

[0311] The / brick / strip message in the header indicates the lossless codec mode.

[0312] c. In one example, the syntax for luma cbf can be skipped.

[0313] d. In one example, the syntax for chroma cbf can be skipped.

[0314] 5. Examples

[0315] 5.1 Example #1

[0316] This embodiment corresponds to bullet 11. Changes are underlined.

[0317]

[0318] Figure 5 is a block diagram illustrating an exemplary video processing system 500 in which the various techniques disclosed herein may be implemented. Various embodiments may include some or all of the components of system 500. System 500 may include an input 502 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8-bit or 10-bit multi-component pixel values, or in a compressed or codec-encoded format. Input 502 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, a passive optical network (PON), and wireless interfaces such as Wi-Fi or a cellular interface.

[0319] System 500 may include a codec component 504 that can implement various codecs or codec encoding methods described in this document. Codec component 504 can reduce the average bit rate of the video from input 502 to the output of codec component 504 to generate a codec representation of the video. Therefore, codec technology is sometimes referred to as video compression or video transcoding technology. The output of codec component 504 can be stored or transmitted via connected communications, as represented by component 506. The stored or communicated bitstream (or coded) representation of the video received at input 502 can be used by component 508 to generate pixel values ​​or send to a displayable video of display interface 510. The process of generating a user-viewable video from the bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it should be understood that the codec tools or operations are used at the encoder, and the corresponding decoding tools or operations that reverse the codec results will be performed by the decoder.

[0320] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), or DisplayPort, etc. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE interfaces, etc. The technology described in this document may be embodied in various electronic devices, such as mobile phones, laptop computers, smart phones, or other devices capable of performing digital data processing and / or video display.

[0321] Figure 6 6 is a block diagram of a video processing device 600. Device 600 can be used to implement one or more of the methods described herein. Device 600 can be embodied in a smartphone, tablet, computer, Internet of Things (IoT) receiver, etc. Device 600 may include one or more processors 602, one or more memories 604, and video processing hardware 606. Processor(s) 602 can be configured to implement one or more of the methods described herein. Memory(s) 604 can be used to store data and code used to implement the methods and techniques described herein. Video processing hardware 606 can be used to implement some of the techniques described herein in hardware circuitry.

[0322] The following solutions may be implemented as preferred solutions in some embodiments.

[0323] The solutions described below can be implemented with the additional techniques described in the items listed in the previous sections (eg, items 1, 4).

[0324] 1. A video processing method (e.g., Figure 7), which includes performing a conversion between a video having a native resolution and a codec representation of the video having a codec resolution, wherein a field in the codec representation indicates a relationship between the native resolution and the codec resolution.

[0325] 2. The method according to solution 1, wherein the field is included at the video parameter set level, the picture parameter set level, the slice header level, the slice level, or the sequence parameter set level.

[0326] 3. The method according to any of solutions 1-2, wherein the field is included together with supplemental enhancement information.

[0327] 4. The method according to any of solutions 1-3, wherein the conversion is based on a screen content codec tool that depends on the relationship between the native resolution and the codec resolution.

[0328] 5. The method according to any of solutions 1-4, wherein the codec resolution is equal to the native resolution due to the use of a screen content codec tool during the conversion.

[0329] The solutions described below can be implemented with the additional techniques described in the items listed in the previous sections (eg, item 2).

[0330] 6. A video processing method, comprising: determining, for conversion between a video segment of a video and a codec representation of the video, the suitability of a screen content codec for conversion of the video segment based on usage rules associated with codec mode usage of blocks in the video segment; and performing the conversion based on the determination.

[0331] 7. The method according to solution 6, wherein the coding mode is a transform skip mode.

[0332] 8. The method according to solution 6, wherein the coding mode is a fuzzy differential pulse coding modulation coding mode.

[0333] 9. The method according to solution 6, wherein the codec mode is an intra block copy codec mode.

[0334] 10. The method according to solution 6, wherein the encoding and decoding mode is a palette mode encoding and decoding mode.

[0335] 11. The method according to any of solutions 6-10, wherein the usage rule is based on the number of occurrences, or based on the ratio of blocks encoded using the codec mode to blocks encoded not using the codec mode, or the area of ​​the area encoded using the codec mode.

[0336] The solutions described below can be implemented with the additional techniques described in the items listed in the previous sections (eg, item 3).

[0337] 12. The method according to any of solutions 1-11, wherein a syntax element in the codec representation indicates that the codec representation uses screen content codec.

[0338] The solutions described below can be implemented with the additional techniques described in the items listed in the previous sections (eg, items 5, 6).

[0339] 13. A video processing method comprising: determining, for a conversion between video units of a video and a codec representation of the video using a screen content codec tool, a characteristic of the conversion caused by a reference picture and a current picture including video units of different sizes; and performing the conversion based on the determination.

[0340] 14. The method of solution 13, wherein the characteristics of the conversion include an interpolation filter used for interpolation of screen content during the conversion.

[0341] 15. The method of solution 13, wherein the characteristics of the conversion include a picture output width or a picture output height associated with the current picture.

[0342] The solutions described below can be implemented with the additional techniques described in the items listed in the previous sections (eg, items 7, 8, 9, 10).

[0343] 16. A video processing method, comprising: determining a constraint rule for a conversion between a video unit of a video and a scalable codec representation of the video; and performing the conversion according to the constraint rule; wherein the scalable codec representation includes a first layer and a second layer, wherein the first layer has a lower quality than the second layer.

[0344] 17. The method of solution 16, wherein the constraint does not allow the second layer to be predicted from the first layer.

[0345] 18. The method of solution 16, wherein if layer j is used to predict layer i, then the constraint does not allow the quantization parameter (QP) in layer i to be greater than the QP in layer j.

[0346] 19. The method according to solution 16, wherein the constraint rule stipulates that the consistency window of the first layer is not larger than the consistency window of the second layer.

[0347] 20. The method of solution 16, wherein the constraint specifies a maximum number of picture layers used in the codec representation and having the same picture order count.

[0348] 21. The method of solution 16, wherein the constraint specifies using a hypothetical reference decoder based on pictures with the same picture order count.

[0349] The solutions described below can be implemented with the additional techniques described in the items listed in the previous sections (eg, items 11, 12).

[0350] 22. A video processing method comprising: for converting between a video and a codec representation of the video using a lossless codec mode, determining that the codec representation omits syntax elements related to signaling of a quantization parameter or codec block signaling; and performing the converting based on the determination.

[0351] 23. The method of solution 22, wherein the codec representation includes an indication of the lossless codec mode as a codec level quantization bypass flag.

[0352] 24. The method according to any of solutions 22-23, wherein the lossless codec mode is indicated at a slice or block or slice or picture or sequence or video level.

[0353] 25. The method of any of solutions 1-24, wherein the converting comprises generating the codec representation from the video region.

[0354] 26. The method of any of solutions 1-24, wherein the converting comprises encoding and decoding the video region to generate the codec representation.

[0355] 27. A video decoding device comprising a processor configured to implement the method according to one or more of solutions 1 to 26.

[0356] 28. A video coding and decoding device, comprising a processor configured to implement the method according to one or more of solutions 1 to 26.

[0357] 29. A computer program product having computer code stored thereon, which, when executed by a processor, causes the processor to implement the method according to any of solutions 1 to 26.

[0358] 30. The method, apparatus, or system described in this document.

[0359] In the above solutions, performing the conversion includes using the results of previous decision steps during the encoding or decoding operation (eg, using or not using certain encoding or decoding steps) to obtain the conversion result.

[0360] Figure 8 A flowchart of an example method for video processing is shown. The method includes: for converting between a picture or video and a bitstream representation of the picture or video, determining (802) whether the picture or video has a native resolution based on an indication included in the bitstream representation; and performing (804) the conversion based on the determination.

[0361] In some examples, the indication is signaled in at least one of the following options: dependency parameter set (DPS), video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice and slice headers.

[0362] In some examples, the indication is signaled in a Supplemental Enhancement Information (SEI) message.

[0363] Figure 9 A flowchart of an example method for video processing is shown. The method includes: determining (902) whether a picture or video is screen content based on the use of one or more codecs or signaling messages for converting between the picture or video and a bitstream representation of the picture or video; and performing (904) the conversion based on the determination.

[0364] In some examples, some of the codec tools include a transform skip codec mode, and the picture or video is determined to be screen content when at least one of the number of regions with the transform skip codec mode, the proportion of regions, and the area of ​​regions exceeds a threshold.

[0365] In some examples, the certain codec tool includes a block differential pulse codec modulation (BDPCM) codec mode, and the picture or video is determined to be screen content when at least one of the number of regions, the proportion of regions, and the area of ​​regions with the BDPCM codec mode exceeds a threshold.

[0366] In some examples, the certain codec tool includes an intra-block copy codec mode, and the picture or video is determined to be screen content when at least one of the number of regions, the proportion of regions, and the area of ​​regions with the intra-block copy codec mode exceeds a threshold.

[0367] In some examples, the certain codec tool includes a palette codec mode, and the picture or video is determined to be screen content when at least one of the number of regions with the palette codec mode, the proportion of regions, and the area of ​​regions exceeds a threshold.

[0368] In some examples, the message is signaled among at least one of the following options: dependency parameter set (DPS), video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), slice and slice header.

[0369] In some examples, when the picture or video is determined to be screen content, no resolution changes are allowed.

[0370] In some examples, when the picture or video is determined to be screen content, and when the reference picture and the current picture have different sizes, an alternative set of interpolation filters may be applied to the picture or video.

[0371] In some examples, the filters in the replacement set of interpolation filters are equivalent to obtaining the nearest integer pixel in the reference picture.

[0372] In some examples, for each interpolation filter in the replacement set of interpolation filters, only one entry has a non-zero coefficient.

[0373] In some examples, the replacement set of interpolation filters includes bilinear interpolation filters.

[0374] In some examples, when determining a current picture or video as screen content, and when a reference picture and the current picture have different sizes, one or more constraints may apply to the picture or video.

[0375] In some examples, the one or more constraints include: the picture output width PicOutputWidthL and / or the picture output height PicOutputHeightL of the current picture are equal to those of the reference picture.

[0376] In some examples, the one or more constraints include: a consistency window of the reference picture and a consistency window of the current picture having the same width.

[0377] In some examples, the one or more constraints include: a consistency window of the reference picture and a consistency window of the current picture having the same height.

[0378] Figure 10 A flowchart of an example method for video processing is shown. The method includes: determining (1002) one or more constraints associated with a scalable video codec for converting a picture or video to a scalable codec bitstream representation of the picture or video; and performing (1004) the conversion based on the determination.

[0379] In some examples, the scalable codec bitstream representation includes at least a first layer and a second layer, wherein the first layer has a low quality and the second layer has a high quality.

[0380] In some examples, the one or more constraints include not being able to predict a second layer having a high quality using a first layer having a low quality.

[0381] In some examples, if a first layer is predicted using a second layer, the quantization parameter (QP) in the first layer is not allowed to be larger than the QP in the second layer.

[0382] In some examples, if the second layer is used to predict the first layer, the maximum allowed QP of a block, transform unit (TU), prediction unit (PU), codec unit (CU), codec tree unit (CTU), or codec tree block (CTB) in the first layer is capped at the QP of the corresponding block, TU, PU, ​​CU, CTU, or CTB in the second layer.

[0383] In some examples, if the second layer is used to predict the first layer, the resolution of the first layer is not allowed to be smaller than the resolution in the second layer.

[0384] In some examples, the constraint is applied to pictures with the same picture order count number.

[0385] In some examples, the one or more constraints include: a consistency window of a first layer having low quality cannot be larger than a consistency window of a second layer having high quality.

[0386] In some examples, the width of the consistency window of the lower resolution picture is not allowed to be larger than the width of the consistency window of the higher resolution picture.

[0387] In some examples, the height of the coherence window of the lower resolution picture is not allowed to be larger than the height of the coherence window of the higher resolution picture.

[0388] In some examples, the constraint is applied to pictures with the same picture order count number.

[0389] In some examples, the one or more constraints include that a maximum number of layers with the same picture order count is settable.

[0390] In some examples, the allowed number of layers with the same picture order count is between 1 and some number T (inclusive), where T is an integer.

[0391] In some examples, a maximum number of layers with the same picture order count is indicated in at least one of the DPS, VPS, SPS, PPS, slice, and slice headers.

[0392] In some examples, hypothetical reference decoder (HRD) parameters are based on pictures with the same picture order count.

[0393] In some examples, the HRD parameters are based on pictures in a decoded picture buffer (DPB) with the same picture order count.

[0394] In some examples, all pictures with the same picture order count are considered as whole pictures to derive HRD parameters.

[0395] In some examples, each layer with the same picture order count has a specific HRD to derive buffer parameters.

[0396] Figure 11 A flowchart of an example method for video processing is shown. The method includes: determining (1102) to skip one or more syntax elements when converting between a picture or video and a bitstream representation of the picture or video using a lossless codec mode; and performing (1104) the conversion based on the determination.

[0397] In some examples, the one or more syntaxes include quantization parameter (QP) related syntax.

[0398] In some examples, lossless codec mode is indicated by cu_transquant_bypass_flag.

[0399] In some examples, for a region, picture, or video, the lossless codec mode is indicated by a message in at least one of a DPS, VPS, SPS, PPS, slice, brick, and band header.

[0400] In some examples, the one or more syntaxes include syntax for QP differences.

[0401] In some examples, the one or more syntaxes include syntax related to chroma QP.

[0402] In some examples, the one or more syntaxes include codec block flag (cbf) related syntax.

[0403] In some examples, lossless codec mode is indicated by cu_transquant_bypass_flag.

[0404] In some examples, for a region, picture, or video, the lossless codec mode is indicated by a message in at least one of a DPS, VPS, SPS, PPS, slice, brick, and band header.

[0405] In some examples, the one or more syntaxes include syntax for luma cbf.

[0406] In some examples, the one or more syntaxes include syntax for chroma cbf.

[0407] In some examples, performing the conversion includes generating a high bitrate representation from the picture or video.

[0408] In some examples, performing the conversion includes generating the picture or video from the bitstream representation.

[0409] The disclosed and other solutions, examples, embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuitry or computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or a combination of one or more thereof. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by a data processing apparatus or to control the operation of the data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that effects a machine-readable propagated signal, or a combination of one or more thereof. The term "data processing apparatus" encompasses all apparatuses, devices, and machines for processing data, including (for example) a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode or decode information for transmission to a suitable receiver device.

[0410] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language (including compiled or interpreted languages) and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one or more computers that are located at a single site or distributed across multiple sites and interconnected by a communications network.

[0411] The processes and logic flows described in this specification can be performed by executing one or more computer programs by one or more programmable processors to perform functions by operating on input data and generating output. These processes and logic flows can also be performed by special purpose logic circuitry, and the apparatus can also be implemented as special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0412] For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally speaking, a processor will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor that executes instructions and one or more memory devices that store instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or be operatively coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD ROM disks. The processor and memory may be supplemented by, or incorporated into, special-purpose logic circuitry.

[0413] Although this patent document contains many details, they should not be interpreted as limitations on any subject matter or the scope of the claims, but rather as descriptions of specific features of specific embodiments of particular technologies. Certain features described in this patent document in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments, either individually or in any suitable subcombination. Furthermore, although certain features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be removed from the combination, and a claimed combination may involve subcombinations or variations of subcombinations.

[0414] Similarly, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, in order to achieve desired results. Furthermore, the division of various system components among the embodiments described in this patent document should not be understood as requiring such division in all embodiments.

[0415] Only a few implementations and examples are described, and other implementations, enhancements, and variations can be made based on what is described and illustrated in this patent document.

Claims

1. A method for video processing, comprising: For a first conversion between a picture or video and a bitstream representation of the picture or video, determining whether the picture or video is screen content based on the use of one or more codec tools or signaling messages; as well as performing the first conversion based on the determination; The certain codec tool includes a transform skip codec mode, and when at least one of the number of regions, the proportion of regions, and the area of ​​regions having the transform skip codec mode exceeds a first threshold, the picture or video is determined as the screen content; The certain codec tool includes a Block Differential Pulse Width Modulation (BDPCM) codec mode, and when at least one of the number of regions, the proportion of regions, and the area of ​​regions having the BDPCM codec mode exceeds a second threshold, the picture or video is determined as the screen content; The certain codec tool includes an intra block copy codec mode, and when at least one of the number of regions, the proportion of regions, and the area of ​​regions having the intra block copy codec mode exceeds a third threshold, the picture or video is determined as the screen content; Wherein, the certain coding tool includes a palette coding mode, and when at least one of the number of regions, the proportion of regions and the area of ​​regions having the palette coding mode exceeds a fourth threshold, the picture or video is determined as the screen content.

2. The method according to claim 1, wherein The message is signaled among at least one of the following options: dependency parameter set DPS, video parameter set VPS, sequence parameter set SPS, picture parameter set PPS, slice and slice header.

3. The method according to claim 1, wherein When the picture or video is determined as the screen content, no resolution change is allowed.

4. The method according to claim 1, wherein When the picture or video is determined to be the screen content, and when a reference picture and a current picture have different sizes, a replacement set of interpolation filters may be applicable to the picture or video.

5. The method according to claim 4, wherein The filters in the replacement set of interpolation filters are equivalent to obtaining the nearest integer pixel in the reference picture.

6. The method according to claim 4, wherein: For each interpolation filter in the replacement set of interpolation filters, only one entry has a non-zero coefficient.

7. The method according to claim 4, wherein: The replacement set of interpolation filters includes bilinear interpolation filters.

8. The method according to claim 1, wherein When determining the picture or video as the screen content, and when a reference picture and a current picture have different sizes, one or more constraints may be applied to the picture or video.

9. The method according to claim 8, wherein The one or more constraints include: the picture output width PicOutputWidthL and / or the picture output height PicOutputHeightL of the current picture is equal to the picture output width PicOutputWidthL and / or the picture output height PicOutputHeightL of the reference picture.

10. The method according to claim 9, wherein: The one or more constraints include: a consistency window of the reference picture and a consistency window of the current picture having the same width.

11. The method according to claim 9, wherein The one or more constraints include: a consistency window of the reference picture and a consistency window of the current picture having the same height.

12. The method according to claim 1, further comprising: for a second conversion between the picture or video and a bitstream representation of the picture or video, determining whether the picture or video has a native resolution based on an indication included in the bitstream representation; as well as The second conversion is performed based on the determination.

13. The method according to claim 12, wherein: The indication is signaled among at least one of the following options: dependency parameter set DPS, video parameter set VPS, sequence parameter set SPS, picture parameter set PPS, slice and slice header.

14. The method according to claim 12, wherein: The indication is signaled in a supplemental enhancement information SEI message.

15. The method according to claim 1, further comprising: determining, for a third conversion between the picture or video and a scalable codec bitstream representation of the picture or video, one or more constraints associated with a scalable video codec for the picture or video; as well as The third conversion is performed based on the determination.

16. The method according to claim 15, wherein The scalable codec bitstream representation comprises at least a first layer and a second layer, wherein the first layer has a low quality and the second layer has a high quality.

17. The method according to claim 16, wherein The one or more constraints include that a second layer having the high quality cannot be predicted using a first layer having the low quality.

18. The method according to claim 16, wherein If the second layer is used to predict the first layer, a quantization parameter QP in the first layer is not allowed to be greater than a QP in the second layer.

19. The method according to claim 16, wherein If the second layer is used to predict the first layer, the maximum allowed QP of the block, transform unit TU, prediction unit PU, codec unit CU, codec tree unit CTU or codec tree block CTB in the first layer is capped by the QP of the corresponding block, TU, PU, ​​CU, CTU or CTB in the second layer.

20. The method according to claim 16, wherein If the second layer is used to predict the first layer, the resolution of the first layer is not allowed to be smaller than the resolution in the second layer.

21. The method according to claim 15, wherein The constraint is applied to pictures with the same picture order count number.

22. The method according to claim 16, wherein The one or more constraints include: a consistency window of the first layer having the low quality cannot be larger than a consistency window of the second layer having the high quality.

23. The method according to claim 21, wherein The width of the consistency window of the lower resolution picture is not allowed to be larger than the width of the consistency window of the higher resolution picture.

24. The method according to claim 21, wherein The height of the consistency window of a lower resolution picture is not allowed to be larger than the height of the consistency window of a higher resolution picture.

25. The method according to claim 22, wherein The constraint is applied to pictures with the same picture order count number.

26. The method according to claim 15, wherein The one or more constraints include that a maximum number of layers having the same picture order count is configurable.

27. The method according to claim 26, wherein The allowed number of layers with the same picture order count is between 1 and a certain number T, inclusive, where T is an integer.

28. The method according to claim 19, wherein The maximum number of layers having the same picture order count is indicated in at least one of a DPS, VPS, SPS, PPS, slice, and slice header.

29. The method according to claim 15, wherein The hypothetical reference decoder HRD parameters are based on pictures with the same picture order count.

30. The method according to claim 29, wherein The HRD parameters are based on pictures with the same picture order count in the decoded picture buffer DPB.

31. The method according to claim 29, wherein The HRD parameters are derived by treating all pictures with the same picture order count as whole pictures.

32. The method of claim 29, wherein: Each layer with the same picture order count has a specific HRD to derive buffer parameters.

33. The method of claim 1, further comprising: For a fourth conversion between the picture or video and a bitstream representation of the picture or video, determining to skip one or more syntax elements when the fourth conversion uses a lossless codec mode; as well as The fourth conversion is performed based on the determination.

34. The method according to claim 33, wherein The one or more syntaxes include quantization parameter (QP) related syntax.

35. The method of claim 33, wherein: The lossless codec mode is indicated by cu_transquant_bypass_flag.

36. The method of claim 33, wherein: For a region, a picture or a video, the lossless codec mode is indicated by a message in at least one of a DPS, a VPS, a SPS, a PPS, a slice, a brick and a slice header.

37. The method of claim 33, wherein: The one or more syntaxes include syntax for QP difference.

38. The method of claim 33, wherein: The one or more syntaxes include syntax related to chroma QP.

39. The method of claim 33, wherein: The one or more syntaxes include codec block flag (cbf) related syntax.

40. The method of claim 39, wherein The lossless codec mode is indicated by cu_transquant_bypass_flag.

41. The method according to claim 40, wherein For a region, a picture or a video, the lossless codec mode is indicated by a message in at least one of a DPS, a VPS, a SPS, a PPS, a slice, a brick and a slice header.

42. The method of claim 39, wherein: The one or more syntaxes include syntax for luma cbf.

43. The method of claim 39, wherein: The one or more syntaxes include syntax for chroma cbf.

44. The method according to any one of claims 1 to 43, wherein The performing of the first conversion, the second conversion, the third conversion, or the fourth conversion includes generating the bitstream representation from the picture or video.

45. The method according to any one of claims 1 to 43, wherein The performing of the first conversion, the second conversion, the third conversion or the fourth conversion comprises generating the picture or video from the bitstream representation.

46. ​​An apparatus in a video system comprising a processor and a non-transitory memory having instructions located thereon, wherein: The instructions, when executed by the processor, cause the processor to implement the method according to any one of claims 1 to 45.

47. A computer program product comprising computer instructions, wherein: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 45 are implemented.

48. A computer-readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 45 are implemented.

Images