Method and apparatus for decoding video using image size-dependent scaling

Abstract

A system, method, and computer-readable storage medium for calculating a scaling ratio are disclosed. An example method may include obtaining a current picture of video data; determining one or more scaling windows associated with the current picture and / or a reference picture selected for use in encoding the current picture; determining a first size of the current picture and / or a second size of the reference picture, the first size including a first height and a first width, and the second size including a second height and a second width; determining one or more scaling ratios associated with the current picture and / or the reference picture based on the first size and / or the second size and the one or more scaling windows.

Description

Technical Field

[0001] The present application is related to video encoding and compression. More specifically, the present application relates to a system and method for calculating scaling ratios. Background Art

[0002] Many devices and systems allow video data to be processed and output for consumption. Digital video data often includes large amounts of data to meet the needs of video consumers and providers. For example, consumers of video data often prefer or expect videos of high quality, high fidelity, high resolution, high frame rate, etc. As a result, the large amounts of video data required to meet these needs place a burden on communication networks and devices that process and store video data.

[0003] Various video coding techniques can be used to compress video data. Video coding techniques can be performed according to one or more video coding standards. Example video coding standards that can be used for video coding include high efficiency video coding (HEVC), advanced video coding (AVC), MPEG-2 Part 2 coding (MPEG stands for Moving Picture Experts Group), VP9, ​​Alliance for Open Media (AOMedia) Video 1 (AV1), basic video coding (EVC), universal video coding (VVC), etc. In order to compress video data, video coding generally utilizes prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.), which utilize redundancy present in video images or sequences. An example goal of video coding is to compress video data into a form using a lower bit rate while avoiding or minimizing the degradation of video quality. With the increasing availability of evolving video services, coding techniques with improved coding accuracy and / or efficiency are needed. Summary of the invention

[0004] Disclosed are systems, methods, and computer-readable media for calculating scaling ratios and decoupling scaling ratio calculations from a picture output process. According to at least one example, a method for calculating scaling ratios and decoupling scaling ratio calculations from a picture output process is provided. An example method may include: obtaining a current picture of video data; determining one or more scaling windows associated with the current picture and / or a reference picture selected for use in encoding the current picture; determining a first size of the current picture and / or a second size of the reference picture, the first size including a first height and a first width of the current picture, and the second size including a second height and a second width of the reference picture; determining one or more scaling ratios associated with the current picture and / or the reference picture based on the one or more scaling windows and the first size and / or the second size.

[0005] According to at least one example, a non-transitory computer-readable storage medium for calculating a scaling ratio and decoupling the scaling ratio calculation from a picture output process is provided. An example non-transitory computer-readable storage medium may include instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the following operations: obtain a current picture of video data; determine one or more scaling windows associated with the current picture and / or a reference picture selected for use in encoding the current picture; determine a first size of the current picture and / or a second size of the reference picture, the first size including a first height and a first width of the current picture, and the second size including a second height and a second width of the reference picture; determine one or more scaling ratios associated with the current picture and / or the reference picture based on the one or more scaling windows and the first size and / or the second size.

[0006] According to at least one example, a device for calculating a scaling ratio and decoupling the scaling ratio calculation from a picture output process is provided. An example device may include: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: obtain a current picture of video data; determine one or more scaling windows associated with the current picture and / or a reference picture selected for use in encoding the current picture; determine a first size of the current picture and / or a second size of the reference picture, the first size including a first height and a first width of the current picture, and the second size including a second height and a second width of the reference picture; based on the one or more scaling windows and the first size and / or the second size, determine one or more scaling ratios associated with the current picture and / or the reference picture.

[0007] According to at least one example, another device for calculating a scaling ratio and decoupling the scaling ratio calculation from a picture output process is provided. The device may include a module for performing the following operations: obtaining a current picture of video data; determining one or more scaling windows associated with the current picture and / or a reference picture selected for use in encoding the current picture; determining a first size of the current picture and / or a second size of the reference picture, the first size including a first height and a first width of the current picture, and the second size including a second height and a second width of the reference picture; based on the one or more scaling windows and the first size and / or the second size, determining one or more scaling ratios associated with the current picture and / or the reference picture.

[0008] In some aspects, the above methods, non-transitory computer-readable media, and apparatus may include adjusting one or more motion compensation parameters associated with the current picture and / or the reference picture based on the one or more scaling ratios.

[0009] In some aspects, the above-described methods, non-transitory computer-readable media, and apparatus may include: including the one or more scaling windows in a picture parameter set (PPS); and storing and / or sending the PPS including the one or more scaling windows.

[0010] In some examples, the one or more scaling ratios are derived for the horizontal direction and the vertical direction.In some examples, the one or more scaling windows may include a scaling window of the current picture, and the scaling window of the current picture may be determined based on a PPS referenced by the current picture.

[0011] In some examples, the one or more scaling windows may include a scaling window of the reference picture, and the scaling window of the reference picture may be determined according to a PPS referenced by the reference picture.

[0012] In some cases, the one or more scaling windows may include a scaling window offset, and the scaling window offset may include at least one of the following: a left scaling window offset, a right scaling window offset, a top scaling window offset, and a bottom scaling window offset. In some examples, at least one of the scaling window offsets may include a negative value. In some cases, determining the one or more scaling ratios may include adding the scaling window offset to the first size of the current picture and / or the second size of the reference picture.

[0013] In some examples, the one or more zoom windows may include a first zoom window associated with the current image and a second zoom window associated with the reference image, and the one or more zoom ratios may be determined based on the first zoom window and the second zoom window.

[0014] In some aspects, the above methods, non-transitory computer-readable media, and apparatus may include: obtaining one or more consistency windows associated with the current picture and the reference picture; and resizing the current picture and / or the reference picture based on the one or more consistency windows. In some cases, the one or more consistency windows are determined according to the SPS.

[0015] In some aspects, the above-described methods, non-transitory computer-readable media, and apparatus may include: the one or more consistency windows in an SPS; and storing and / or sending the SPS including the one or more consistency windows.

[0016] In some aspects, the above methods, non-transitory computer-readable media, and apparatus may include: obtaining a maximum picture size indicating a maximum picture width and a maximum picture height; and generating an output picture based on the maximum picture size and a consistency window including one or more window offsets. In some examples, the size of the current picture is equal to the maximum picture size, and the output picture includes the current picture having the size equal to the maximum picture size cropped based on the consistency window. In some examples, the current picture is cropped by applying the one or more window offsets to the current picture. In some cases, the maximum picture size and the consistency window are determined according to an SPS.

[0017] In some aspects, the above-described methods, non-transitory computer-readable media, and apparatus may include: the maximum picture size and the consistency window in an SPS; and storing and / or sending the SPS including the maximum picture size and the consistency window.

[0018] In some examples, the one or more scaling windows are further associated with one or more additional reference pictures selected for use in encoding the current picture, and the one or more scaling ratios are further associated with the one or more additional reference pictures.

[0019] In some examples, the above-mentioned methods, non-transitory computer-readable media, and devices may include: generating an encoded video bitstream including the one or more scaling windows and a consistency window set, the consistency window set including one or more first window offsets for resizing the current image and one or more second window offsets for resizing the reference image; and storing and / or sending the encoded video bitstream.

[0020] In some aspects, any of the above apparatuses may include a mobile device having a camera for capturing one or more pictures. In some aspects, any of the above apparatuses may include a display for displaying one or more pictures.

[0021] This summary is neither intended to identify key or essential features of the claimed subject matter nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0022] The foregoing as well as other features and embodiments will become more fully apparent after reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to describe the manner in which the various advantages and features of the present disclosure can be obtained, a more particular description of the above principles will be provided with reference to specific embodiments thereof shown in the accompanying drawings. It is to be understood that these drawings depict only exemplary embodiments of the present disclosure and are not to be considered as limiting the scope thereof. The principles of the present invention will be described and explained with additional particularity and detail through the use of the accompanying drawings, in which:

[0024] Figure 1 is a block diagram illustrating examples of encoding devices and decoding devices according to some examples;

[0025] Figure 2A is a conceptual diagram illustrating example spatial neighboring motion vector candidates for merge mode according to some examples;

[0026] Figure 2B is a conceptual diagram illustrating example spatial neighboring motion vector candidates for an Advanced Motion Vector Prediction (AMVP) mode according to some examples;

[0027] Figure 3A is a conceptual diagram illustrating example temporal motion vector predictor (TMVP) candidates according to some examples;

[0028] Figure 3B is a conceptual diagram illustrating examples of motion vector scaling according to some examples;

[0029] Figure 4A is a conceptual diagram illustrating an example of neighboring samples of a current coding unit for estimating motion compensation parameters for the current coding unit according to some examples;

[0030] Figure 4B is a conceptual diagram illustrating an example of neighboring samples of a reference block for estimating motion compensation parameters for a current coding unit according to some examples;

[0031] Figure 5 is a conceptual diagram illustrating example scaling calculations decoupled from a picture output process according to some examples;

[0032] Figure 6 is a conceptual diagram illustrating example scaling calculations based on a current picture and a reference picture according to some examples;

[0033] Figure 7 is a flow chart illustrating an example process for calculating a scaling ratio and decoupling the scaling ratio calculation from a picture output process according to some examples;

[0034] Figure 8 is a block diagram illustrating an example video encoding device according to some examples; and

[0035] Fig. 9 is a block diagram illustrating an example video decoding device according to some examples. DETAILED DESCRIPTION

[0036] Provided below are some aspects and embodiments of the present disclosure. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that each embodiment may be implemented without these specific details. The accompanying drawings and description are not intended to be restrictive.

[0037] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. Specifically, the following description of these exemplary embodiments will provide a feasible description for implementing the exemplary embodiments to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0038] Video encoding devices implement video compression techniques to efficiently encode and decode video data. Video compression techniques may include applying different prediction modes (including spatial prediction (e.g., intra-frame prediction or intra-prediction), temporal prediction (e.g., inter-frame prediction or inter-prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques) to reduce or remove redundancy inherent in video sequences. The video encoder may partition each picture of the original video sequence into rectangular areas, which are referred to as video blocks or coding units (described in more detail below). These video blocks may be encoded using specific prediction modes.

[0039] Video blocks can be divided into one or more groups of smaller blocks in one or more ways. Blocks may include coding tree blocks, prediction blocks, transform blocks, or other appropriate blocks. Unless otherwise specified, references to "blocks" may generally refer to such video blocks (e.g., coding tree blocks, coding blocks, prediction blocks, transform blocks, or other appropriate blocks or sub-blocks, as will be understood by those of ordinary skill in the art). In addition, each of these blocks may also be referred to interchangeably herein as a "unit" (e.g., coding tree unit (CTU), coding unit, prediction unit (PU), transform unit (TU), etc.). In some cases, a unit may indicate a coding logic unit encoded in a bitstream, and a block may indicate a portion of a video frame buffer for which the process is directed.

[0040] For inter-prediction mode, the video encoder can search for a block similar to the block being encoded at another temporal position in a frame (or picture), referred to as a reference frame or reference picture. The video encoder can limit the search to a specific spatial displacement from the block to be encoded. A two-dimensional (2D) motion vector including a horizontal displacement component and a vertical displacement component can be used to locate the best match. For intra-prediction mode, the video encoder can use spatial prediction techniques to form a prediction block based on data from previously encoded neighboring blocks within the same picture.

[0041] The video encoder can determine the prediction error. For example, the prediction can be determined as the difference between the pixel values ​​in the block being encoded and the predicted block. The prediction error can also be referred to as a residual. The video encoder can also apply a transform to the prediction error (e.g., a discrete cosine transform (DCT) or other appropriate transform) to generate transform coefficients. After the transform, the video encoder can quantize the transform coefficients. The quantized transform coefficients and motion vectors can be represented using syntax elements and form a coded representation of the video sequence together with the control information. In some cases, the video encoder can entropy encode the syntax elements to further reduce the number of bits required for their representation.

[0042] The video decoder can use the syntax elements and control information discussed above to construct prediction data (e.g., a prediction block) for decoding the current frame. For example, the video decoder can add the prediction block and the compressed prediction error. The video decoder can determine the compressed prediction error by weighting the transform basis function using the quantization coefficients. The difference between the reconstructed frame and the original frame is called the reconstruction error.

[0043] Some coding schemes can implement reference picture resampling (RPR), which allows the current picture and the reference picture to have different resolutions. With RPR, the difference in picture resolution can be taken into account during interpolation filtering in motion compensation by adjusting the interpolation filter phase and the reference block start position. The scaling ratio can be derived for both the horizontal and vertical directions based on the current picture and reference picture width and height and the current picture and reference picture consistency window.

[0044] In some examples, variables PicOutputWidthL and PicOutputHeightL corresponding to picture width and picture height may be derived for luma samples as follows:

[0045] PicOutputWidthL=pic_width_in_luma_samples-SubWidthC*(conf_win_right_offset+

[0046] conf_win_left_offset)

[0047] Equation (1)

[0048] PicOutputHeightL=pic_height_in_luma_samples-SubHeightC*(conf_win_bottom_offset+

[0049] conf_win_top_offset)

[0050] Equation (2)

[0051] PicOutputWidthL and PicOutputHeightL may be calculated for the current picture and the reference picture. In addition, pic_width_in_luma_samples and pic_height_in_luma_samples may refer to the width and height of a picture (e.g., a current picture, a reference picture, a decoded picture, an output picture, etc.) in units of luma samples, SubWidthC and SubHeightC may correspond to chroma samples associated with luma samples (e.g., when there is a corresponding chroma sample for each luma sample (e.g., as in a 4:4:4: format), both are equal to 1; when there is only one chroma sample for four luma samples (e.g., as in a 4:2:0 format), both are equal to 2, etc.), and conf_win_right_offset, conf_win_left_offset, conf_win_bottom_offset, conf_win_top_offset may refer to the left, right, bottom, and top offsets in a consistency window, and a consistency window may define the window size of a decoded picture output.

[0052] A scaling factor that may be used in some examples to scale the motion vector and / or ensure that the motion vector uses the correct coordinates for the picture may be derived as follows:

[0053] hori_scale_fp=((fRefWidth<<14)+(PicOutputWidthL>>1)) / PicOutputWidthL

[0054] Equation (3)

[0055] vert_scale_fp=((fRefHeight<<14)+(PicOutputHeightL>>1)) / PicOutputHeightL

[0056] Equation (4)

[0057] PicOutputWidthL and PicOutputHeightL may be picture width and height variables derived for a picture. In some examples, in equations (3) and (4), PicOutputWidthL and PicOutputHeightL may correspond to the width and height of the current picture. Additionally, the variable fRefWidth may be set equal to PicOutputWidthL of the reference picture expressed in luma samples, and the variable fRefHeight may be set equal to PicOutputHeightL of the reference picture expressed in luma samples.

[0058] The starting position (upper left corner) of the block in the reference picture, denoted as (xIntL, yIntL), and the filter phase or offset given in 1 / 16 sample units, denoted as (xFracL, yFracL), can be derived as follows. Given (refxSbL, refySbL) and (refxL, refyL) representing the luma position (given in 1 / 16 sample units) pointed to by the motion vector (refMvLX[0], refMvLX[1]), the variables refxSbL, refxL, refySbL, and refyL can be derived as follows:

[0059] refxSbL=((xSb<<4)+refMvLX[0])*hori_scale_fp

[0060] Equation (5)

[0061] refxL=((Sign(refxSb)*((Abs(refxSb)+128)>>8)+xL*((hori_scale_fp+8)>>4))+32)>>6

[0062] Equation (6)

[0063] refySbL=((ySb<<4)+refMvLX[1])*vert_scale_fp

[0064] Equation (7)

[0065] refyL=((Sign(refySb)*((Abs(refySb)+128)>>8)+yL*((vert_scale_fp+8)>>4))+32)>>6

[0066] Equation (8)

[0067] The variables xIntL, yIntL, xFracL, and yFracL can also be derived as follows:

[0068] xIntL=refxL>>4

[0069] Equation (9)

[0070] yIntL=refyL>>4

[0071] Equation (10)

[0072] xFracL=refxL&15

[0073] Equation (11)

[0074] yFracL=refyL&15

[0075] Equation (12)

[0076] Among them, (xSb, ySb) can specify the upper left corner sample of the current coding subblock relative to the upper left corner luma sample of the current picture, and refMvLX can be the motion vector for the reference picture list LX (X is 0 and 1).

[0077] Equations (1) to (12) are described above with respect to luma samples. However, the above equations (1) to (12) may be implemented and / or repeated for chroma color components. Thus, the above process for calculating and using picture width and height, scaling, and position, sample, and motion vector information may be repeated for chroma components.

[0078] When the scaling ratio is calculated based on the picture size and the consistency window offset as described above, various problems and challenges may arise. For example, the consistency window can define the picture output (e.g., for the picture to be displayed). In other words, the output picture can be cropped by applying the consistency window. When the scaling ratio is calculated using a consistency window (e.g., a scaling ratio between a cropped input picture and a cropped output picture can be derived, but the derived scaling ratio is applied to scale the decoded (e.g., before cropping) input picture to ensure that the size of the picture matches the decoded output picture size), the scaling ratio calculated for the cropped picture and the decoded picture may be different. As a result, the rescaling process may result in inaccurate and / or erroneous rescaling of the picture.

[0079] In another illustrative example, since the decoded picture size is a multiple of N samples (e.g., in VVC, N=8), the consistency window offset may be different for different picture sizes. In some cases, the consistency window offset may be larger for smaller decoded picture sizes, and smaller for larger picture sizes.

[0080] In another illustrative example, since the scaling ratio can be derived based on the cropped picture size, but is applied to the decoded picture size, the rescaled picture size may be different from the required and / or expected rescaled picture size due to the possible difference in the scaling ratio derived for the cropped picture and the decoded picture. In VVC, the output picture is the decoded picture after cropping. Therefore, in VVC, the resolution change may cause the output picture to have different sizes, which may cause problems and may be undesirable in some applications. For example, in some applications such as video conferencing applications, the user may not want to see the size of the picture change over time. Therefore, in this example, the different output picture sizes will be undesirable for the user.

[0081] The techniques described herein can solve the aforementioned problems associated with scaling derivation. In some aspects, one or more systems and methods for processing video data involve determining a scaling ratio and decoupling the scaling ratio determination from the picture output process. In some examples, the techniques described herein for determining a scaling ratio can prevent mismatches (and associated rescaling problems) between scaling ratios calculated for cropped pictures and decoded pictures, different consistency window offsets for different picture sizes, differences between the size of the rescaled picture generated using the scaling ratio and the size required and / or expected for the rescaled picture, and differences in output picture sizes when picture resolution changes occur. In some cases, the techniques described herein for determining a scaling ratio can prevent motion compensation problems, such as differences between motion vectors and corresponding picture coordinates.

[0082] The technology of this article will be described in the following disclosure as follows. The discussion begins with a description of example systems and techniques for encoding video data and calculating scaling, such as Figures 1 to 6 Then, what follows will be a description of an example method for calculating the scaling ratio and decoupling the scaling ratio calculation from the picture output process, such as Figure 7 The discussion ends with a description of example encoding and decoding devices suitable for encoding video data and determining scaling, such as Figure 8 and Fig. 9 The present disclosure now turns to Figure 1 .

[0083] Figure 11 is a block diagram showing an example of a system 100 including an encoding device 104 and a decoding device 112. The encoding device 104 may be part of a source device, and the decoding device 112 may be part of a receiving device. The source device and / or the receiving device may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television set, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the source device and the receiving device may include one or more wireless transceivers for wireless communication. The encoding techniques described herein are applicable to video encoding in various multimedia applications, including streaming video transmission (e.g., via the Internet), television broadcasting or transmission, encoding of digital videos for storage on data storage media, decoding of digital videos stored on data storage media, or other applications. As used herein, the term encoding may refer to encoding and / or decoding. In some examples, system 100 may support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, gaming, and / or video telephony.

[0084] The encoding device 104 (or encoder) can be used to encode video data using a video coding standard or protocol to generate an encoded video bitstream. Examples of video coding standards include ITU-T H.261, ISO / IEC MPEG-1 video (visual), ITU-T H.262 or ISO / IEC MPEG-2 video, ITU-T H.263, ISO / IEC MPEG-4 video, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC) (including its scalable video coding (SVC) and multi-view video coding (MVC) extensions), and high efficiency video coding (HEVC) or ITU-T H.265. There are various extensions to HEVC involving multi-layer video coding, including range and screen content coding extensions, 3D video coding (3D-HEVC) and multi-view extensions (MV-HEVC) and scalable extensions (SHVC). HEVC and its extensions have been developed by the Joint Collaborative Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG), and the Joint Collaborative Team on 3D Video Coding Extension Development (JCT-3V).

[0085] MPEG and ITU-T VCEG have also formed a joint exploration video team (JVET) to explore new coding tools for the next generation of video coding standards, which is named Versatile Video Coding (VVC). The reference software is called the VVC Test Model (VTM) (or JEM (Joint Exploration Model)). The goal of VVC is to provide significant improvements in compression performance compared to the existing HEVC standard, thereby assisting the deployment of higher quality video services and emerging applications (e.g., such as 360 ° omnidirectional immersive multimedia, high dynamic range (HDR) video, etc.). VP9, ​​Open Media Alliance (AOMedia) Video 1 (AV1) and Basic Video Coding (EVC) are other video coding standards to which the technology described herein can be applied.

[0086] The techniques described herein may be applied to any of the existing video codecs (e.g., High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), or other suitable existing video codecs), and / or may be efficient coding tools for any video coding standard being developed and / or future video coding standards (such as, for example, VVC and / or other video coding standards being developed or to be developed). For example, the examples described herein may be performed using video codecs such as VVC, HEVC, AVC, and / or their extensions. However, the techniques and systems described herein may also be applicable to other coding standards, such as MPEG, JPEG (or other coding standards for still images), VP9, ​​AV1, their extensions, or other suitable coding standards that are already available or not yet available or developed. Therefore, although the techniques and systems described herein may be described with reference to a particular video coding standard, it will be understood by those of ordinary skill in the art that the description should not be interpreted as being applicable only to that particular standard.

[0087] Reference Figure 1 , video source 102 may provide video data to encoding device 104. Video source 102 may be part of a source device, or may be part of a device other than a source device. Video source 102 may include a video capture device (e.g., a video camera, a camera phone, a video phone, etc.), a video archiving unit containing stored video, a video server or content provider providing video data, a video feed interface receiving video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of such sources, or any other suitable video source.

[0088] The video data from the video source 102 may include one or more input pictures or frames. A picture or frame is a still image, which in some cases is part of a video. In some examples, the data from the video source 102 may be a still image that is not part of a video. In HEVC, VVC, and other video coding specifications, a video sequence may include a series of pictures. A picture may include three sample arrays, which are represented as SL, SCb, and SCr. SL is a two-dimensional array of luma samples, SCb is a two-dimensional array of Cb chroma samples, and SCr is a two-dimensional array of Cr chroma samples. Chroma samples may also be referred to as "chroma" samples in this article. A pixel may refer to all three components (luminance and chroma samples) for a given position in an array of a picture. In other cases, a picture may be monochrome and may include only an array of luma samples, in which case the terms pixel and sample may be used interchangeably. With respect to the example techniques for referring to individual samples described herein for illustrative purposes, the same techniques may be applied to pixels (e.g., all three sample components for a given position in an array of a picture). With respect to the example techniques described herein for illustrative purposes referring to pixels (eg, all three sample components for a given position in an array of a picture), the same techniques may be applied to individual samples.

[0089] The encoder engine 106 (or encoder) of the encoding device 104 encodes the video data to generate a coded video bitstream. In some examples, the coded video bitstream (or "video bitstream" or "bitstream") is a series of one or more coded video sequences. A coded video sequence (CVS) includes a series of access units (AUs) starting from an AU having a random access point picture in a base layer and having certain attributes until the next AU having a random access point picture in a base layer and having certain attributes, and does not include the next AU. For example, certain attributes of the random access point picture that starts the CVS may include a RASL flag equal to 1 (e.g., NoRaslOutputFlag). Otherwise, the random access point picture (where the RASL flag is equal to 0) does not start the CVS. The access unit (AU) includes one or more coded pictures and control information corresponding to the coded pictures sharing the same output time. The coded slices of the picture are encapsulated as data units at the bitstream level, and the data units are called network abstraction layer (NAL) units. For example, an HEVC video bitstream may include one or more CVSs, which include NAL units. Each NAL unit in the NAL unit has a NAL unit header. In one example, the header is one byte for H.264 / AVC (except for multi-layer extensions) and two bytes for HEVC. The syntax elements in the NAL unit header use designated bits and are therefore visible to all kinds of systems and transport layers (such as transport streams, real-time transport (RTP) protocols, file formats, and others).

[0090] There are two types of NAL units in the HEVC standard, including video coding layer (VCL) NAL units and non-VCL NAL units. The VCL NAL unit includes a slice or slice segment (described below) of the coded picture data, and the non-VCL NAL unit includes control information related to one or more coded pictures. In some cases, the NAL unit may be referred to as a packet. The HEVC AU includes: a VCL NAL unit containing the coded picture data, and a non-VCL NAL unit corresponding to the coded picture data (if any).

[0091] A NAL unit may contain a sequence of bits that form an encoded representation of video data (e.g., a coded video bitstream, a CVS of a bitstream, etc.), such as an encoded representation of a picture in a video. The encoder engine 106 generates an encoded representation of a picture by partitioning each picture into a plurality of slices. A slice is independent of other slices, so that information in the slice can be encoded without relying on data from other slices within the same picture. A slice includes one or more slice segments, including an independent slice segment and, if present, one or more dependent slice segments that depend on previous slice segments.

[0092] In HEVC, the slice is then divided into coding tree blocks (CTBs) of luma samples and chroma samples. The CTB of luma samples and one or more CTBs of chroma samples together with the syntax for the samples are called coding tree units (CTUs). CTUs can also be called "tree blocks" or "largest coding units" (LCUs). CTUs are the basic processing units for HEVC encoding. CTUs can be split into multiple coding units (CUs) of different sizes. A CU contains arrays of luma and chroma samples called coding blocks (CBs).

[0093] Luma and chroma CBs can be further split into prediction blocks (PBs). A PB is a block of samples of a luminance component or a chroma component that uses the same motion parameters for inter-frame prediction or intra-block copy (IBC) prediction (when available or enabled for use). A luminance PB and one or more chroma PBs together with associated syntax form a prediction unit (PU). For inter-frame prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) is signaled in the bitstream for each PU and is used for inter-frame prediction of the luminance PB and one or more chroma PBs. Motion parameters may also be referred to as motion information. A CB may also be partitioned into one or more transform blocks (TBs). A TB represents a square block of samples of a color component, to which a residual transform (e.g., in some cases, the same two-dimensional transform) is applied to encode a prediction residual signal. A transform unit (TU) represents a TB of luminance and chroma samples and corresponding syntax elements. Transform coding is described in more detail below.

[0094] The size of the CU corresponds to the size of the coding mode and may be a square shape. For example, the size of the CU may be 8x 8 samples, 16x 16 samples, 32x 32 samples, 64x 64 samples, or any other appropriate size up to the size of the corresponding CTU. The phrase "N x N" is used herein to refer to the pixel size of the video block in the vertical and horizontal dimensions (e.g., 8 pixels x 8 pixels). The pixels in the block may be arranged in rows and columns. In some implementations, the block may not have the same number of pixels in the horizontal direction as in the vertical direction. The syntax data associated with the CU may describe, for example, the partitioning of the CU into one or more PUs. The partitioning mode may differ between whether the CU is encoded in an intra-frame prediction mode or an inter-frame prediction mode. The PU may be partitioned into a non-square shape. The syntax data associated with the CU may also, for example, describe the partitioning of the CU into one or more TUs according to the CTU. The TU may be square or non-square in shape.

[0095] According to the HEVC standard, a transform unit (TU) may be used to perform the transform. TUs may be different for different CUs. The size of the TU may be set based on the size of the PU within a given CU. The TU may have the same size as the PU or be smaller than the PU. In some examples, a quadtree structure called a residual quadtree (RQT) may be used to subdivide the residual samples corresponding to the CU into smaller units. The leaf nodes of the RQT may correspond to the TU. The pixel difference values ​​associated with the TU may be transformed to generate transform coefficients. The transform coefficients may then be quantized by the encoder engine 106.

[0096] Once the picture of the video data is partitioned into CUs, the encoder engine 106 predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). For each CU, the prediction mode can be signaled in the bitstream using syntax data. The prediction mode may include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction exploits the correlation between spatially adjacent samples within a picture. For example, using intra-frame prediction, each PU is predicted from adjacent image data in the same picture, using, for example, DC prediction to find an average value for the PU, using plane prediction to fit the plane surface to the PU, using directional prediction to infer from adjacent data, or using any other appropriate prediction type. Inter-frame prediction uses temporal correlation between pictures to derive motion compensated prediction for blocks of image samples. For example, using inter-frame prediction, each PU is predicted from image data in one or more reference pictures (before or after the current picture in output order) using motion compensated prediction. For example, a decision can be made at the CU level whether to use inter-picture prediction or intra-picture prediction to encode a picture area.

[0097] The encoder engine 106 and the decoder engine 116 (described in more detail below) can be configured to operate according to VVC. According to VVC, a video encoding device (coder) (such as an encoder engine 106 and / or a decoder engine 116) partitions a picture into multiple coding tree units (CTUs) (wherein the CTB of a luminance sample and one or more CTBs of a chrominance sample, and the syntax for the sample are collectively referred to as CTUs). The video encoding device can partition the CTU according to a tree structure (such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure). The QTBT structure removes the concept of multiple partition types, such as the distinction between CU, PU, ​​and TU in HEVC. The QTBT structure includes two levels, including: a first level partitioned according to a quadtree partition, and a second level partitioned according to a binary tree partition. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to coding units (CUs).

[0098] In the MTT segmentation structure, blocks can be segmented using quadtree segmentation, binary tree segmentation, and one or more types of ternary tree segmentation. Ternary tree segmentation is a segmentation in which a block is divided into three sub-blocks. In some examples, ternary tree segmentation divides a block into three sub-blocks without dividing the original block by the center. The segmentation types in MTT (e.g., quadtree, binary tree, and ternary tree) can be symmetric or asymmetric.

[0099] In some examples, the video encoding device may use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, the video encoding device may use two or more QTBT or MTT structures, such as one QTBT or MTT structure for the luma component and another QTBT or MTT structure for the two chroma components (or two QTBT and / or MTT structures for corresponding chroma components).

[0100] The video encoding device may be configured to use quadtree segmentation, QTBT segmentation, MTT segmentation, or other segmentation structures in accordance with HEVC. For the purpose of illustration, the description herein may refer to QTBT segmentation. However, it should be understood that the technology of the present disclosure may also be applied to video encoding devices configured to use quadtree segmentation or also use other types of segmentation.

[0101] In some examples, one or more slices of a picture are assigned a slice type. Slice types include I slices, P slices, and B slices. I slices (intra-frame, independently decodable) are slices of a picture that are encoded only by intra-frame prediction, and are therefore independently decodable because an I slice only requires data within a frame to predict any prediction unit or prediction block of the slice. P slices (unidirectional predicted frames) are slices of a picture that can be encoded using intra-frame prediction and unidirectional inter-frame prediction. Each prediction unit or prediction block within a P slice is encoded using intra-frame prediction or inter-frame prediction. When inter-frame prediction is applied, the prediction unit or prediction block is predicted only by one reference picture, and therefore the reference sample comes from only one reference region of one frame. B slices (bidirectional predicted frames) are slices of a picture that can be encoded using intra-frame prediction and inter-frame prediction (e.g., bidirectional prediction or unidirectional prediction). The prediction unit or prediction block of a B slice may be bidirectionally predicted from two reference pictures, where each picture contributes a reference region, and the sample sets of the two reference regions are weighted (e.g., with equal weights or with different weights) to generate a prediction signal for the bidirectional prediction block. As explained above, the slices of a picture are encoded independently. In some cases, a picture may be encoded as only one slice.

[0102] As mentioned above, intra-picture prediction of a picture exploits the correlation between spatially adjacent samples within the picture. There are multiple intra-frame prediction modes (also referred to as "intra-frame modes"). In some examples, intra-frame prediction of a luminance block includes 35 modes, including a planar mode, a DC mode, and 33 angular modes (e.g., a diagonal intra-frame prediction mode and an angular mode adjacent to a diagonal intra-frame prediction mode). The 35 intra-frame prediction modes are indexed as shown in Table 1 below. In other examples, more intra-frame modes may be defined, including prediction angles that may not yet be represented by the 33 angular modes. In other examples, the prediction angles associated with the angular modes may be different from those used in HEVC.

[0103]

[0104]

[0105] Table 1 - Specification of intra prediction modes and associated names

[0106] Inter-picture prediction uses temporal correlation between pictures in order to derive motion compensated prediction for blocks of image samples. Using a translational motion model, the position of a block in a previously decoded picture (reference picture) is represented by a motion vector (Δx, Δy), where Δx specifies the horizontal displacement of the reference block relative to the position of the current block, and Δy specifies the vertical displacement of the reference block relative to the position of the current block. In some cases, the motion vector (Δx, Δy) may be integer sample precision (also referred to as integer precision), in which case the motion vector points to an integer pixel grid (or integer pixel sampling grid) of a reference frame. In some cases, the motion vector (Δx, Δy) may have fractional sample precision (also referred to as fractional pixel precision or non-integer precision) to more accurately capture the motion of the underlying object without being limited to the integer pixel grid of the reference frame. The precision of the motion vector may be expressed by the quantization level of the motion vector. For example, the quantization level may be integer precision (e.g., 1 pixel) or fractional pixel precision (e.g., 1 / 4 pixel, 1 / 2 pixel, or other values ​​below a pixel). When the corresponding motion vector has fractional sample precision, interpolation is applied to the reference picture to derive the prediction signal. For example, samples available at integer positions may be filtered (e.g., using one or more interpolation filters) to estimate values ​​at fractional positions. A previously decoded reference picture is indicated by a reference index (refIdx) to a reference picture list. Motion vectors and reference indices may be referred to as motion parameters. Two types of inter-picture prediction may be performed, including uni-prediction and bi-prediction.

[0107] In the case of inter-frame prediction using bi-prediction (also known as bidirectional inter-frame prediction), two motion parameter sets (Δx0, y0, refIdx0 and Δx1, y1, refIdx1) are used to generate two motion compensated predictions (from the same reference picture or possibly from different reference pictures). For example, in the case of bi-prediction, each prediction block uses two motion compensated prediction signals and generates B prediction units. The two motion compensated predictions are then combined to obtain the final motion compensated prediction. For example, the two motion compensated predictions can be combined by averaging. In another example, weighted prediction can be used, in which case different weights can be applied to each motion compensated prediction. The reference pictures that can be used in bi-prediction are stored in two separate lists, represented as list 0 and list 1, respectively. The motion parameters can be derived using a motion estimation process at the encoder.

[0108] In the case of inter prediction using uni-prediction (also known as unidirectional inter prediction), a motion parameter set (Δx0, y0, refIdx0) is used to generate motion compensated prediction from a reference picture. For example, in the case of uni-prediction, each prediction block uses at most one motion compensated prediction signal and generates P prediction units.

[0109] The PU may include data related to the prediction process (e.g., motion parameters or other appropriate data). For example, when the PU is encoded using intra prediction, the PU may include data describing the intra prediction mode for the PU. As another example, when the PU is encoded using inter prediction, the PU may include data defining a motion vector for the PU. The data defining the motion vector for the PU may describe, for example, the horizontal component of the motion vector (Δx), the vertical component of the motion vector (Δy), the resolution for the motion vector (e.g., integer precision, quarter-pixel precision, or eighth-pixel precision), the reference picture to which the motion vector points, a reference index, a reference picture list for the motion vector (e.g., list 0, list 1, or list C), or any combination thereof.

[0110] After performing prediction using intra prediction and / or inter prediction, the encoding device 104 can then perform transformation and quantization. For example, after prediction, the encoder engine 106 can calculate the residual value corresponding to the PU. The residual value may include the pixel difference between the current pixel block (PU) being encoded and the prediction block (e.g., the predicted version of the current block) used to predict the current block. For example, after generating a prediction block (e.g., performing inter prediction or intra prediction), the encoder engine 106 can generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values, which quantizes the difference between the pixel values ​​of the current block and the pixel values ​​of the prediction block. In some examples, the residual block can be represented by a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In such an example, the residual block is a two-dimensional representation of pixel values.

[0111] Any residual data that may remain after performing the prediction is transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other appropriate transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., sizes 32x 32, 16x 16, 8x 8, 4x 4, or other appropriate sizes) may be applied to the residual data in each CU. In some embodiments, a TU may be used for the transform and quantization process implemented by the encoder engine 106. A given CU with one or more PUs may also include one or more TUs. As described in further detail below, the residual values ​​may be transformed into transform coefficients using a block transform, and then quantized and scanned using the TU to produce serialized transform coefficients for entropy coding.

[0112] In some embodiments, after intra-prediction or inter-prediction encoding using the PU of the CU, the encoder engine 106 may calculate residual data for the TU of the CU. The PU may include pixel data in the spatial domain (or pixel domain). The TU may include coefficients in the transform domain after applying the block transform. As described above, the residual data may correspond to the pixel difference between the pixel of the unencoded picture and the predicted value corresponding to the PU. The encoder engine 106 may form a TU including the residual data for the CU, and then may transform the TU to generate transform coefficients for the CU.

[0113] The encoder engine 106 may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient having an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

[0114] Once quantization is performed, the encoded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction mode, motion vectors, block vectors, etc.), segmentation information, and any other appropriate data (such as other syntax data). The different elements of the encoded video bitstream can then be entropy encoded by the encoder engine 106. In some examples, the encoder engine 106 can scan the quantized transform coefficients using a predefined scan order to generate a serialized vector that can be entropy encoded. In some examples, the encoder engine 106 can perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoder engine 106 can entropy encode the vector. For example, the encoder engine 106 can use context adaptive variable length coding, context adaptive binary arithmetic coding, grammar-based context adaptive binary arithmetic coding, probability interval segmentation entropy coding, or another appropriate entropy coding technique.

[0115] As mentioned above, the HEVC bitstream includes a set of NAL units, which include VCL NAL units and non-VCL NAL units. The VCL NAL unit includes coded picture data that forms a coded video bitstream. For example, there is a bit sequence that forms a coded video bitstream in the VCL NAL unit. In addition to other information, the non-VCL NAL unit may also contain a parameter set with high-level information related to the coded video bitstream. For example, the parameter set may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). Examples of the goals of the parameter set include bit rate efficiency, error resilience, and providing a system layer interface. Each slice references a single valid PPS, SPS, and VPS to access information that the decoding device 112 can use to decode the slice. An identifier (ID) can be encoded for each parameter set, including a VPS ID, an SPS ID, and a PPS ID. The SPS includes an SPS ID and a VPS ID. The PPS includes a PPS ID and an SPS ID. Each slice header includes a PPS ID. Using these IDs, a valid parameter set for a given slice can be identified.

[0116] The PPS includes information applicable to all slices in a given picture. Therefore, all slices in a picture refer to the same PPS. Slices in different pictures may also refer to the same PPS. The SPS includes information applicable to all pictures in the same coded video sequence (CVS) or bitstream. As previously described, a coded video sequence is a series of access units (AUs) starting with a random access point picture (e.g., an instant decoding reference (IDR) picture or a broken link access (BLA) picture or other appropriate random access point picture) in the base layer and having certain properties (as described above) until the next AU (or the end of the bitstream) with a random access point picture in the base layer and having certain properties and not including the next AU. The information in the SPS may not change between pictures in the coded video sequence. The pictures in the coded video sequence may use the same SPS. The VPS includes information applicable to all layers in the coded video sequence or bitstream. The VPS includes a syntax structure having syntax elements applicable to the entire coded video sequence. In some embodiments, the VPS, SPS, or PPS may be sent in-band with the coded bitstream. In some embodiments, the VPS, SPS, or PPS may be sent out-of-band in a different transport than the NAL units containing the encoded video data.

[0117] The video bitstream may also include a supplemental enhancement information (SEI) message. For example, a SEI NAL unit may be part of a video bitstream. In some cases, the SEI message may contain information that is not needed for the decoding process. For example, the information in the SEI message may not be necessary for the decoder to decode the video picture of the bitstream, but the decoder may use the information to improve the display or processing of the picture (e.g., the decoded output). The information in the SEI message may be embedded metadata. In an illustrative example, the decoder side entity may use the information in the SEI message to improve the visibility of the content. In some cases, certain application standards may force the presence of such SEI messages in the bitstream, so that quality improvements can be brought to all devices that comply with the application standard (e.g., carrying frame encapsulation SEI messages for frame-compatible planar stereoscopic 3DTV video formats, wherein SEI messages are carried for each frame of the video, processing recovery point SEI messages, and using pan scanning in DVB to scan rectangular SEI messages).

[0118] The output 110 of the encoding device 104 can send the NAL units constituting the encoded video bitstream data to the decoding device 112 of the receiving device on the communication link 120. The input 114 of the decoding device 112 can receive the NAL unit. The communication link 120 may include a channel provided by a wireless network, a wired network, or a combination of a wired network and a wireless network. The wireless network may include any wireless interface or a combination of wireless interfaces, and may include any appropriate wireless network (e.g., the Internet or other wide area network, a packet-based network, WiFiTM, radio frequency (RF), UWB, WiFi direct connection, cellular, long-term evolution (LTE), WiMaxTM, etc.). The wired network may include any wired interface (e.g., optical fiber, Ethernet, power line Ethernet, coaxial cable Ethernet, digital signal line (DSL), etc.). Various devices may be used to implement wired and / or wireless networks, such as base stations, routers, access points, bridges, gateways, switches, etc. The encoded video bitstream data may be modulated according to a communication standard such as a wireless communication protocol and sent to a receiving device.

[0119] In some examples, the encoding device 104 may store the encoded video bitstream data in the storage unit 108. The output 110 may obtain the encoded video bitstream data from the encoder engine 106 or from the storage unit 108. The storage unit 108 may include any of a variety of distributed or locally accessed data storage media. For example, the storage unit 108 may include a hard drive, a storage disk, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing the encoded video data. The storage unit 108 may also include a decoded picture buffer (DPB) for storing reference pictures for use in inter-frame prediction. In another example, the storage unit 108 may correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device. In such a case, the receiving device including the decoding device 112 can access the stored video data from the storage device via streaming or downloading. The file server can be any type of server capable of storing the encoded video data and sending the encoded video data to the receiving device. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The receiving device can access the encoded video data through any standard data connection (including an Internet connection). This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of the two suitable for accessing the encoded video data stored on a file server. The transmission of the encoded video data from the storage unit 108 may be a streaming transmission, a download transmission, or a combination thereof.

[0120] The input 114 of the decoding device 112 receives the encoded video bitstream data, and the video bitstream data may be provided to the decoder engine 116, or to the storage unit 118 for later use by the decoder engine 116. For example, the storage unit 118 may include a DPB for storing reference pictures for use in inter-frame prediction. A receiving device including the decoding device 112 may receive the encoded video data to be decoded via the storage unit 108. The encoded video data may be modulated according to a communication standard such as a wireless communication protocol and sent to the receiving device. The communication medium for sending the encoded video data may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other device that may be used to facilitate communication from a source device to a receiving device.

[0121] The decoder engine 116 can decode the coded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more coded video sequences that constitute the coded video data. The decoder engine 116 can then rescale the coded video bitstream data and perform an inverse transform on it. The residual data is then passed to the prediction stage of the decoder engine 116. The decoder engine 116 then predicts a pixel block (e.g., a PU). In some examples, the prediction is added to the output of the inverse transform (residual data).

[0122] The decoding device 112 may output the decoded video to a video destination device 122, which may include a display or other output device for displaying the decoded video data to a consumer of the content. In some aspects, the video destination device 122 may be part of a receiving device that includes the decoding device 112. In some aspects, the video destination device 122 may be part of a separate device distinct from the receiving device.

[0123] In some embodiments, the video encoding device 104 and / or the video decoding device 112 may be integrated with the audio encoding device and the audio decoding device, respectively. The video encoding device 104 and / or the video decoding device 112 may also include other hardware or software necessary to implement the above-mentioned encoding technology, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. The video encoding device 104 and the video decoding device 112 may be integrated as part of a combined encoder / decoder (codec) in the corresponding device. The following reference Figure 8 An example of specific details of the encoding device 104 is described below. Fig. 9 An example of specific details of the decoding device 112 is described.

[0124] Figure 1 The example system shown in is only an illustrative example that can be used in this article. The technology for processing video data using the technology described in this article can be performed by any digital video encoding and / or decoding device. Although in general, the technology of the present disclosure is performed by a video encoding device or a video decoding device, the technology can also be performed by a combined video encoder / decoder commonly referred to as a "CODEC". In addition, the technology of the present disclosure can also be performed by a video preprocessor. The source device and the receiving device are only examples of such encoding devices, wherein the source device generates encoded video data for transmission to the receiving device. In some examples, the source device and the receiving device can operate in a substantially symmetrical manner so that each of these devices includes a video encoding and decoding component. Therefore, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting or video telephony.

[0125] Extensions to the HEVC standard include a multi-view video coding extension called MV-HEVC and a scalable video coding extension called SHVC. The MV-HEVC and SHVC extensions share the concept of layered coding, in which different layers are included in the coded video bitstream. Each layer in the coded video sequence is addressed by a unique layer identifier (ID). A layer ID may be present in the header of a NAL unit to identify the layer to which the NAL unit is associated. In MV-HEVC, different layers may represent different views of the same scene in a video bitstream. In SHVC, different scalable layers are provided to represent video bitstreams at different spatial resolutions (or picture resolutions) or different reconstruction fidelity. Scalable layers may include a base layer (where layer ID = 0) and one or more enhancement layers (where layer ID = 1, 2, ... n). The base layer may conform to the profile of the first version of HEVC and represent the lowest available layer in the bitstream. Compared to the base layer, the enhancement layer has increased spatial resolution, temporal resolution or frame rate and / or reconstruction fidelity (or quality). Enhancement layers are organized hierarchically and may or may not be dependent on lower layers. In some examples, different layers may be encoded using a single standard codec (e.g., all layers may be encoded using HEVC, SHVC, or other coding standards). In some examples, different layers may be encoded using a multi-standard codec. For example, a base layer may be encoded using AVC, while one or more enhancement layers may be encoded using SHVC and / or the MV-HEVC extension of the HEVC standard.

[0126] Typically, a layer includes a set of VCL NAL units and a corresponding set of non-VCL NAL units. A specific layer ID value is assigned to the NAL unit. Layers can be hierarchical in the sense that a layer can depend on a lower layer. A layer set refers to a self-contained set of layers represented in a bitstream, which means that the layers in the layer set can depend on other layers in the layer set during the decoding process, but do not depend on any other layer for decoding. Therefore, each layer in the layer set can form an independent bitstream that can represent the video content. A group of layers in a layer set can be obtained from another bitstream through the operation of a sub-bitstream extraction process. When the decoder wishes to operate according to certain parameters, the layer set can correspond to the group of layers to be decoded.

[0127] As described above, for each block, a set of motion information (also referred to herein as motion parameters) may be available. The motion information set contains motion information for a forward prediction direction and a backward prediction direction. The forward prediction direction and the backward prediction direction may be two prediction directions of a bidirectional prediction mode, in which case the terms "forward" and "backward" do not necessarily have a geometric meaning. Instead, "forward" and "backward" correspond to reference picture list 0 (RefPicList0 or L0) and reference picture list 1 (RefPicList1 or L1) of the current picture. In some examples, when only one reference picture list is available for a picture or slice, only RefPicList0 is available, and the motion information for each block of the slice is always forward.

[0128] In some examples, a motion vector and its reference index are used in the encoding process (e.g., motion compensation). Such a motion vector with an associated reference index is represented as a single prediction set of motion information. For each prediction direction, the motion information may contain a reference index and a motion vector. In some cases, for simplicity, the motion vector itself may be referenced in a manner assuming that it has an associated reference index. The reference index is used to identify a reference picture in the current reference picture list (RefPicList0 or RefPicList1). The motion vector has a horizontal component and a vertical component, which provides an offset from a coordinate position in the current picture to a coordinate position in the reference picture identified by the reference index. For example, a reference index may indicate a specific reference picture that should be used for a block in the current picture, and a motion vector may indicate where the best matching block in the reference picture (the block that best matches the current block) is located in the reference picture.

[0129] The picture order count (POC) can be used in video coding standards to identify the display order of pictures. Although there are cases where two pictures in a coded video sequence may have the same POC value, it does not usually occur within a coded video sequence. When there are multiple coded video sequences in the bitstream, pictures with the same POC value may be closer to each other in decoding order. The POC value of a picture can be used for reference picture list construction (such as reference picture set derivation in HEVC) and motion vector scaling.

[0130] In H.264 / AVC, each inter-frame macroblock (MB) can be partitioned in four different ways, including: one 16x16 MB partition; two 16x8 MB partitions; two 8x16 MB partitions; and four 8x8 MB partitions. Different MB partitions in one MB can have different reference index values ​​(RefPicList0 or RefPicList1) for each direction. In some cases, when the MB is not partitioned into four 8x8 MB partitions, it can have only one motion vector for each MB partition in each direction. In some cases, when the MB is partitioned into four 8x8 MB partitions, each 8x8 MB partition can be further partitioned into sub-blocks, in which case each sub-block can have a different motion vector in each direction. In some examples, there are four different ways to get a sub-block from an 8x8 MB partition: one 8x8 sub-block; two 8x4 sub-blocks; two 4x8 sub-blocks; and four 4x4 sub-blocks. Each sub-block can have a different motion vector in each direction. Therefore, the motion vector can exist at a level equal to or higher than the sub-block.

[0131] In AVC, for skip or direct mode in B slices, temporal direct mode can be implemented at the MB or MB partition level. For each MB partition, the motion vector is derived using the motion vector of the block co-located with the current MB partition in the RefPicList1[0] of the current block. Each motion vector in the co-located block can be scaled based on the POC distance.

[0132] In AVC, a spatial direct mode can also be implemented. For example, in AVC, the direct mode can also predict motion information based on spatial neighbors.

[0133] As mentioned above, in HEVC, the largest coding unit in a slice is called a coding tree block (CTB). The CTB contains a quadtree whose nodes are coding units. In the HEVC main profile, the size of the CTB can range from 16x16 to 64x64. In some cases, 8x8 CTBs can be supported. The coding unit (CU) can have the same size as the CTB and be as small as 8x8. In some cases, a mode can be used to encode each coding unit. When the CU is inter-coded, the CU can be further split into 2 or 4 prediction units (PUs), or when further splitting is not applicable, the CU can become only one PU. When there are two PUs in a CU, they can be half-sized rectangles, or two rectangles with 1 / 4 or 3 / 4 the size of the CU.

[0134] When the CU is inter-coded, there is one motion information set for each PU. In addition, each PU can be encoded using a unique inter prediction mode to derive a motion information set.

[0135] For motion prediction in HEVC, there are two inter prediction modes for prediction units (PUs), including merge mode and advanced motion vector prediction (AMVP) mode. Special cases that are considered merges are skipped. In AMVP or merge mode, a motion vector (MV) candidate list for multiple motion vector predictors can be maintained. In merge mode, the motion vector of the current PU and the reference index are generated by extracting a candidate from the MV candidate list. In some examples, one or more scaling window offsets can be included in the MV candidate list along with the stored motion vectors.

[0136] In an example where an MV candidate list is used for motion prediction of a block, the MV candidate list may be constructed by an encoding device and a decoding device, respectively. For example, the MV candidate list may be generated by an encoding device when encoding a block, and may be generated by a decoding device when decoding a block. Information related to motion information candidates in the MV candidate list (e.g., information related to one or more motion vectors, information related to one or more LIC flags that may be stored in the MV candidate list in some cases, and / or other information) may be signaled between the encoding device and the decoding device. For example, in merge mode, an index value for the stored motion information candidate may be signaled from the encoding device to the decoding device (e.g., in a syntax structure such as a picture parameter set (PPS), a sequence parameter set (SPS), a video parameter set (VPS), a slice header, a supplemental enhancement information (SEI) message sent in a video bitstream or separately from a video bitstream, and / or other signaling). The decoding device may construct an MV candidate list, and use the signaled reference or index to obtain one or more motion information candidates from the constructed MV candidate list for motion compensated prediction. For example, the decoding device 112 may construct an MV candidate list and use the motion vector from the index position (and the LIC flag in some cases) to perform motion prediction on the block. In the case of AMVP mode, in addition to the reference or index, a difference or residual value may also be signaled as a delta. For example, for AMVP mode, the decoding device may construct one or more MV candidate lists and apply the delta value to one or more motion information candidates obtained using the signaled index value when performing motion compensated prediction on the block.

[0137] In some examples, the MV candidate list contains up to five candidates for merge mode and two candidates for AMVP mode. In other examples, different numbers of candidates may be included in the MV candidate list for merge mode and / or AMVP mode. The merge candidate may include a motion information set. For example, a motion information set may include motion vectors corresponding to two reference picture lists (list 0 and list 1) and a reference index. If the merge candidate is identified by a merge index, the reference picture is used for prediction of the current block, and the associated motion vector is determined. However, in AMVP mode, for each potential prediction direction from list 0 or list 1, the reference index needs to be explicitly signaled together with the MVP index to the MV candidate list, because the AMVP candidate only contains motion vectors. In AMVP mode, the predicted motion vector can be further refined.

[0138] As seen above, the merge candidate corresponds to a complete set of motion information, while the AMVP candidate contains only one motion vector and reference index for a specific prediction direction.Candidates for both modes can be derived similarly from the same spatial and temporal neighboring blocks.

[0139] In some examples, merge mode allows an inter-predicted PU to inherit the same one or more motion vectors, prediction direction, and one or more reference picture indexes from an inter-predicted PU that includes a motion data position selected from a set of spatially adjacent motion data positions and one of two temporally co-located motion data positions. For AMVP mode, one or more motion vectors of a PU may be predictively encoded relative to one or more motion vector predictors (MVPs) from an AMVP candidate list constructed by an encoder and / or decoder. In some cases, for unidirectional inter prediction of a PU, an encoder and / or decoder may generate a single AMVP candidate list. In some cases, for bidirectional prediction of a PU, an encoder and / or decoder may generate two AMVP candidate lists, one using motion data of spatially and temporally adjacent PUs in a forward prediction direction, and one using motion data of spatially and temporally adjacent PUs in a backward prediction direction.

[0140] Candidates for both modes can be derived from spatial and / or temporal neighboring blocks. Figure 2A and Figure 2B Included is a conceptual diagram showing spatial neighbor candidates. Figure 2A Spatial neighboring motion vector (MV) candidates for merge mode are shown. Figure 2BSpatial neighboring motion vector (MV) candidates for AMVP mode are shown. Although the method of generating candidates according to blocks is different for merge and AMVP modes, spatial MV candidates are derived according to neighboring blocks for a specific PU (PU0).

[0141] In merge mode, the encoder and / or decoder may form a merge candidate list by considering merge candidates from various motion data locations. Figure 2A As shown, about Figure 2A Up to four spatial MV candidates can be derived from the spatially adjacent motion data positions indicated by numbers 0-4 in the merge candidate list. In the merge candidate list, the MV candidates can be sorted in the order indicated by numbers 0-4. For example, the positions and order may include: left position (0), top position (1), top right position (2), bottom left position (3), and top left position (4).

[0142] exist Figure 2B In the AVMP mode shown in , the neighboring blocks can be divided into two groups: the left group including blocks 0 and 1, and the upper group including blocks 2, 3 and 4. For each group, potential candidates in the neighboring blocks that reference the same reference picture as the reference picture indicated by the signaled reference index have the highest priority to be selected to form the final candidate for the group. It is possible that all neighboring blocks do not contain motion vectors pointing to the same reference picture. Therefore, if such a candidate cannot be found, the first available candidate can be scaled to form the final candidate, so that the temporal distance difference can be compensated.

[0143] Figure 3A and Figure 3B A conceptual diagram showing temporal motion vector prediction is included. A temporal motion vector predictor (TMVP) candidate (if enabled and available) is added to the MV candidate list after the spatial motion vector candidate. The process of motion vector derivation for the TMVP candidate is the same for both merge and AMVP modes. However, in some cases, the target reference index for the TMVP candidate in merge mode can be set to zero or can be derived based on the reference index of the neighboring block.

[0144] The main block position for TMVP candidate derivation is the lower right block outside the co-located PU (such as Figure 3AThe motion vector for the TMVP candidate is derived from the co-located PU of the co-located picture indicated at the slice level. Similar to the temporal direct mode in AVC, motion vector scaling can be performed on the motion vector of the TMVP candidate, which can be performed to compensate for the distance difference.

[0145] Other aspects of motion prediction are covered in the HEVC standard. For example, several other aspects of merge mode and AMVP mode are covered. One aspect includes motion vector scaling. Regarding motion vector scaling, it is assumed that the value of the motion vector is proportional to the distance of the picture in the presentation time. The motion vector associates two pictures, the reference picture and the picture containing the motion vector (i.e., the containing picture). When a motion vector is used to predict other motion vectors, the distance between the containing picture and the reference picture is calculated based on the picture order count (POC) value.

[0146] For a motion vector to be predicted, both its associated containing picture and reference picture can be different. Therefore, a new distance is calculated (based on POC). Moreover, the motion vector is scaled based on these two POC distances. For spatial neighboring candidates, the containing picture for the two motion vectors is the same, while the reference picture is different. In HEVC, motion vector scaling applies to both TMVP and AMVP for both spatial and temporal neighboring candidates.

[0147] Another aspect of motion prediction includes artificial motion vector candidate generation. For example, if the motion vector candidate list is incomplete, an artificial motion vector candidate is generated and inserted at the end of the list until all lists are obtained. In merge mode, there are two types of artificial MV candidates: combined candidates derived only for B slices; and zero candidates used only for AMVP (if the first type does not provide enough artificial candidates). For each pair of candidates that are already in the candidate list and have the necessary motion information, a bidirectional combined motion vector candidate is derived by combining the motion vector of the first candidate with reference to the picture in list 0 and the motion vector of the second candidate with reference to the picture in list 1.

[0148] In some implementations, a pruning process may be performed when a new candidate is added or inserted into the MV candidate list. For example, in some cases, MV candidates from different blocks may include the same information. In such a case, storing repeated motion information of multiple MV candidates in the MV candidate list may result in redundancy in the MV candidate list and reduced efficiency. In some examples, the pruning process may eliminate or minimize redundancy in the MV candidate list. For example, the pruning process may include comparing the potential MV candidate to be added to the MV candidate list with the MV candidate already stored in the MV candidate list. In an illustrative example, the horizontal displacement (Δx) and vertical displacement (Δy) of the stored motion vector (indicating the position of the reference block relative to the position of the current block) may be compared with the horizontal displacement (Δx) and vertical displacement (Δy) of the potential candidate. If the comparison reveals that the motion vector of the potential candidate does not match any of the one or more stored motion vectors, the potential candidate is not considered to be a candidate to be pruned and may be added to the MV candidate list. If a match is found based on the comparison, the potential MV candidate is not added to the MV candidate list, thereby avoiding insertion of the same candidate. In some cases, to reduce complexity, only a limited number of comparisons are performed during the pruning process, rather than comparing each potential MV candidate with all existing candidates.

[0149] In some coding schemes such as HEVC, weighted prediction (WP) is supported, in which case a scaling factor (denoted by a), a shift amount (denoted by s), and an offset (denoted by b) are used in motion compensation. Assuming that the pixel value at position (x, y) of the reference picture is p(x, y), then p'(x, y) = ((a*p(x, y) + (1 << (s-1))) >> s) + b is used as the prediction value in motion compensation instead of p(x, y).

[0150] When WP is enabled, for each reference picture of the current slice, a flag is signaled to indicate whether WP is applicable to the reference picture. If WP is applicable to a reference picture, a WP parameter set (i.e., a, s, and b) is sent to the decoder, and the WP parameter set is used for motion compensation from the reference picture. In some examples, in order to flexibly turn on / off WP for luma and chroma components, WP flags and WP parameters for luma and chroma components are signaled separately. In WP, one same WP parameter set can be used for all pixels in one reference picture.

[0151] Figure 4Ais a diagram showing an example of neighboring reconstructed samples of a current block 402 and neighboring samples of a reference block 404 for unidirectional inter prediction. A motion vector MV may be encoded for the current block 402, wherein the MV may include a reference index to a reference picture list and / or other motion information for identifying the reference block 404. For example, the MV may include a horizontal component and a vertical component that provide an offset from a coordinate position in the current picture to a coordinate in the reference picture identified by the reference index. Figure 4B is a diagram showing an example of neighboring reconstructed samples of a current block 422 and neighboring samples of a first reference block 424 and a second reference block 426 for bidirectional inter prediction. In this case, two motion vectors MV0 and MV1 may be encoded for the current block 422 to identify the first reference block 424 and the second reference block 426, respectively.

[0152] In some cases, changes to the reference picture and / or the current picture may result in different picture sizes and / or resolutions. For example, certain tasks (such as reference picture resampling (RPR)) may result in the current picture and the reference picture having different resolutions. However, the difference between the current picture and the reference picture size and / or resolution may cause various problems, as explained previously. For example, the difference between the current picture and the reference picture size and / or resolution may cause motion compensation problems, such as the difference between the motion vector and the corresponding picture coordinates. To address these and other problems, the techniques for determining the scaling ratio described herein can decouple the scaling ratio calculation from the picture output process and can prevent motion compensation problems such as the difference between the motion vector and the corresponding picture coordinates.

[0153] Figure 5 is a diagram showing an example scaling calculation decoupled from the image output process. In some cases, Figure 5 The example scaling calculations in can prevent motion compensation problems, mismatches between scalings calculated for cropped pictures and decoded pictures (and associated rescaling problems), different consistency window offsets for different picture sizes, differences between the size of a rescaled picture generated using a scaling and the required and / or expected size for the rescaled picture, differences in output picture sizes when picture resolution changes occur, and other problems associated with scaling or encoding video data.

[0154] Furthermore, by decoupling the scaling calculation from the picture output process, the consistency window (e.g., 542) used for the picture output process can be decoupled from the scaling window (e.g., 502) used for scaling calculation. Decoupling the consistency window from the scaling calculation can allow the consistency window to be modified by rewriting the PPS in the bitstream to display pictures of different sizes without further modifying the bitstream (e.g., at the block level), which may not be possible if the scaling depends on the consistency window (e.g., is calculated based on the consistency window), because changing the consistency window may cause the scaling to change, which may affect the prediction derivation in the block.

[0155] exist Figure 5 In the embodiment of the present invention, the scaling ratio calculation 500 is decoupled from the picture output process 540. The scaling ratio calculation 500 can be performed for an input picture 504 (e.g., a current picture, a reference picture, etc.) based on the scaling window 502. In some examples, the encoding device (e.g., 104) can signal the scaling window 502. The scaling window 502 can identify a window offset 506 for calculating the scaling ratio for the input picture 504. For example, the scaling window 502 can define a top offset, a bottom offset, a left offset, and / or a right offset, which can be added to the height 508 and the width 510 of the input picture 504 to generate a scaling ratio output 530 indicating the scaling ratio calculated for the input picture 504. Therefore, the scaling ratio for the input picture 504 (e.g., the scaling ratio output 530) can be calculated using the input picture 504 and the window offset 506 from the scaling window 502.

[0156] In some cases, the scaling ratio output 530 may include vertical (e.g., height) and horizontal (e.g., width) scaling ratios. For example, the scaling ratio output 530 may indicate a vertical scaling ratio for an input picture (e.g., 504) and a horizontal scaling ratio for an input picture. In addition, in some examples, a scaling window (e.g., 502) may be signaled for each picture (e.g., for each current picture and each reference picture). As previously explained, a window offset (e.g., 506) from the scaling window may be used to derive a scaling ratio (e.g., 530).

[0157] In the picture output process 540, the size of the input picture 504 can be modified using a consistency window 542. In some examples, the encoding device (e.g., 104) can signal the consistency window 542. The consistency window 542 can identify a window offset 546 for changing (e.g., scaling, cropping, etc.) the size of the input picture 504. For example, the consistency window 542 can define the following top offset, bottom offset, left offset, and / or right offset, which are to be applied and / or implemented to modify the size of the input picture 504 from the top, bottom, left, and / or right, respectively, based on the top offset, bottom offset, left offset, and / or right offset. The window offset 546 can be the same or different from the window offset 506 in the scaling window 502. In some examples, the window offset 546 is the same as the window offset 506 in the scaling window 502.

[0158] Window offset 546 from consistency window 542 can be used to resize (e.g., scale, crop, etc.) input picture 504 to a specific height 548 and width 550. Thus, input picture 504 and window offset 546 from consistency window 542 can be used to generate output picture 560 having specific height 548 and width 550. Specific height 548 and width 550 can be the same as or different from specific height 508 and width 510. In some examples, specific height 548 and width 550 are the same as specific height 508 and width 510 used with scale window 502 in scale ratio calculation 500.

[0159] Figure 6 604 and the reference picture 610. In this example, the zoom window 602 can be used to calculate the zoom ratio for the current picture 604, and the zoom window 608 can be used to calculate the zoom ratio for the reference picture 610. The zoom windows 602 and 608 can identify the top offset, bottom offset, left offset, and / or right offset used to calculate the zoom ratio for the current picture 604 and the reference picture 610, respectively. For example, the zoom window 602 can include a top offset, a bottom offset, a left offset, and a right offset, which can be added to the height and width of the current picture 604 (e.g., from the top, bottom, left, and right of the current picture 604) to determine the zoom ratio for the current picture 604. Similarly, the zoom window 608 can include a top offset, a bottom offset, a left offset, and a right offset, which can be added to the width and width of the reference picture 610 (e.g., from the top, bottom, left, and right of the reference picture 610) to determine the zoom ratio for the reference picture 610.

[0160] Thus, the scaling windows 602, 609 can be used to calculate a scaling output 620 for the current picture 604 and the reference picture 610. The scaling output 620 can include one or more scaling ratios. In some examples, the scaling output 620 can include vertical (e.g., height) and horizontal (e.g., width) scaling ratios for the current picture 604 and / or the reference picture 610. In some cases, the scaling output 620 can indicate a vertical scaling ratio for the current picture 604 and the reference picture 610, and a horizontal scaling ratio for the current picture 604 and the reference picture 610.

[0161] In some cases, the scaling ratio (e.g., 530, 620) may be calculated using equations (13) to (16) and / or equations (25) to (28) below. For example, in some illustrative implementations, variables PicOutputWidthL and PicOutputHeightL for luma (e.g., the width and height of the modified current or reference picture) may be derived as follows:

[0162] PicOutputWidthL=pic_width_in_luma_samples-SubWidthC*(scaling_win_right_offset+

[0163] scaling_win_left_offset)

[0164] Equation (13)

[0165] PicOutputHeightL=pic_height_in_luma_samples-SubHeightC*(scaling_win_bottom_offset+

[0166] scaling_win_top_offset)

[0167] Equation (14)

[0168] PicOutputWidthL and PicOutputHeightL may be calculated for a current picture (e.g., 604) and a reference picture (e.g., 610). In some cases, PicOutputWidthL and PicOutputHeightL may be calculated for a current picture (e.g., 604) and may represent the width and height of a modified current picture (e.g., 606), and may also be calculated for a reference picture (e.g., 610), in which case they may represent the width and height of a modified reference picture (e.g., 612). In addition, pic_width_in_luma_samples and pic_height_in_luma_samples can refer to the width and height of a picture (e.g., a current picture, a reference picture, a decoded picture, an output picture, etc.) in units of luma samples, SubWidthC and SubHeightC can correspond to chroma samples related to luma samples (e.g., when there is a corresponding chroma sample for each luma sample (e.g., as in a 4:4:4: format), both are equal to 1; when there is only one chroma sample for 4 luma samples (e.g., as in a 4:2:0 format), both are equal to 2, and so on), and scale_win_right_offset, scaling_win_left_offset, scaling_win_bottom_offset, scaling_win_top_offset can correspond to the left offset, right offset, bottom offset, and top offset (e.g., 506) in the scaling window (e.g., 502, 602, 608).

[0169] The scaling ratio (e.g., 530, 620) can be derived as follows:

[0170] hori_scale_fp=((fRefWidth<<14)+(PicOutputWidthL>>1)) / PicOutputWidth

[0171] Equation (15)

[0172] vert_scale_fp=((fRefHeight<<14)+(PicOutputHeightL>>1)) / PicOutputHeightL

[0173] Equation (16)

[0174] In equations (15) and (16), PicOutputWidthL and PicOutputHeightL may be picture width and height variables derived for a picture. For example, PicOutputWidthL and PicOutputHeightL may be width and height variables of a modified current picture (e.g., 606). In some cases, the variable fRefWidth may be set equal to PicOutputWidthL of a reference picture expressed in luma samples, and the variable fRefHeight may be set equal to PicOutputHeightL of a reference picture expressed in luma samples. In other cases, fRefWidth and fRefHeight may be set equal to pic_width_in_luma_samples and pic_height_in_luma_samples of the reference picture, respectively. Thus, in some examples, hori_scale_fp may provide a horizontal scaling ratio between a reference picture and a current picture, and vert_scale_fp may provide a vertical scaling ratio between a reference picture and a current picture.

[0175] In some examples, the starting position (upper left corner) of the block in the reference picture (denoted as (xIntL, yIntL)) and the filter phase or offset given in 1 / 16 sample units (denoted as (xFracL, yFracL)) can be derived as follows. Given (refxSbL, refySbL) and (refxL, refyL) representing the luma position given in 1 / 16 sample units pointed to by the motion vector (refMvLX[0], refMvLX[1]), the variables refxSbL, refxL, refySbL, and refyL can be derived as follows:

[0176] sbX L =((xSb<<4)+refMvLX[0])*hori_scale_fp

[0177] Equation (17)

[0178] refxL=((Sign(refxSb)*((Abs(refxSb)+128)>>8)+xL*((hori_scale_fp+8)>>4))+32)>>6

[0179] Equation (18)

[0180] refySbL=((ySb<<4)+refMvLX[1])*vert_scale_fp

[0181] Equation (19)

[0182] refyL=((Sign(refySb)*((Abs(refySb)+128)>>8)+yL*((vert_scale_fp+8)>>4))+32)>>6

[0183] Equation (20)

[0184] The variables xIntL, yIntL, xFracL, and yFracL can also be derived as follows:

[0185] xIntL=refxL>>4

[0186] Equation (21)

[0187] yIntL=refyL>>4

[0188] Equation (22)

[0189] xFracL=refxL&15

[0190] Equation (23)

[0191] yFracL=refyL&15

[0192] Equation (24)

[0193] Among them, (xSb, ySb) can specify the upper left sample of the current coding subblock relative to the upper left luma sample of the current picture, and refMvLX can be the motion vector for the reference picture list LX (X is 0 and 1).

[0194] Equations (13) to (24) are described above with respect to luma samples. However, the above equations (13) to (24) may be implemented and / or repeated for chroma color components. Thus, the above process (or any portion thereof) for calculating and using picture width and height, scaling, and position, sample, and other motion compensation information may be repeated for chroma components.

[0195] For example, in some cases, for chroma samples, equations (13) to (16) and equations (17) to (24) may be similarly performed, as described in the following illustrative examples. In some examples, the variables PicOutputWidthC and PicOutputHeightC for chroma may be derived as follows:

[0196] PicOutputWidthC = pic_width_in_chroma_samples - SubWidthC * (scaling_win_right_offset +

[0197] scaling_win_left_offset)

[0198] Equation (25)

[0199] PicOutputHeightC = pic_height_in_chroma_samples - SubHeightC * (scaling_win_bottom_offset +

[0200] scaling_win_top_offset)

[0201] Equation (26)

[0202] PicOutputWidthC and PicOutputHeightC may be calculated for the current picture (e.g., 604) and the reference picture (e.g., 610). In some cases, PicOutputWidthC and PicOutputHeightC may be calculated for the current picture (e.g., 604) and may represent the width and height of the modified current picture (e.g., 606), and may also be calculated for the reference picture (e.g., 610), in which case they may represent the width and height of the modified reference picture (e.g., 612). In addition, pic_width_in_chroma_samples and pic_height_in_chroma_samples can refer to the width and height of a picture (e.g., a current picture, a reference picture, a decoded picture, an output picture, etc.) in units of chroma samples, SubWidthC and SubHeightC can correspond to chroma samples related to luma samples (e.g., when there is a corresponding chroma sample for each luma sample (e.g., as in a 4:4:4: format), both are equal to 1; when there is only one chroma sample for 4 luma samples (e.g., as in a 4:2:0 format), both are equal to 2, and so on), and scaling_win_right_offset, scaling_win_left_offset, scaling_win_bottom_offset, scaling_win_top_offset can correspond to the left offset, right offset, bottom offset, and top offset (e.g., 506) in the scaling window (e.g., 502, 602, 608).

[0203] The scaling ratio (e.g., 530, 620) can be derived as follows:

[0204] hori_scale_fp=((fRefWidth<<14)+(PicOutputWidthC>>1)) / PicOutputWidthC

[0205] Equation (27)

[0206] vert_scale_fp=((fRefHeight<<14)+(PicOutputHeightC>>1)) / PicOutputHeightC

[0207] Equation (28)

[0208] In equations (27) and (28), PicOutputWidthC and PicOutputHeightC may be picture width and height variables derived for a picture. For example, PicOutputWidthC and PicOutputHeightC may be width and height variables of a modified current picture (e.g., 606). In some examples, the variable fRefWidth may be set equal to PicOutputWidthC of a reference picture expressed in chroma samples, and the variable fRefHeight may be set equal to PicOutputHeightC of a reference picture expressed in chroma samples. In other examples, fRefWidth and fRefHeight may be set equal to pic_width_in_chroma_samples and pic_height_in_chroma_samples of the reference picture, respectively. Thus, in some examples, hori_scale_fp may provide a horizontal scaling ratio between a modified reference picture and a modified current picture, and vert_scale_fp may provide a vertical scaling ratio between a modified reference picture and a modified current picture.

[0209] In some examples, the starting position (upper left corner) of the block in the reference picture, denoted as (xIntC, yIntC), and the filter phase or offset given in units of 1 / 16 samples, denoted as (xFracC, yFracC), can be derived as follows. Given (refxSbC, refySbC) and (refxC, refyC) representing the chroma position given in units of 1 / 16 samples pointed to by the motion vector (refMvLX[0], refMvLX[1]), the variables refxSbC, refxC, refySbC, and refyC can be derived as follows:

[0210] refxSbC=((xSb<<4)+refMvLX[0])*hori_scale_fp

[0211] Equation (29)

[0212] RefxC=((Sign(refxSb)*((Abs(refxSb)+128)>>8)+xC*((hori_scale_fp+8)>>4))+32)>>6

[0213] Equation (30)

[0214] refySbC=((ySb<<4)+refMvLX[1])*vert_scale_fp

[0215] Equation (31)

[0216] refyC=((Sign(refySb)*((Abs(refySb)+128)>>8)+yC*((vert_scale_fp+8)>>4))+32)>>6

[0217] Equation (32)

[0218] You can also derive the variable xInt as follows C ,yInt C 、xFrac C and yFrac C :

[0219] xIntC=refxC>>4

[0220] Equation (33)

[0221] yIntC=refyC>>4

[0222] Equation (34)

[0223] xFracC=refxC&15

[0224] Equation (35)

[0225] yFracC=refyC&15

[0226] Equation (36)

[0227] Among them, (xSb, ySb) can specify the upper left sample of the current coding subblock relative to the upper left chroma sample of the current picture, and refMvLX can be the motion vector for the reference picture list LX (X is 0 and 1).

[0228] In some implementations, additional window parameters (eg, coordinates, sample positions, component positions, units, etc.) may be specified to derive values ​​for fRefWidth and fRefHeight. The additional window parameters may allow a subset of a reference picture to be used to predict a subset of a current picture.

[0229] In some cases, the scaling window (e.g., 502, 602, 608) may be signaled in a picture parameter set (PPS), a video parameter set (VPS), a sequence parameter set (SPS), or any parameter set or portion of a bitstream. As previously mentioned, in some cases, the scaling window may be signaled for each picture (e.g., current picture, reference picture, etc.). In some cases, the scaling window for each picture (e.g., scaling windows for the current picture and the reference picture) may be used to derive the scaling ratio (e.g., 530, 620). In other cases, only one scaling window (e.g., a scaling window associated with the current picture or the reference picture) may be used to calculate the scaling ratio (e.g., 530, 620).

[0230] For example, in some cases, the size of the current picture can be modified using a scaling window of the current picture. The scaling ratio can then be derived using the modified size of the current picture and the decoded or cropped size of the reference picture. In another example, the size of the reference picture can be modified using a scaling window of the reference picture. The scaling ratio can then be derived using the modified size of the reference picture and the decoded or cropped size of the current picture.

[0231] In some examples, a zoom window specified using one picture can also be used to modify the size of another picture, and the size of the other picture can then be used to calculate the zoom ratio. For example, a zoom window specified using a current picture can be applied to modify the size of a reference picture, or a zoom window specified using a reference picture can be applied to modify the size of the current picture.

[0232] In some examples, a scaling window or a consistency window may be used to derive a scaling ratio. In addition, in some cases, the decoded picture may be rescaled based on a scaling window, a consistency window, or a scaling ratio.

[0233] In some cases, more than one scaling window may be signaled for a picture. For example, for each reference picture having a different picture size, a different scaling window may be signaled and used for the current picture. Such signaling of multiple scaling windows may be accomplished in various ways. For example, in some cases, a scaling window list may be signaled in a parameter set (e.g., SPS, PPS, etc.) or other portion of a bitstream. To illustrate, in some cases, for each reference picture of the current picture, a scaling window may be determined based on one or more signaled syntax elements (e.g., an index to a scaling window list, etc.), or derived based on picture resolutions and cropping window parameters of the current picture and the reference pictures. For each reference picture of the current picture, a scaling ratio may be calculated using a corresponding scaling window, and one or more samples may be encoded using the scaling ratio and starting position for a block of the reference picture.

[0234] In other examples, the scaling window may be signaled in the PPS, another parameter set, or elsewhere. When the scaling window for a reference picture is different from a previous scaling window, a new scaling window may be signaled for the reference picture. The signaling for the new scaling window may include scaling window parameters and an indication (e.g., a flag, a bit, etc.) that the reference picture is associated with the signaled new scaling window.

[0235] In some cases, a set of picture resolutions may be specified, and a scaling window may be specified for each picture resolution in the set of picture resolutions or for one or more picture resolutions in the set of picture resolutions (e.g., in a PPS, SPS, or any other part of the bitstream). In some examples, one or more scaling windows may be signaled for each pair of picture resolutions. For example, a scaling window may be signaled for picture resolution A and picture resolution B, and it may be indicated that the scaling window should be used when the current picture has picture resolution A and the reference picture has picture resolution B, and vice versa.

[0236] In some cases, the scaling window and / or consistency window may identify a portion of the picture to be rescaled (e.g., a region of interest (ROI) of the picture). In some examples, the scaling window and / or consistency window may identify the ROI of the picture to be rescaled at the ROI and not at other portions of the picture. In some cases, the picture may be cropped prior to rescaling according to a window offset specified in the scaling window and / or consistency window.

[0237] As mentioned above, in VVC, the output picture is the decoded picture after cropping, and the cropping is performed according to the offset specified in the consistency window. In the same coded video sequence (CVS), the output pictures may have different sizes, which may cause problems in some applications. The technology described herein can indicate the expected output picture size (e.g., the size expected for the display), also referred to as the expected output picture in this article. An indication (e.g., a flag, a bit, a value, etc.) can be signaled in a VPS, SPS, or elsewhere to indicate that a replacement for the decoded cropped picture can be output in the CVS. In some examples, since the output process may not affect the decoding process (e.g., the decoding process may be the same regardless of the output picture size), the expected output picture size can be signaled in a video usage information (VUI) message or a supplementary enhancement information (SEI) message.

[0238] In some cases, the desired output picture size (e.g., the width and height of the desired output picture) may be signaled. Furthermore, in some examples, the signaling of the desired output picture size may not be a multiple of 8 samples, as it may be for the pic_width_in_luma_samples and pic_height_in_luma_samples variables, because the picture is not encoded and the minimum sample multiplier constraint is not applied.

[0239] In some cases, the desired output picture size may be signaled, and constraints may be added with the condition that the signaled size of the desired output picture cannot be larger than the maximum picture size specified in the SPS (eg, pic_width_max_in_luma_samples and pic_height_max_in_luma_samples).

[0240] In order to save bits in the desired output picture signaling (e.g., signaling of the desired output picture size), the desired output picture width and height can be signaled as incremental information (e.g., difference) over the existing picture width and height signaled in the PSS or the maximum picture width and height signaled in the SPS.

[0241] In some cases, output picture size signaling may be conditional on the desired output picture indicator (e.g., flag, value, etc.) applied. For example, when the desired output picture is applied, then the output picture size may be signaled. In other cases, the desired output picture size may always be signaled regardless of the desired output picture indicator applied. If enabled, the desired output picture indicator may indicate the output picture size signaled by the application. The desired output picture indicator may provide various benefits. For example, if or when it is necessary to switch from a cropped decoded picture output to a desired output picture (or vice versa) in the same bitstream, the desired output picture indicator may be beneficial because if the desired output picture is used / applied, only one indicator needs to be rewritten to change the associated value in the bitstream (e.g., change the indicator value from 0 to 1 or from 1 to 0), rather than adding picture size signaling.

[0242] In some cases, when the desired output picture indicator is enabled, the maximum picture size signaled in the SPS can be used as the output picture size. In such a case, no additional signaling of the desired output picture width and height is required.

[0243] Since the picture before being resized (e.g., rescaled) for output may have various widths and heights, in some cases, the resampling process may be applied at a specific scaling ratio, and the resized / rescaled picture size may be different from the desired output picture size. In order to bring the resized / rescaled picture to the desired output picture size, an output consistency window (e.g., 542) with one or more offsets (e.g., left offset, right offset, top offset, and / or bottom offset) may be signaled. One or more offsets may be applied to crop the resized / rescaled picture before output.

[0244] The resampling process may include an upsampling process and / or a downsampling process, depending on the value of the scaling ratio. One or more filters may be selected based on the scaling ratio. For example, a downsampling filter may be selected when the scaling ratio is less than one, and an upsampling filter may be selected when the scaling ratio is greater than one. In some cases, filter selection may also depend on a specific scaling ratio value or one or more other syntax elements, such as, for example, a picture type. In some examples, downsampling and / or upsampling filter selection may be indicated by a group of blocks, CTUs, tiles, slices, bricks, pictures, or other blocks or units. In some cases, filters may be specified in a parameter set, such as SPS, PPS, or APS, or may be signaled elsewhere. In other cases, the filter may be fixed. In some examples, there may be N downsampling filter options and M upsampling filter options, and options for downsampling and / or upsampling may be indicated by a group of blocks, CTUs, tiles, slices, bricks, pictures, or other blocks or units.

[0245] In some cases, the selection of filters may also depend on the scaling. For example, in some cases, more filter options may be provided for more important scalings, and fewer (or one) filter options may be provided for other less important scalings. In some cases, when fewer (or one) filter options are provided, a filter indicator may not be required.

[0246] The output consistency window may be signaled in a VPS or SPS. In some cases, since the output consistency window offset value may depend on the current picture size, an indication of the current picture size may be used to signal the output consistency window. The indication of the current picture size may be signaled in a parameter set (such as a PPS or VPS or elsewhere in the bitstream). In some cases, the output consistency window signaling may be conditional on whether the desired output picture is indicated. In other cases, the desired output picture indication may be repeated at the same location (e.g., in a PPS, VPS, etc.) where the output consistency window is signaled. In order to avoid parameter set parsing dependencies (such as in some cases where the desired output picture indication is signaled in an SPS), the output consistency window may be signaled in the PPS to avoid PPS parsing being dependent on the desired output picture indication signaled in the SPS.

[0247] In some examples, to indicate the desired output picture and / or output consistency window, an additional PPS indication may be signaled in the slice header. In some cases, a PPS indication has already been signaled in the slice header to indicate the desired output picture size, consistency window, and / or other parameters. Instead of introducing additional syntax elements for the desired output picture and output consistency window parameters, an additional PPS indicator may be signaled, which may be denoted as the desired output picture PPS indicator, because the PPS may already contain picture size and consistency window information, which may be considered as the desired output picture and output consistency window parameters. In some cases, the additional PPS indicator signaling may depend on whether the desired output picture is used (e.g., whether the desired output picture indicator is enabled).

[0248] If the desired output picture is used and the size of the current picture is different from the desired output picture size, a resizing / rescaling process may be applied to the current picture. The scaling ratio may be derived based on the current picture size and the desired output picture size. In some examples, the size of the current picture may be the size of the decoded current picture, the size of the cropped current picture, or the size of the current picture after applying a scaling window offset.

[0249] In some cases, more than one output picture size may be specified. In such a case, to indicate which output picture size is used, an index may be signaled. For example, an index may be signaled per picture, which index may be used to select a desired output picture size from a plurality of output picture sizes.

[0250] In some examples, when the output picture size is indicated in the SEI or VUI, there may be no consistency point specified or defined for the bitstream. In such a case, the decoder side entity (e.g., decoding device 112) may choose to use the output picture size parameter and perform scaling and output operations where applicable and appropriate.

[0251] An example implementation using VVC is shown below.

[0252]

[0253] Table 2

[0254] In some cases, the "if (sps_alternative_output_picture_flag)" condition may be optional. In addition, sps_alternative_output_picture_flag equal to 1 specifies that each output picture has an alternative to the output picture size specified for the picture in the PPS, and sps_alternative_output_picture_flag equal to 0 specifies that the decoded picture is output after applying the consistency cropping window specified for the picture in the PPS. The variable output_pic_width_in_luma_samples specifies the width of each output picture of the reference SPS in units of luma samples, and pic_width_in_luma_samples may not be equal to 0 and may be less than or equal to pic_width_max_in_luma_samples. The variable output_pic_height_in_luma_samples specifies the height of each output picture of the reference SPS in units of luma samples, and pic_height_in_luma_samples may not be equal to 0 and may be less than or equal to pic_height_in_luma_samples.

[0255]

[0256]

[0257] Table 3

[0258] In Table 3, scaling_window_flag equal to 1 indicates that the scaling window offset parameter follows immediately in the PPS, and scaling_window_flag equal to 0 indicates that there is no scaling window offset parameter. In addition, scaling_win_left_offset, scaling_win_right_offset, scaling_win_top_offset, and scaling_win_bottom_offset specify the offset in units of luma samples to be applied to the picture size for scaling calculation. When scaling_window_flag is equal to 0, the values ​​of scale_win_left_offset, scaling_win_right_offset, scaling_win_top_offset, and scaling_win_bottom_offset are inferred to be equal to 0.

[0259] The variables PicOutputWidthL and PicOutputHeightL may be derived using equations (13) and (14), and the scaling ratio may be derived using equations (15) and (16), as previously described. output_cropping_window_flag equal to 1 indicates that the output cropping window offset parameter immediately follows in the PPS.

[0260] The variable output_cropping_window_flag equal to 0 indicates that there is no output cropping window offset parameter, and output_win_left_offset, output_win_right_offset, output_win_top_offset, and output_win_bottom_offset specify samples of the picture output from the decoding process in the CVS (e.g., in terms of specifying a rectangular area for output in picture coordinates). When output_window_flag is equal to 0, the values ​​of output_win_left_offset, output_win_right_offset, output_win_top_offset, and output_win_bottom_offset are inferred to be equal to 0.

[0261] The output cropping window contains luma samples with horizontal picture coordinates from SubWidthC*output_win_left_offset to output_pic_width_in_luma_samples-(SubWidthC*output_win_right_offset+1), and vertical picture coordinates from SubHeightC*output_win_top_offset to output_pic_height_in_luma_samples-(SubHeightC*output_win_bottom_offset+1).

[0262] The value of SubWidthC*(output_win_left_offset+output_win_right_offset) may be less than or equal to pic_width_max_in_luma_samples, and the value of SubHeightC*(output_win_top_offset+output_win_bottom_offset) may be less than pic_height_max_in_luma_samples.

[0263] In some cases, the picture output process described below may be implemented instantaneously / simultaneously (or nearly instantaneously / simultaneously) at the CPB removal time or CpbRemovalTime[n] of access unit n.

[0264] When picture n has PictureOutputFlag equal to 1, the DPB output time DpbOutputTime[n] of the picture is derived as follows:

[0265]

[0266]

[0267] Equation (37)

[0268] where picDpbOutputDelay is the value of pic_dpb_output_delay, and picDpbOutputDelta is derived from cpb_removal_delay_minus1[OpTid] and cpb_removal_delay_delta_idx[OpTid] in the picture timing SEI message associated with access unit n, and cpb_removal_delay_delta[cpb_removal_delay_delta_idx[OpTid]] in the buffering period SEI message associated with access unit n. The value of DpbOutputDelta[OpTid], and picSptDpbOutputDuDelay is the value of pic_spt_dpb_output_du_delay in the decoding unit information SEI message associated with access unit n (if present), or the value of pic_dpb_output_du_delay in the picture timing SEI message associated with access unit n (when there is no decoding unit information SEI message associated with access unit n or the decoding unit information SEI message associated with access unit n does not have a present pic_spt_dpb_output_du_delay).

[0269] In some cases, when the syntax element pic_spt_dpb_output_du_delay is not present in any decoding unit information SEI message associated with access unit n, the value may be inferred to be equal to pic_dpb_output_du_delay in the picture timing SEI message associated with access unit n.

[0270] Output of the current picture is specified as follows. If PictureOutputFlag is equal to 1 and DpbOutputTime[n] is equal to CpbRemovalTime[n], the current picture is output. Otherwise, if PictureOutputFlag is equal to 0, the current picture is not output but is stored in the DPB. Otherwise, when PictureOutputFlag is equal to 1 and DpbOutputTime[n] is greater than CpbRemovalTime[n], the current picture is output later and stored in the DPB, and is output at time DpbOutputTime[n], unless indicated by decoding or inference of no_output_of_prior_pics_flag equal to 1 not to be output at a time before DpbOutputTime[n].

[0271] When outputting, if sps_alternative_output_picture_flag is equal to 1, and the cropped picture size and the picture size after applying the consistency cropping window specified for the picture in the PPS are different from the output picture size specified for the picture in the PPS, then the scaling ratio may be derived based on the cropped picture size and the output picture size as follows:

[0272] PicCroppedWidthL=pic_width_in_luma_samples–SubWidthC*(conf_win_right_offset+

[0273] conf_win_left_offset)

[0274] Equation (38)

[0275] PicCroppedHeightL=pic_height_in_luma_samples–SubHeightC*(conf_win_bottom_offset+

[0276] conf_win_top_offset)

[0277] Equation (39)

[0278] hor_scale=((PicCroppedWidthL<<14)+(output_pic_width_in_luma_samples>>1)) /

[0279] output_pic_width_in_luma_samples

[0280] Equation (40)

[0281] ver_scale=((PicCroppedHeightL<<14)+(output_pic_height_in_luma_samples>>1)) /

[0282] output_pic_height_in_luma_samples

[0283] Equation (41)

[0284] The rescaling process may be applied to the cropped picture using the hor_scale and ver_scale scaling factors. The rescaled picture may be cropped using the output cropping window specified for the picture in the PPS. Otherwise, when sps_alternative_output_picture_flag is equal to 0 or the cropped picture size is equal to the output picture size, the picture is cropped using the consistent cropping window specified for the picture in the PPS.

[0285] When picture n is an output picture other than the last picture in the output bitstream, the value of the variable DpbOutputInterval[n] can be derived as follows:

[0286] DpbOutputInterval[n]=DpbOutputTime[nextPicInOutputOrder]-DpbOutputTime[n]

[0287] Equation (42)

[0288] Among them, nextPicInOutputOrder is the picture after picture n in the output order and has PictureOutputFlag equal to 1.

[0289] The "bumping" process can be implemented according to the following steps. First, the picture set to be output first is selected as the picture with the minimum value of PicOrderCntVal among all pictures marked as "need to be output" in the DPB. If sps_alternative_output_picture_flag is equal to 1, and the cropped picture size (e.g., the picture size after applying the consistency cropping window specified for the picture in the PPS) is different from the output picture size specified for the picture in the PPS, the scaling ratio can be derived based on the cropped picture size and the output picture size as follows:

[0290] PicCroppedWidthL = pic_width_in_luma_samples - SubWidthC * (conf_win_right_offset +

[0291] conf_win_left_offset)

[0292] Equation (43)

[0293] PicCroppedHeightL = pic_height_in_luma_samples - SubHeightC * (conf_win_bottom_offset +

[0294] conf_win_top_offset)

[0295] Equation (44)

[0296] hor_scale = ((PicCroppedWidthL << 14) + (output_pic_width_in_luma_samples >> 1)) /

[0297] output_pic_width_in_luma_samples

[0298] Equation (45)

[0299] ver_scale = ((PicCroppedHeightL << 14) + (output_pic_height_in_luma_samples >> 1)) /

[0300] output_pic_height_in_luma_samples

[0301] Equation (46)

[0302] The rescaling process can be applied to the cropped picture using the hor_scale and ver_scale scaling factors. The rescaled picture can be cropped using the output cropping window specified for the picture in the PPS. Otherwise, when sps_alternative_output_picture_flag is equal to 0 or the cropped picture size is equal to the output picture size, the picture is cropped using the consistency cropping window specified for the picture in the SPS, the cropped picture is output, and the picture is marked as "not required to be output". When the picture storage buffer including the cropped and output picture contains a picture marked as "not used for reference", the picture storage buffer can be cleared. In some examples, for any two pictures picA and picB belonging to the same CVS and output by the "bumping process", when picA is output earlier than picB, the value of PicOrderCntVal of picA is less than the value of PicOrderCntVal of picB.

[0303] In some examples, one or more offset parameters (e.g., scaling window, etc.) specified in the methods disclosed herein may be signaled as se(v) (e.g., a signed integer 0-order Exp-Golomb coded syntax element with the left bit first), and / or more generally allowed to be negative values. Signaling negative values ​​enables definition of several more scaling ratios that may not otherwise be indicated.

[0304] In other examples, the values ​​of the output picture width and height (output_pic_width_in_luma_samples and output_pic_height_luma_samples) may not be signaled, but the output crop window offset may be used to determine the output picture size and scaling for resampling.

[0305] In some cases, the variables hor_scale and ver_scale in the picture output process may be derived using pic_width_in_luma_samples and pic_height_luma_samples of the current picture instead of PicCroppedWidthL and PicCroppedHeightL.

[0306] The various division operations specified herein (e.g., using shifts, additions, and offsets) provide non-limiting illustrations of how fixed-point arithmetic can be used, and the parameters used herein are provided as non-limiting examples for illustrative purposes. It should be understood that the methods and techniques disclosed herein are applicable to other forms and / or uses of division and other mathematical operations.

[0307] In some of the above examples, the scaling is derived based on the picture width and height expressed in luma samples. In other examples, the picture width and height expressed in samples of each component may be used to derive the scaling for that component.

[0308] Figure 7 7 is a flow chart illustrating an example method 700 for calculating a scaling ratio and decoupling the scaling ratio calculation from a picture output process. At block 702, method 700 may include obtaining a current picture of video data (e.g., 604), and at block 704, method 700 may include determining one or more scaling windows (e.g., 502, 602, 608) associated with the current picture and / or a reference picture (e.g., 610) selected for use in encoding the current picture. In some cases, one or more scaling windows may be associated with one or more additional reference pictures. For example, one or more scaling windows may be associated with multiple reference pictures for the current picture (e.g., bidirectional inter-frame prediction).

[0309] In some examples, the one or more zoom windows may include a first zoom window for the current picture (e.g., 602) and a second zoom window for the reference picture (e.g., 608). In other examples, the one or more zoom windows may include a zoom window for both the current picture and the reference picture. In some examples, the one or more zoom windows may be derived for the horizontal direction and / or the vertical direction.

[0310] In some examples, one or more scaling windows may be determined based on a picture parameter set (PPS). In some cases, one or more scaling windows may be included in a PPS, and the PPS including the one or more scaling windows may be stored and / or sent (e.g., in a coded video bitstream). For example, in some cases, a scaling window for a current picture may be determined based on a PPS referenced / referenced by the current picture, and / or a scaling window for a reference picture may be determined based on a PPS referenced and / or referenced by a reference window.

[0311] In some aspects, one or more zoom windows may include a zoom window offset (e.g., 506). In some examples, the zoom window offset may include a left zoom window offset, a right zoom window offset, a top zoom window offset, and / or a bottom zoom window offset. In some cases, one or more zoom windows may include negative values. In addition, zoom window offset signaling may allow negative values.

[0312] At block 706, method 700 may include determining a first size of the current picture and / or a second size of the reference picture. In some examples, the first size may include a first height and / or a first width of the current picture, and the second size may include a second height and / or a second width of the reference picture.

[0313] At block 708, method 700 may include determining one or more scaling ratios associated with the current picture and / or the reference picture based on one or more scaling windows and a first size (e.g., of the current picture) and / or a second size (e.g., of the reference picture) (e.g., 530, 620). In some cases, the one or more scaling ratios may be determined based on scaling window offsets (e.g., top, bottom, right, and / or left scaling window offsets) associated with the one or more scaling windows. For example, in some cases, one or more scaling window offsets associated with the scaling window of the current picture may be added to a first height and / or a first width of the current picture to determine the scaling ratio associated with the current picture. Similarly, in some cases, one or more scaling window offsets associated with the scaling window of the reference picture may be added to a second height and / or a second width of the reference picture to determine the scaling ratio associated with the reference picture.

[0314] In some cases where the one or more zoom windows include a first zoom window for a current picture and a second zoom window for a reference picture, one or more zoom ratios may be determined based on the first zoom window and the second zoom window.

[0315] In some aspects, method 700 may include: adjusting one or more motion compensation parameters associated with the current picture and / or the reference picture based on one or more scaling ratios. In some examples, the one or more motion compensation parameters may include a starting position of one or more blocks in the reference picture, an adjusted interpolation filter phase, one or more component (e.g., luminance, chrominance) positions referenced by one or more motion vectors and / or corresponding to one or more motion vectors, and / or one or more motion vectors. In some examples, the adjusted interpolation filter phase may include one or more offsets.

[0316] In addition, in some cases, the starting position, interpolation filter phase, and / or one or more component positions of one or more blocks in the reference picture can be adjusted to take into account one or more scaling ratios. For example, in some cases, the starting position, interpolation filter phase, and / or one or more component positions of one or more blocks in the reference picture can be calculated or implemented using the above equations (17-20), equations (21-24), equations (29)-(32), and / or equations (33)-(36).

[0317] In some aspects, method 700 may include: obtaining one or more consistency windows associated with the current picture and / or the reference picture (e.g., 542); and resizing the current picture and / or the reference picture based on the one or more consistency windows. In some examples, the one or more consistency windows may include one or more window offsets. In some cases, the one or more consistency windows may be obtained, determined, and / or signaled from / via a sequence parameter set (SPS). In some examples, one or more consistency windows and one or more scaling windows are signaled in the coded video bitstream.

[0318] In some examples, method 700 may include: including one or more consistency windows in the SPS; and storing and / or sending the SPS including the one or more consistency windows.

[0319] In some aspects, the method 700 may further include, after resizing the current picture and / or the reference picture based on the one or more consistency windows, performing motion compensation on the current picture based on the adjusted one or more motion compensation parameters.

[0320] In some aspects, method 700 may include: obtaining a predetermined maximum picture size indicating a maximum picture width and a maximum picture height; and generating an output picture based on the predetermined maximum picture size and a consistency window including one or more window offsets. In some examples, the output picture may be a current picture that is resized based on the predetermined maximum picture size and the consistency window. In some cases, the current picture may be resized by applying one or more window offsets to the maximum picture width and the maximum picture height.

[0321] In addition, in some aspects, a predetermined maximum picture size and / or consistency window may be determined based on a sequence parameter set (SPS). In some aspects, the predetermined maximum picture size and / or consistency window may be included in the SPS, and the SPS including the predetermined maximum picture size and / or consistency window may be stored and / or sent.

[0322] In some aspects, method 700 may include generating a coded video bitstream including one or more scaling windows and a consistency window set. The consistency window set may include one or more first window offsets for resizing a current picture and one or more second window offsets for resizing a reference picture. In some examples, method 700 may include storing and / or transmitting a coded video bitstream including one or more scaling windows and a consistency window set.

[0323] In some aspects, method 700 may include generating (eg, via encoding device 104) an encoded video bitstream comprising video data. In other aspects, method 700 may include decoding (eg, via decoding device 112) the encoded video bitstream.

[0324] In some examples, method 700 may include generating a coded bitstream including one or more scaling windows and / or one or more consistency windows. In some examples, the one or more scaling windows and / or consistency windows may be included in the coded bitstream. In some examples, the one or more scaling windows and / or consistency windows may be signaled and may include one or more corresponding values ​​(e.g., one or more corresponding window offsets) corresponding to the one or more scaling windows and / or consistency windows.

[0325] In some aspects, method 700 may include: obtaining a maximum picture size indicating a maximum picture width and a maximum picture height; and generating an output picture based on the maximum picture size and a consistency window including one or more window offsets. In some examples, the size of the current picture may be equal to the maximum picture size, and the output picture may include the current picture cropped with a size equal to the maximum picture size based on the consistency window. In some cases, the current picture (with a size equal to the maximum picture size) may be cropped by applying one or more window offsets to the current picture. In some cases, the maximum picture size and the consistency window are determined according to a sequence parameter set.

[0326] In some examples, any of the above equations (13) to (36) may be used to perform blocks 704, 706, 708 and / or other aspects of the above method 700. In addition, in some cases, at least some of the above equations (13) to (46) may be used to implement at least some aspects of the above method 700.

[0327] In some implementations, the processes (or methods) described herein (including method 700) may be performed by a computing device or apparatus (such as, for example, Figure 1 For example, these processes may be performed by the system 100 shown in Figure 1 and Figure 8 The encoding device 104 shown in FIG. 1 , another video source side device or a video transmission device, Figure 1 and Fig. 9, and / or another client-side device (such as a player device, a display, or any other client-side device). In some cases, the computing device or apparatus may include one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components configured to perform the steps of method 700.

[0328] In some examples, the computing device may include a mobile device, a desktop computer, a server computer and / or a server system, or other types of computing devices. The components of the computing device (e.g., one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components) may be implemented in a circuit system. For example, the components may include electronic circuits or other electronic hardware and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other appropriate circuits), and / or may include computer software, firmware, or any combination thereof and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. In some examples, the computing device or apparatus may include a camera configured to capture video data (e.g., a video sequence) including video frames. In some examples, the camera or other capture device that captures video data is separate from the computing device, in which case the computing device receives or obtains the captured video data. The computing device may include a network interface configured to transmit video data. The network interface may be configured to transmit data based on an Internet Protocol (IP) or other types of data. In some examples, a computing device or apparatus may include a display for displaying output video content (eg, samples of pictures of a video bitstream).

[0329] The processes may be described with respect to a logical flow diagram, the operations of which represent a series of operations that may be implemented using hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the processes.

[0330] In addition, these processes can be performed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or a combination thereof. As mentioned above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including multiple instructions that can be executed by one or more processors. The computer-readable storage medium or machine-readable storage medium can be non-transitory.

[0331] The coding techniques discussed herein can be implemented in an example video encoding and decoding system (e.g., system 100). In some examples, the system includes a source device that provides encoded video data to be decoded later by a destination device. Specifically, the source device provides the video data to the destination device via a computer-readable medium. The source device and the destination device may include any of a variety of devices, including a desktop computer, a notebook computer (i.e., a laptop computer), a tablet computer, a set-top box, a telephone handset (e.g., a so-called "smart" phone), a so-called "smart" board, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, etc. In some cases, the source device and the destination device may be equipped for wireless communication.

[0332] The destination device may receive the encoded video data to be decoded via a computer-readable medium. The computer-readable medium may be any type of medium or device capable of moving the encoded video data from the source device to the destination device. In one example, the computer-readable medium may be a communication medium for enabling the source device to directly send the encoded video data to the destination device in real time. The encoded video data may be modulated according to a communication standard such as a wireless communication protocol and sent to the destination device. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form a part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other device that may be used to facilitate communication from the source device to the destination device.

[0333] In some examples, the encoded data can be output from the output interface to the storage device. Similarly, the encoded data can be accessed from the storage device through the input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a DVD, a CD-ROM, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device may correspond to a file server or another intermediate storage device, which may store the encoded video generated by the source device. The destination device may access the stored video data from the storage device via streaming or downloading. The file server may be any type of server capable of storing encoded video data and sending the encoded video data to the destination device. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The destination device may access the encoded video data through any standard data connection (including an Internet connection). This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both suitable for accessing the encoded video data stored on the file server. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0334] The technology of the present disclosure is not necessarily limited to wireless applications or settings. The technology can be applied to video encoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission (e.g., dynamic adaptive streaming over HTTP (DASH)), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, the system can be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0335] In one example, the source device includes a video source, a video encoder, and an output interface. The destination device may include an input interface, a video decoder, and a display device. The video encoder of the source device may be configured to apply the technology disclosed herein. In other examples, the source device and the destination device may include other components or arrangements. For example, the source device may receive video data from an external video source such as an external camera. Similarly, the destination device may be docked with an external display device rather than including an integrated display device.

[0336] The example system above is only an example. The technology for processing video data in parallel can be performed by any digital video encoding and / or decoding device. Although in general, the technology of the present disclosure is performed by a video encoding device, the technology can also be performed by a video encoder / decoder commonly referred to as a "CODEC". In addition, the technology of the present disclosure can also be performed by a video preprocessor. The source device and the destination device are only examples of such encoding devices: wherein the source device generates encoded video data for transmission to the destination device. In some examples, the source device and the destination device can operate in a substantially symmetrical manner so that each of these devices includes a video encoding and decoding component. Therefore, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0337] The video source can include a video capture device, such as a video camera, a video archive unit comprising previously captured video, and / or a video feed interface for receiving video from a video content provider. As an alternative, the video source can generate computer graphics-based data as the source video, or generate a combination of real-time video, archived video, and computer-generated video. In some cases, if the video source is a video camera, the source device and the destination device can form a so-called camera phone or video phone. However, as mentioned above, the technology described in this disclosure can be generally applicable to video encoding, and can be applied to wireless and / or wired applications. In each case, the captured, pre-captured or computer-generated video can be encoded by a video encoder. Then, the encoded video information can be output to a computer-readable medium by an output interface.

[0338] As mentioned, computer-readable media may include temporary media such as wireless broadcasts or wired network transmissions, or storage media such as hard disks, flash drives, compact discs, digital versatile discs, Blu-ray discs (i.e., non-temporary storage media), or other computer-readable media. In some examples, a network server (not shown) may receive encoded video data from a source device, for example, via a network transmission, and provide the encoded video data to a destination device. Similarly, a computing device of a media production facility such as a disc stamping facility may receive encoded video data from a source device and manufacture a disc containing the encoded video data. Therefore, in various examples, a computer-readable medium may be understood to include one or more computer-readable media in various forms.

[0339] The input interface of the destination device receives information from the computer-readable medium. The information of the computer-readable medium may include syntax information defined by the video encoder (which is also used by the video decoder), the syntax information including syntax elements describing the characteristics and / or processing of blocks and other coding units (e.g., groups of pictures (GOPs)). The display device displays the decoded video data to a user and may include any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device. Various embodiments of the present application have been described.

[0340] exist Figure 8 and Fig. 9 Specific details of the encoding device 104 and the decoding device 112 are shown respectively. Figure 8 1 is a block diagram illustrating an example encoding device 104 that can implement one or more of the techniques described in the present disclosure. The encoding device 104 can, for example, generate the syntax structures described herein (e.g., syntax structures of VPS, SPS, PPS or other syntax elements). The encoding device 104 can perform intra-frame prediction and inter-frame prediction encoding of video blocks within a video slice. As previously described, intra-frame coding relies at least in part on spatial prediction to reduce or remove spatial redundancy within a given video frame or picture. Inter-frame coding relies at least in part on temporal prediction to reduce or remove temporal redundancy within adjacent or surrounding frames of a video sequence. Intra-frame mode (I mode) can refer to any of several spatial-based compression modes. Modes such as unidirectional prediction (P mode) or bidirectional prediction (B mode) can refer to any of several temporal-based compression modes.

[0341] The encoding device 104 includes a segmentation unit 35, a prediction processing unit 41, a filter unit 63, a picture memory 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 includes a motion estimation unit 42, a motion compensation unit 44, and an intra-frame prediction processing unit 46. For video block reconstruction, the encoding device 104 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and a summer 62. The filter unit 63 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although in Figure 8Filter unit 63 is shown as an in-loop filter in FIG. 1 , but in other configurations, filter unit 63 may be implemented as a post-loop filter. Post-processing device 57 may perform additional processing on the encoded video data generated by encoding device 104. In some cases, the techniques of the present disclosure may be implemented by encoding device 104. However, in other cases, one or more of the techniques of the present disclosure may be implemented by post-processing device 57.

[0342] like Figure 8 As shown, the encoding device 104 receives video data, and the segmentation unit 35 segments the data into video blocks. This segmentation may also include, for example, segmentation into slices, slice segments, tiles or other larger units according to the quadtree structure of LCU and CU, as well as video block segmentation. The encoding device 104 generally shows a component for encoding a video block within a video slice to be encoded. A slice may be divided into a plurality of video blocks (and may be divided into a set of video blocks called tiles). The prediction processing unit 41 may select one of a plurality of possible encoding modes for the current video block based on error results (e.g., coding rate and distortion level, etc.), such as one of a plurality of intra-frame prediction encoding modes or one of a plurality of inter-frame prediction encoding modes. The prediction processing unit 41 may provide the resulting intra-frame or inter-frame coded block to the summer 50 to generate residual block data, and to the summer 62 to reconstruct the coded block for use as a reference picture.

[0343] Intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction encoding of the current block relative to one or more neighboring blocks in the same frame or slice as the current video block to be encoded to provide spatial compression. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction encoding of the current video block relative to one or more prediction blocks in one or more reference pictures to provide temporal compression.

[0344] Motion estimation unit 42 may be configured to determine an inter-prediction mode for a video slice according to a predetermined pattern for a video sequence. The predetermined pattern may designate a video slice in a sequence as a P slice, a B slice, or a GPB slice. Motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates motion for a video block. A motion vector may, for example, indicate the displacement of a prediction unit (PU) of a video block within a current video frame or picture relative to a prediction block within a reference picture.

[0345] A prediction block is a block that is found to closely match a PU of a video block to be encoded in terms of pixel difference, which may be determined by sum of absolute difference (SAD), sum of squared difference (SSD), or other difference metrics. In some examples, encoding device 104 may calculate values ​​for sub-integer pixel positions of a reference picture stored in picture memory 64. For example, encoding device 104 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of a reference picture. Thus, motion estimation unit 42 may perform motion searches relative to full pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision.

[0346] Motion estimation unit 42 calculates a motion vector for a PU by comparing the position of the PU of the video block in the inter-coded slice with the position of the prediction block of the reference picture. The reference picture may be selected from a first reference picture list (list 0) or a second reference picture list (list 1), each of which identifies one or more reference pictures stored in picture memory 64. Motion estimation unit 42 sends the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44.

[0347] Motion compensation performed by motion compensation unit 44 may involve obtaining or generating a prediction block based on a motion vector determined by motion estimation, possibly performing interpolation to sub-pixel precision. Upon receiving a motion vector for a PU of a current video block, motion compensation unit 44 may locate the prediction block pointed to by the motion vector in a reference picture list. Encoding device 104 forms a residual video block by subtracting pixel values ​​of the prediction block from pixel values ​​of the current video block being encoded to form pixel difference values. The pixel difference values ​​form residual data for the block and may include both a luminance difference component and a chrominance difference component. Summer 50 represents one or more components that perform such subtraction operations. Motion compensation unit 44 may also generate syntax elements associated with video blocks and video slices for use by decoding device 112 when decoding video blocks of video slices.

[0348] As described above, the intra-prediction processing unit 46 may perform intra-prediction on the current block as an alternative to the inter-prediction performed by the motion estimation unit 42 and the motion compensation unit 44. Specifically, the intra-prediction processing unit 46 may determine the intra-prediction mode to be used for encoding the current block. In some examples, the intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during a separate encoding process, and the intra-prediction processing unit 46 may select a suitable intra-prediction mode from the tested modes to use. For example, the intra-prediction processing unit 46 may calculate a rate-distortion value using a rate-distortion analysis for various tested intra-prediction modes, and may select an intra-prediction mode with the best rate-distortion characteristic in the tested modes. The rate-distortion analysis typically determines the amount of distortion (or error) between the coded block and the original uncoded block that is coded to generate the coded block, and the bit rate (i.e., the number of bits) used to generate the coded block. Intra-prediction processing unit 46 may calculate ratios from the distortions and rates for the various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0349] In any case, after selecting an intra-prediction mode for a block, the intra-prediction processing unit 46 may provide information indicating the intra-prediction mode selected for the block to the entropy encoding unit 56. The entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode. The encoding device 104 may include in the transmitted bitstream configuration data the definition of the coding contexts for the various blocks and the most probable intra-prediction mode to be used for each of these contexts, the intra-prediction mode index table, and the indication of the modified intra-prediction mode index table. The bitstream configuration data may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also referred to as codeword mapping tables).

[0350] After the prediction processing unit 41 generates a prediction block for the current video block via inter-frame prediction or intra-frame prediction, the encoding device 104 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and applied to the transform processing unit 52. The transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform such as a discrete cosine transform (DCT) or a conceptually similar transform. The transform processing unit 52 may convert the residual video data from a pixel domain to a transform domain such as a frequency domain.

[0351] Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of these coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of the matrix including the quantized transform coefficients. Alternatively or in addition, entropy encoding unit 56 may perform the scan.

[0352] After quantization, entropy coding unit 56 performs entropy coding on the quantized transform coefficients. For example, entropy coding unit 56 may perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique. After entropy coding by entropy coding unit 56, the encoded bitstream may be sent to decoding device 112 or archived for later transmission or retrieval by decoding device 112. Entropy coding unit 56 may also entropy encode motion vectors and other syntax elements for the current video slice being encoded.

[0353] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain for later use as a reference block of a reference picture. Motion compensation unit 44 may calculate a reference block by adding the residual block to a prediction block of one of the reference pictures in the reference picture list. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for motion estimation. Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reference block for storage in picture memory 64. The reference block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block to inter-predict a block in a subsequent video frame or picture.

[0354] In this way, Figure 8 The encoding device 104 represents an apparatus configured to perform any of the techniques described herein (including the above-mentioned Figure 7 In some cases, some of the techniques of the present disclosure may also be implemented by post-processing device 57.

[0355] Fig. 91 is a block diagram illustrating an example decoding device 112. The decoding device 112 includes an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, a summer 90, a filter unit 91, and a picture memory 92. The prediction processing unit 81 includes a motion compensation unit 82 and an intra-frame prediction processing unit 84. In some examples, the decoding device 112 may perform operations generally associated with the processing of the image from Figure 8 The decoding stage is the opposite of the encoding stage described by the encoding device 104.

[0356] During the decoding process, the decoding device 112 receives a coded video bitstream sent by the encoding device 104, which represents the video blocks and associated syntax elements of the coded video slice. In some embodiments, the decoding device 112 may receive the coded video bitstream from the encoding device 104. In some embodiments, the decoding device 112 may receive the coded video bitstream from the encoding device 104. In some embodiments, the decoding device 112 may receive the coded video bitstream from a network entity 79 (such as a server, a media-aware network element (MANE), a video editor / splicer, or other such device configured to implement one or more of the above-mentioned techniques). The network entity 79 may or may not include the encoding device 104. Before the network entity 79 sends the coded video bitstream to the decoding device 112, the network entity 79 may implement some of the techniques described in the present disclosure. In some video decoding systems, the network entity 79 and the decoding device 112 may be parts of separate devices, while in other cases, the functions described with respect to the network entity 79 may be performed by the same device including the decoding device 112.

[0357] The entropy decoding unit 80 of the decoding device 112 entropy decodes the bitstream to generate quantized coefficients, motion vectors, and other syntax elements. The entropy decoding unit 80 forwards the motion vectors and other syntax elements to the prediction processing unit 81. The decoding device 112 can receive syntax elements at the video slice level and / or the video block level. The entropy decoding unit 80 can process and parse both fixed length syntax elements and variable length syntax elements in more parameter sets such as VPS, SPS, and PPS.

[0358] When the video slice is encoded as an intra-frame coded (I) slice, the intra-frame prediction processing unit 84 of the prediction processing unit 81 can generate prediction data for the video block of the current video slice based on the signaled intra-frame prediction mode and the data in the previously decoded block from the current frame or picture. When the video frame is encoded as an inter-frame coded (i.e., B, P or GPB) slice, the motion compensation unit 82 of the prediction processing unit 81 generates a prediction block for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 80. The prediction block can be generated from one of the reference pictures in the reference picture list. The decoding device 112 can construct the reference frame lists, i.e., List 0 and List 1, using a default construction technique based on the reference pictures stored in the picture memory 92.

[0359] Motion compensation unit 82 determines prediction information for video blocks of the current video slice by parsing motion vectors and other syntax elements, and uses the prediction information to generate a prediction block for the current video block being decoded. For example, motion compensation unit 82 may use one or more syntax elements in a parameter set to determine a prediction mode (e.g., intra or inter prediction) for encoding video blocks of a video slice, an inter-prediction slice type (e.g., a B slice, a P slice, or a GPB slice), construction information for one or more reference picture lists for the slice, motion vectors for each inter-coded video block of the slice, inter-prediction status for each inter-coded video block of the slice, and other information for decoding video blocks in the current video slice.

[0360] The motion compensation unit 82 may also perform interpolation based on an interpolation filter. The motion compensation unit 82 may calculate interpolated values ​​for pixels below the integer of the reference block using the interpolation filter used by the encoding device 104 during encoding of the video block. In this case, the motion compensation unit 82 may determine the interpolation filter used by the encoding device 104 based on the received syntax elements, and may use the interpolation filter to generate a prediction block.

[0361] The inverse quantization unit 86 inverse quantizes or dequantizes the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 80. The inverse quantization process may include determining the degree of quantization using the quantization parameters calculated by the encoding device 104 for each video block in the video slice, and likewise determining the degree of inverse quantization that should be applied. The inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT or other suitable inverse transform), an inverse integer transform, or a conceptually similar inverse transform process to the transform coefficients to produce residual blocks in the pixel domain.

[0362] After the motion compensation unit 82 generates a prediction block for the current video block based on the motion vector and other syntax elements, the decoding device 112 forms a decoded video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82. Summer 90 represents one or more components that perform such a summing operation. If desired, a loop filter (in the encoding loop or after the encoding loop) may also be used to smooth pixel transitions or otherwise improve video quality. Filter unit 91 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although in Fig. 9 Filter unit 91 is shown as an in-loop filter in FIG. 1 , but in other configurations, filter unit 91 may be implemented as a post-loop filter. The decoded video blocks in a given frame or picture are then stored in picture memory 92, which stores reference pictures for subsequent motion compensation. Picture memory 92 also stores the decoded video for later viewing on a display device such as a Figure 1 122).

[0363] In this way, Fig. 9 The decoding device 112 represents a device configured to perform any of the techniques described herein (including the above with respect to Figure 7 An example of a video decoder of the process described above.

[0364] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and does not include the following: carrier waves and / or temporary electronic signals propagated wirelessly or on a wired connection. Examples of non-transitory media may include, but are not limited to: disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices. Computer-readable media may have codes and / or machine-executable instructions stored thereon, which may represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent via any appropriate means including memory sharing, message passing, token passing, network transmission, etc.

[0365] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0366] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be appreciated by those of ordinary skill in the art that these embodiments can be implemented without these specific details. For clarity of explanation, in some cases, the technology herein can be presented as a separate functional block including the following functional blocks, which include equipment, device components, steps or routines in the method embodied in software, or a combination of hardware and software. In addition to the components shown in the figures and / or described herein, additional components can also be used. For example, circuits, systems, networks, processes, and other components can be shown as components in block diagram form, so as not to obscure these embodiments in unnecessary details. In other cases, known circuits, processes, algorithms, structures, and techniques may be shown as not having unnecessary details, so as to avoid obscuring these embodiments.

[0367] Each embodiment may be described as a process or method above, and the process or method is depicted as a flow chart, a schematic flow diagram, a data flow diagram, a structure diagram or a block diagram. Although the flow chart can describe the operation as a sequential process, many operations in these operations can be performed in parallel or simultaneously. In addition, the order of the operation can be rearranged. The process is terminated after its operation is completed, but may have an extra step that is not included in the figure. A process (process) may correspond to a method, a function, a procedure (procedure), a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0368] The process and method according to the above examples can be implemented using computer executable instructions, which are stored in a computer readable medium or can be obtained from a computer readable medium in other ways. Such instructions may include, for example, instructions or data, which cause a general-purpose computer, a special-purpose computer or a processing device to execute or otherwise be configured to perform a specific function or a specific set of functions. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, etc.

[0369] The equipment implementing the process and method according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language or any combination thereof, and may adopt any of a variety of form factors. When implemented with software, firmware, middleware or microcode, the program code or code segment (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc., of laptop computers, smart phones, mobile phones, tablet devices or other small form factors. The functions described herein may also be embodied in peripheral devices or plug-in cards. By way of additional example, this function may also be implemented on circuit boards between different chips or different processes executed in a single device.

[0370] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0371] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative embodiments of the present application have been described in detail herein, it should be understood that the inventive concept can be embodied and adopted differently in other ways, and the appended claims are intended to be interpreted as including such variations, except for variations limited by the prior art. The various features and aspects of the above-mentioned applications can be used individually or collectively. In addition, without departing from the broader spirit and scope of this specification, the embodiments can be used in any number of environments and applications other than the environments and applications described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that in an alternative embodiment, the method can be performed in an order different from the described order.

[0372] It will be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0373] When a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing circuits or other hardware to perform the operation, by programming programmable circuits (e.g., a microprocessor or other appropriate circuitry) to perform the operation, or any combination thereof.

[0374] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other appropriate communication interface).

[0375] Claim language or other language that recites "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that recites "at least one of A and B" means A, B, or A and B. In another example, claim language that recites "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that recites "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0376] Various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, frames, modules, circuits and steps have been generally described above around their functions. Whether this function is implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. Technicians can implement the described functions in different ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the scope of the application.

[0377] The technology described herein can also be implemented with electronic hardware, computer software, firmware, or any combination thereof. Such technology can be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device mobile phone, or an integrated circuit device with multiple uses (including applications in wireless communication devices and other devices). Any features described as modules or components can be implemented together in an integrated logic device, or as separate but interoperable logic devices. If implemented with software, the technology can be implemented at least in part by a computer-readable data storage medium, which includes a program code, and the program code includes instructions for executing one or more of the above methods when executed. The computer-readable data storage medium can form a part of a computer program product, and the computer program product can include packaging materials. The computer-readable medium can include a memory or a data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, etc. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium (such as a propagated signal or wave) that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0378] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any technology described in this disclosure. A general-purpose processor can be a microprocessor, but in an alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, the term "processor" as used herein can refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the technology described herein. In addition, in some aspects, the functions described herein can be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated in a combined video encoder-decoder (CODEC).

Claims

1. A method for encoding video data, the method comprising: Get the current picture of the video data; determining one or more scaling windows identifying at least one of a top offset, a bottom offset, a left offset, or a right offset for use in determining at least one scaling ratio for at least one of the current picture or a reference picture selected for use in encoding the current picture; Determine at least one of a first size of the current picture or a second size of the reference picture, the first size comprising a first height and a first width of the current picture, and the second size comprising a second height and a second width of the reference picture; determining, based on the one or more scaling windows and at least one of the first size or the second size, one or more scaling ratios associated with adjusting one or more motion compensation parameters corresponding to at least one of the current picture or the reference picture; as well as One or more consistency windows associated with resizing at least one of the current picture or the reference picture for output are determined, wherein the one or more consistency windows are different from the one or more scaling windows.

2. The method according to claim 1, wherein: The one or more scaling factors are derived for a horizontal direction and a vertical direction.

3. The method according to claim 1, wherein: The one or more zoom windows include a zoom window of the current picture, wherein the zoom window of the current picture is determined according to a picture parameter set (PPS) referenced by the current picture.

4. The method according to claim 1, wherein: The one or more scaling windows include a scaling window of the reference picture, wherein the scaling window of the reference picture is determined according to a picture parameter set (PPS) referenced by the reference picture.

5. The method according to claim 1, wherein: The one or more zoom windows include a zoom window offset, wherein the zoom window offset includes at least one of: a left zoom window offset, a right zoom window offset, a top zoom window offset, or a bottom zoom window offset.

6. The method according to claim 5, wherein: At least one of the scaling window offsets comprises a negative value.

7. The method according to claim 5, wherein: Determining the one or more scaling ratios includes adding the scaling window offset to at least one of the first size of the current picture or the second size of the reference picture.

8. The method according to claim 1, wherein: The one or more scaling windows include a first scaling window associated with the current picture and a second scaling window associated with the reference picture, and wherein the one or more scaling ratios are determined based on the first scaling window and the second scaling window.

9. The method according to claim 1, further comprising: including the one or more scaling windows in a picture parameter set PPS; as well as At least one of the following operations is performed: storing the PPS including the one or more zoom windows, or sending the PPS including the one or more zoom windows.

10. The method according to claim 1, further comprising: One or more motion compensation parameters associated with at least one of the current picture or the reference picture are adjusted based on the one or more scaling ratios.

11. The method according to claim 1, further comprising: At least one of the current picture or the reference picture is resized based on the one or more consistency windows.

12. The method according to claim 11, wherein: The one or more consistency windows are determined according to a sequence parameter set (SPS).

13. The method according to claim 11, further comprising: including the one or more consistency windows in a sequence parameter set SPS; as well as At least one of the following operations is performed: storing the SPS including the one or more consistency windows, or sending the SPS including the one or more consistency windows.

14. The method according to claim 1, further comprising: Get the maximum image size indicating the maximum image width and maximum image height; as well as An output picture is generated based on the maximum picture size and a consistency window including one or more window offsets, wherein the size of the current picture is equal to the maximum picture size, and wherein the output picture includes the current picture cropped with the size equal to the maximum picture size based on the consistency window.

15. The method according to claim 14, wherein: The current picture is cropped by applying the one or more window offsets to the current picture.

16. The method according to claim 14, wherein: The maximum picture size and the consistency window are determined according to a sequence parameter set (SPS).

17. The method according to claim 14, further comprising: Including the maximum picture size and the consistency window in a sequence parameter set SPS; as well as At least one of the following operations is performed: storing the SPS including the maximum picture size and the consistency window, or sending the SPS including the maximum picture size and the consistency window.

18. The method according to claim 1, wherein: The one or more scaling windows are also associated with one or more additional reference pictures selected for use in encoding the current picture, wherein the one or more scaling ratios are further associated with the one or more additional reference pictures.

19. The method of claim 1, further comprising: generating a coded video bitstream including the one or more scaling windows and a consistency window set, the consistency window set including one or more first window offsets for resizing the current picture and one or more second window offsets for resizing the reference picture; as well as At least one of the following operations is performed: storing the encoded video bitstream, or sending the encoded video bitstream.

20. An apparatus comprising: Memory; as well as one or more processors coupled to the memory, the one or more processors configured to: Get the current picture of the video data; determining one or more scaling windows identifying at least one of a top offset, a bottom offset, a left offset, or a right offset for use in determining at least one scaling ratio for at least one of the current picture or a reference picture selected for use in encoding the current picture; Determine at least one of a first size of the current picture or a second size of the reference picture, the first size comprising a first height and a first width of the current picture, and the second size comprising a second height and a second width of the reference picture; determining, based on the one or more scaling windows and at least one of the first size or the second size, one or more scaling ratios associated with adjusting one or more motion compensation parameters corresponding to at least one of the current picture or the reference picture; as well as One or more consistency windows associated with resizing at least one of the current picture or the reference picture for output are determined, wherein the one or more consistency windows are different from the one or more scaling windows.

21. The device according to claim 20, wherein: The one or more scaling factors are derived for a horizontal direction and a vertical direction.

22. The device according to claim 20, wherein: The one or more zoom windows include a zoom window of the current picture, wherein the zoom window of the current picture is determined according to a picture parameter set (PPS) referenced by the current picture.

23. The device according to claim 20, wherein: The one or more scaling windows include a scaling window of the reference picture, wherein the scaling window of the reference picture is determined according to a picture parameter set (PPS) referenced by the reference picture.

24. The device according to claim 20, wherein: The one or more zoom windows include a zoom window offset, wherein the zoom window offset includes at least one of: a left zoom window offset, a right zoom window offset, a top zoom window offset, or a bottom zoom window offset.

25. The device according to claim 24, wherein: At least one of the scaling window offsets comprises a negative value.

26. The device according to claim 24, wherein: Determining the one or more scaling ratios includes adding the scaling window offset to at least one of the first size of the current picture or the second size of the reference picture.

27. The device according to claim 20, wherein: The one or more scaling windows include a first scaling window associated with the current picture and a second scaling window associated with the reference picture, and wherein the one or more scaling ratios are determined based on the first scaling window and the second scaling window.

28. The apparatus of claim 20, wherein the one or more processors are configured to: including the one or more scaling windows in a picture parameter set PPS; and At least one of the following operations is performed: storing the PPS including the one or more zoom windows, or sending the PPS including the one or more zoom windows.

29. The apparatus of claim 20, wherein the one or more processors are configured to: One or more motion compensation parameters associated with at least one of the current picture or the reference picture are adjusted based on the one or more scaling ratios.

30. The apparatus of claim 20, wherein the one or more processors are configured to: At least one of the current picture or the reference picture is resized based on the one or more consistency windows.

31. The device according to claim 30, wherein The one or more consistency windows are determined according to a sequence parameter set (SPS).

32. The apparatus of claim 30, wherein the one or more processors are configured to: including the one or more consistency windows in a sequence parameter set SPS; and At least one of the following operations is performed: storing the SPS including the one or more consistency windows, or sending the SPS including the one or more consistency windows.

33. The apparatus of claim 20, wherein the one or more processors are configured to: Get the maximum image size indicating the maximum image width and maximum image height; and generating an output picture based on the maximum picture size and a consistency window including one or more window offsets, wherein: The size of the current picture is equal to the maximum picture size, and wherein the output picture comprises the current picture cropped based on the consistency window and having the size equal to the maximum picture size.

34. The device according to claim 33, wherein The current picture is cropped by applying the one or more window offsets to the current picture.

35. The device according to claim 33, wherein The maximum picture size and the consistency window are determined according to a sequence parameter set (SPS).

36. The apparatus of claim 33, wherein the one or more processors are configured to: including the maximum picture size and the consistency window in a sequence parameter set SPS; and At least one of the following operations is performed: storing the SPS including the maximum picture size and the consistency window, or sending the SPS including the maximum picture size and the consistency window.

37. The apparatus of claim 20, wherein: The one or more scaling windows are also associated with one or more additional reference pictures selected for use in encoding the current picture, wherein the one or more scaling ratios are further associated with the one or more additional reference pictures.

38. The apparatus of claim 20, wherein the one or more processors are configured to: generating a coded video bitstream including the one or more scaling windows and a consistency window set, the consistency window set including one or more first window offsets for resizing the current picture and one or more second window offsets for resizing the reference picture; and At least one of the following operations is performed: storing the encoded video bitstream, or sending the encoded video bitstream.

39. The device according to claim 20, wherein: The device is a mobile device.

40. A non-transitory computer-readable storage medium comprising: Instructions stored thereon, which when executed by one or more processors cause the one or more processors to: Get the current picture of the video data; determining one or more scaling windows identifying at least one of a top offset, a bottom offset, a left offset, or a right offset for use in determining at least one scaling ratio for at least one of the current picture or a reference picture selected for use in encoding the current picture; Determine at least one of a first size of the current picture or a second size of the reference picture, the first size comprising a first height and a first width of the current picture, and the second size comprising a second height and a second width of the reference picture; determining, based on the one or more scaling windows and at least one of the first size or the second size, one or more scaling ratios associated with adjusting one or more motion compensation parameters corresponding to at least one of the current picture or the reference picture; as well as One or more consistency windows associated with resizing at least one of the current picture or the reference picture for output are determined, wherein the one or more consistency windows are different from the one or more scaling windows.

Images