Reference picture resampling with switchable filters

Abstract

Techniques for video encoding and decoding using reference picture resampling with switchable filters are described. One example involves obtaining a current picture and a reference picture; identifying filter index information for a current block of the current picture; and determining that a first picture size value of the current picture is different from a second picture size value of the reference picture. Based on determining that the first picture size value of the current picture is different from the second picture size value of the reference picture, a resampling process is performed by replacing a current filter index identified by the filter index information with a default filter index. Additional examples may use the current filter index identified by the filter index information in a subsequent block. In various examples, the current filter index may be derived or signaled.

Description

Technical Field

[0001] The present application relates to video coding and compression. More particularly, the present application relates to systems and methods for providing reference picture resampling using switchable filters. Background Art

[0002] Many devices and systems allow video data to be processed and output for consumption. Digital video data includes a large amount of data to meet the needs of consumers and video providers. For example, consumers of video data expect the best quality video with high fidelity, resolution, frame rate, etc. Therefore, the large amount of video data required to meet these needs puts a burden on the communication networks and devices that process and store video data.

[0003] Various video codec techniques can be used to compress video data. Video coding is performed according to one or more video codec standards. For example, video codec standards include High Efficiency Video Codec (HEVC), Advanced Video Codec (AVC), Moving Picture Experts Group (MPEG) Part 2 Codec, VP9, ​​Alliance for Open Media (AOMedia) Video 1 (AV1), Basic Video Codec (EVC), etc. Video codecs typically use prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.), which utilize redundancy present in video images or sequences. An important goal of video codec technology is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality. As evolving video services become available, coding techniques with higher codec efficiency are needed. Summary of the invention

[0004] Systems and methods for providing improved video processing techniques are described herein. Some examples of video processing use video compression techniques with prediction to efficiently encode and decode video data. For example, a video encoder can perform prediction on a current block being encoded by comparing pixel values ​​in the current block with a prediction block (e.g., a block that has not yet been encoded or has been previously decoded). A prediction error (also known as a residual) can be determined as the difference between the current block and the prediction block. When a block in a picture uses one or more blocks in one or more other pictures to perform prediction (known as inter-frame prediction), a motion vector can be used to indicate the position of the prediction block relative to the current block. The video encoder can apply a transform (e.g., a discrete cosine transform (DCT) or other suitable transform) to the prediction error to generate transform coefficients. After the transform, the video encoder can quantize the transform coefficients. The quantized transform coefficients and motion vectors can be represented using syntax elements and, together with control information, form a codec representation of a video sequence. In some cases, the video encoder can perform entropy encoding and decoding on the syntax elements, which can further reduce the number of bits required to represent the syntax elements.

[0005] In some examples, sampling rate conversion can be applied to video data. Sampling rate conversion can include resampling, downsampling, upsampling, interpolation, and / or other sampling rate conversion techniques. For example, interpolation can include piecewise constant interpolation, linear interpolation, polynomial interpolation, and spline interpolation, among other interpolation techniques. Resampling can include downsampling, upsampling, and / or a combination of downsampling and upsampling, or obtaining new samples at different sampling rates.

[0006] In some examples, motion prediction can be enhanced using inter-frame prediction block merging techniques to simplify block-by-block motion data signaling. Since the displacement of objects between pictures (e.g., between the current picture and the reference picture) does not depend on the pixel grid of the image, an interpolation task is used in this case. Thus, fractional accuracy (e.g., estimating the layout between pixels in the image) can be used to more accurately capture motion. Interpolation filters can be used in video processing to achieve fractional accuracy, especially when the reference picture and the current picture being processed share the same size. Interpolation filters are used to filter image data to estimate image values ​​at fractional positions, and the fractional position data can be used with prediction or other image processing operations.

[0007] Resampling refers to taking an image with certain samples (e.g., pixels in a certain grid) and modifying those samples (e.g., creating new pixels on a different grid) while maintaining data from the original samples. As described above, resampling can include upsampling, where pixels from a lower resolution grid are used to create pixels in a higher resolution grid, and downsampling, where pixels from a higher resolution grid are used to generate pixels in a lower resolution grid. In the context of motion prediction and motion vectors described above, when the reference picture and the current picture being compared have different sizes, reference picture resampling can be used to normalize the pixel grid as part of generating an image with fractional precision. When the image processing system is configured to allow the reference picture to have a different size than the current picture, the filter indicated by the filter index can be applied to process these images for interpolation filtering (e.g., fractional precision when the reference and current pictures are the same size), but not for fractional precision in the case of reference picture resampling (e.g., fractional precision when the reference picture is different in size from the current picture). In some systems, for example, a smoothing filter for fractional precision may produce good results by interpolation when the reference picture and the current picture are the same size, but may produce poor results when used with reference picture resampling when the reference picture has a different pixel grid (e.g., a different size) than the current picture. Examples described herein improve operation of decoding and encoding devices by identifying appropriate filter values ​​and generating improved encoding and decoding results during operation of the encoding and decoding devices.

[0008] In various examples, this improvement can be achieved by identifying filter index information for a current block of a current picture and determining that a first picture size value of the current picture is different from a second picture size value of a reference picture (e.g., where reference picture resampling is used). In this case, the example uses a default filter index to replace the current filter index identified by the filter index information to perform a resampling process (e.g., without using the current filter index).

[0009] In an illustrative example, a device for processing video data is provided. The device includes a memory and one or more processors implemented in circuits. The one or more processors are configured to obtain a current picture and a reference picture. The processor is also configured to identify filter index information of a current block of the current picture and determine that a first picture size value of the current picture is different from a second picture size value of the reference picture. Based on determining that the first picture size value of the current picture is different from the second picture size value of the reference picture, the one or more processors can use a default filter index to replace the current filter index identified by the filter index information to perform a sampling rate conversion process.

[0010] In another example, a method for processing video data is provided. The method includes obtaining a current picture and a reference picture. The method also includes identifying filter index information of a current block of the current picture, and determining that a first picture size value of the current picture is different from a second picture size value of the reference picture. The method includes: based on determining that the first picture size value of the current picture is different from the second picture size value of the reference picture, replacing a current filter index identified by the filter index information with a default filter index to perform a sampling rate conversion process.

[0011] In another example, a computer-readable storage medium stores instructions that, when executed, cause one or more processors of a device for processing video data to: obtain a current picture and a reference picture; identify filter index information of a current block of the current picture, and determine that a first picture size value of the current picture is different from a second picture size value of the reference picture; and based on determining that the first picture size value of the current picture is different from the second picture size value of the reference picture, replace the current filter index identified by the filter index information with a default filter index to perform a sampling rate conversion process.

[0012] In another example, an apparatus for processing video data is provided. The apparatus includes: a component for obtaining a current picture and a reference picture; a component for identifying filter index information of a current block of the current picture; a component for determining that a first picture size value of the current picture is different from a second picture size value of the reference picture; and a component for performing a sampling rate conversion process by replacing a current filter index identified by the filter index information with a default filter index based on determining that the first picture size value of the current picture is different from the second picture size value of the reference picture.

[0013] In some examples, methods, apparatus, and instructions stored in a computer-readable storage medium include: obtaining a current picture and a reference picture; identifying filter index information of a current block of the current picture; determining that a first picture size value of the current picture is different from a second picture size value of the reference picture; and based on determining that the first picture size value of the current picture is different from the second picture size value of the reference picture, performing a sampling rate conversion process without using a current filter index identified by the filter index information.

[0014] In some examples, methods, apparatus, and instructions stored in a computer-readable storage medium include: acquiring a current picture and a reference picture; identifying filter index information of a current block of the current picture; determining that a first picture size value of the current picture is different from a second picture size value of the reference picture; and performing a sampling rate conversion process by replacing a current filter index identified by the filter index information with a default index value based on determining that the first picture size value of the current picture is different from the second picture size value of the reference picture.

[0015] In some cases, the methods, apparatus, and instructions stored in the above-mentioned computer-readable storage medium include: storing a current filter index identified by the filter index information together with the current block, and using the current filter index in motion vector candidate derivation of a subsequent block based on the current filter index being identified by the filter index information.

[0016] In some cases, the methods, apparatus, and instructions include: assigning a default value to the current filter index to temporarily replace the original value to create a default filter index when the current filter index is not used to perform a sample rate conversion process; and assigning the original value to the current filter index after the sample rate conversion process. In some cases, the default value of the current filter index indicates that the alternative switchable filtering is not applied as part of the sample rate conversion process.

[0017] In some cases, the filter index information includes neighboring block information, and wherein the current filter index is derived from the neighboring block information during merge mode motion vector candidate list derivation. In some cases, the filter index information includes adaptive motion vector resolution (AMVR) mode signaling, and wherein the current filter index is signaled via AMVR mode signaling. In some cases, the sampling rate conversion process is a motion compensation process. In some cases, the sampling rate conversion process is a resampling process. In some cases, the sampling rate conversion process is an interpolation process.

[0018] In some cases, the method, apparatus, and instructions stored in the above-mentioned computer-readable storage medium operate by using a reference picture to perform inter-frame prediction on a current block of a current picture. In some cases, the method, apparatus, and instructions stored in the above-mentioned computer-readable storage medium include determining that the current filter index is not used in the sampling rate conversion process; and in response to determining that the current filter index is not used in the sampling rate conversion process, replacing the value of the current filter index with a default value. In some cases, the method, apparatus, and instructions stored in the above-mentioned computer-readable storage medium include processing the default value to determine that the alternative switchable filter is not applied. In some examples, the alternative switchable filter is a smoothing filter.

[0019] In some cases, in response to determining that the current filter index is not used in a sample rate conversion process, the current filter index value is not signaled. In some cases, the current filter index is selected from a filter set, wherein a subset of the filter set is used to perform a sample rate conversion process, and wherein a remainder of the filter set is not available for sample rate conversion. In some cases, the current filter index is selected using a syntax element that indicates that the current filter index is prohibited from being selected from the remainder of the filter set.

[0020] In some cases, the sampling rate conversion process includes a resampling process. In this case, the method, apparatus, and instructions stored in the above-mentioned computer-readable storage medium include identifying a first filter bank for interpolation and a second filter bank for resampling, wherein the current filter index is from the second filter bank.

[0021] In some cases, the method, apparatus, and instructions stored in the above-mentioned computer-readable storage medium include: obtaining a second current picture and a second reference picture; determining that a third picture size value of the second current picture and a fourth picture size value of the second reference picture are the same size value; and based on determining that the third picture size value of the second current picture and the fourth picture size value of the second reference picture are the same size value, performing a second resampling process of the second current block using a second selected filter, wherein the second selected filter is from the first filter group.

[0022] In some cases, each filter in the first filter bank and the second filter bank is identified by an associated filter index that indicates an associated index value from a filter set for interpolation or a filter set for resampling.

[0023] In some cases, methods, apparatus, and instructions stored in the above-mentioned computer-readable storage medium include: obtaining a current filter index from the filter index by a lookup operation for a selected filter index value; wherein the selected filter index value is associated with the second filter group; and wherein the association between the selected filter index value and the second filter group is based on determining that the first picture size value is different from the second picture size value.

[0024] In some cases, methods, apparatus, and instructions stored in the computer-readable storage medium operate, wherein determining that a first picture size value of a current picture is different from a second picture size value of a reference picture includes determining at least one of the following: a current picture width is different from a reference picture width; and a current picture height is different from a reference picture height.

[0025] In some cases, the method, apparatus, and instructions stored in the above-mentioned computer-readable storage medium include: generating a coded video bitstream including a current picture and a reference picture. In some examples, the method, apparatus, and instructions stored in the above-mentioned computer-readable storage medium include sending the coded video bitstream to a decoding device, the coded video bitstream is sent together with signaling information, and the signaling information includes a default filter index. In some examples, the method, apparatus, and instructions stored in the above-mentioned computer-readable storage medium include storing the coded video bitstream.

[0026] In some cases, the method, apparatus, and instructions stored in the above-mentioned computer-readable storage medium include: obtaining a coded video bitstream including a current picture and a reference picture; identifying signaling information associated with the coded video bitstream, the signaling information including filter index information; and decoding a current block of the current picture from the coded video bitstream. In some examples, decoding the current block of the current picture from the coded video bitstream includes reconstructing the current block based on a default filter index.

[0027] In some aspects, the apparatus for processing video data may include a decoding device or device and / or an encoding device or device. The apparatus for processing video data may be a camera, a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, the apparatus for processing video data includes one or more cameras for capturing one or more images. In some aspects, the apparatus for processing video data includes a display for displaying one or more images, notifications, and / or other displayable data.

[0028] The above-described aspects relating to any of the above-described methods, apparatuses, and computer-readable media may be used alone or in any suitable combination.

[0029] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to the entire specification of this patent, any or all of the drawings, and appropriate portions of each claim.

[0030] The foregoing together with other features and embodiments will become more apparent with reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The illustrative embodiments of the present application are described in detail below with reference to the accompanying drawings:

[0032] Figure 1 is an example block diagram illustrating a system including an encoding device and a decoding device according to some examples;

[0033] Figure 2A is a conceptual diagram illustrating example spatial neighboring motion vector candidates for merge mode according to some examples;

[0034] Figure 2B is a conceptual diagram illustrating example spatial neighboring motion vector candidates for an advanced motion vector prediction (AMVP) mode according to some examples;

[0035] Figure 3 is a flow chart illustrating a technique for processing video data;

[0036] Figure 4 is a diagram illustrating aspects of reference picture resampling according to some examples;

[0037] Figure 5 is a flow chart illustrating aspects of a method for reference picture resampling according to some examples;

[0038] Figure 6 is a flow chart illustrating aspects of reference picture resampling according to some examples;

[0039] Figure 7 is a block diagram illustrating an example video encoding device according to some examples; and

[0040] Figure 8 is a block diagram illustrating an example video decoding device according to some examples. DETAILED DESCRIPTION

[0041] Specific aspects and embodiments of the present disclosure are provided below. It will be appreciated by those skilled in the art that some of these aspects and embodiments may be applied independently and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that various embodiments may be practiced without these specific details. The accompanying drawings and description are not intended to be restrictive.

[0042] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the exemplary embodiments will provide an enabling description for implementing the exemplary embodiments for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0043] Video codec devices implement video compression techniques to efficiently encode and decode video data. Video compression techniques may include applying different prediction modes, including spatial prediction (e.g., intra-frame prediction or intra-prediction), temporal prediction (e.g., inter-frame prediction or inter-prediction), inter-layer prediction (across different layers of video data, and / or other prediction techniques that reduce or remove inherent redundancy in video sequences. A video encoder may partition each frame of an original video sequence into rectangular regions called video blocks or codec units (described in more detail below). These video blocks may be encoded using specific prediction modes.

[0044] The video block may be divided into one or more groups of smaller blocks in one or more ways. Blocks may include codec tree blocks, prediction blocks, transform blocks, and / or other suitable blocks. Unless otherwise specified, general references to "blocks" may refer to such video blocks (e.g., codec tree blocks, codec blocks, prediction blocks, transform blocks, or other suitable blocks or sub-blocks, as understood by those skilled in the art). Further, each of these blocks may also be interchangeably referred to herein as a "unit" (e.g., codec tree unit (CTU), codec unit, prediction unit (PU), transform unit (TU), etc.). In some cases, a unit may indicate a codec logic unit encoded in a bitstream, and a block may indicate a portion of a video frame buffer to which a process is directed.

[0045] For inter-frame prediction mode, the video encoder can search for a block similar to the block being encoded in a frame (or picture) located in another time position, which is called a reference frame or reference picture. The video encoder can limit the search to a certain spatial displacement from the block to be encoded. A two-dimensional (2D) motion vector including a horizontal displacement component and a vertical displacement component can be used to locate the best match. For intra-frame prediction mode, the video encoder can use spatial prediction techniques to form a prediction block based on data of neighboring blocks that have been previously encoded in the same picture.

[0046] The video encoder can determine a prediction error. For example, the prediction can be determined as the difference between the pixel values ​​in the block being encoded and the predicted block. The prediction error can also be referred to as a residual. The video encoder can also apply a transform (e.g., a discrete cosine transform (DCT) or other appropriate transform) to the prediction error to generate a transform coefficient. After the transform, the video encoder can quantize the transform coefficients. Syntax elements can be used to represent the quantized transform coefficients and motion vectors, and together with the control information form a coded representation of the video sequence. In some examples, the video encoder can entropy encode and decode the syntax elements to further reduce the number of bits required for their representation.

[0047] The video decoder can use the syntax elements and control information discussed above to construct prediction data (e.g., a prediction block) for decoding the current frame. For example, the video decoder can add the prediction block and the compressed prediction error. The video decoder can use the quantized coefficients to determine the compressed prediction error by weighting the transform basis function. The difference between the reconstructed frame and the original frame is called the reconstruction error.

[0048] In some examples, sampling rate conversion can be applied to video data. Sampling rate conversion can include resampling, downsampling, upsampling, interpolation and / or other sampling rate conversion techniques. For example, interpolation can include piecewise constant interpolation, linear interpolation, polynomial interpolation and spline interpolation, as well as other interpolation techniques. Resampling can include downsampling, upsampling and / or a combination of downsampling and upsampling, or obtaining new samples at different sampling rates.

[0049] In some systems, inter-frame prediction block merging techniques can be used to enhance motion prediction to simplify block-by-block motion data signaling. In the case where the displacement of objects between pictures is independent of the pixel grid of the image, interpolation tasks are used. Therefore, fractional precision (e.g., estimating the positioning between arrangements in the image) can be used to capture motion more accurately. This fractional precision can make it possible to infer motion data from previously decoded blocks. For interpolation and fractional reference picture samples, high precision can improve filtering. The example system can support motion vectors with quarter pixel precision, eighth pixel precision, sixteenth pixel precision, or other such precisions depending on the system. In some examples, luminance and chrominance components can have different fractional precisions in the same system. In this case, interpolation filters can be used to process images. In addition, multiple different interpolation filters can be used in different situations (e.g., based on motion vector values). Selecting between available different interpolation filters is referred to as interpolation filter switching in this article.

[0050] Resampling refers to taking pixel values ​​from an image and "resampling" data from the pixel values ​​to generate new pixel values ​​for a new image. As described above, resampling can include downsampling and upsampling, in which pixel values ​​from a higher resolution image are used to generate a lower resolution image. In upsampling, pixel values ​​from a lower resolution image are used to generate a higher resolution image. In the context of motion prediction described above, reference images and current images of different sizes (e.g., resolutions) can be used to generate motion vectors and perform motion prediction. In this case, resampling can be used to manage the scaling difference between the reference picture and the current picture. In addition, when the size of the reference picture and the current picture is different, the filter can be indicated by a filter index, and the filter index is applicable to interpolation filtering without resampling and is not applicable to interpolation filtering for reference picture resampling. In some image processing systems, for example, when a smoothing filter used for interpolation is used for reference picture resampling, undesirable results may be produced. The examples described herein improve the operation of decoding and encoding devices by identifying appropriate filter values ​​and generating improved encoding and decoding results during the operation of the encoding and decoding devices. In one example, the improvement is achieved by identifying filter index information for a current block of a current picture and determining that a first picture size value of the current picture is different from a second picture size value of a reference picture (e.g., the reference picture resampling is used). In this case, the example uses a default filter index to replace the current filter index identified by the filter index information to perform a resampling process (e.g., the current filter index is not used). Additional aspects of various examples are described below.

[0051] The techniques described herein can be applied to any of the existing video codecs (e.g., High Efficiency Video Codec (HEVC), Advanced Video Codec (AVC), or other suitable existing video codecs), and / or can be used as an effective codec tool for the following video codec standards being developed and / or future video codec standards: such as, for example, Versatile Video Codec (VVC)), Joint Exploration Model (JEM), and / or other video codec standards under development or to be developed.

[0052] Figure 11 is a block diagram showing an example of a system 100 including an encoding device 104 and a decoding device 112. The encoding device 104 may be part of a source device, and the decoding device 112 may be part of a receiving device. The source device and / or the receiving device may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the source device and the receiving device may include one or more wireless transceivers for wireless communication. The coding and decoding techniques described herein may be applied to video coding and decoding in various multimedia applications, including streaming video transmission (e.g., via the Internet), television broadcasting or transmission, encoding of digital video for storage on a data storage medium, decoding of digital video for storage on a data storage medium, or other applications. In some examples, the system 100 may support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, gaming, and / or video telephony.

[0053] The encoding device 104 (or encoder) can be used to encode video data using a video codec standard or protocol to generate a coded video bitstream. Examples of video codec standards include ITU-T H.261, ISO / IEC MPEG-1 Vision, ITU-T H.262 or ISO / IEC MPEG-2 Vision, ITU-T H.263, ISO / IEC MPEG-4 Vision, ITU-TH.264 (also known as ISO / IEC MPEG-4 AVC), including its scalable video codec (SVC) and multi-view video codec (MVC) extensions, and high-efficiency video codec (HEVC) or ITU-T H.265. There are various extensions of HEVC to handle multi-layer video codecs, including range and screen content codec extensions, 3D video codec (3D-HEVC) and multi-view extensions (MV-HEVC) and scalable extensions (SHVC). HEVC and its extensions are developed by the Joint Collaboration Team on Video Coding and Decoding (JCT-VC) and the Joint Collaboration Team on 3D Video Codec Extension Development (JCT-3V) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). MPEG and ITU-T VCEG also formed the Joint Exploration Video Team (JVET) to explore new codec tools for the next generation of video codec standards. The reference software is called JEM (Joint Exploration Model). The new video codec standard being developed by JVET is called Versatile Video Codec (VVC).

[0054] HEVC was finalized in 2013 by the Joint Collaborative Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG). The collaborative team formed by MPEG and VCEG of ITU-T Study Group 16, the Joint Video Experts Team (JVET), is working on the new VVC video codec standard. The goal of VVC is to provide significant improvements in the compression performance of the existing HEVC standard, helping to deploy high-quality video services and emerging applications (e.g., 360° immersive multimedia, high dynamic range (HDR) video, etc.).

[0055] The multiple embodiments described herein provide examples using the JEM model, VVC, HEVC standards, and / or their extensions. However, as described above, the techniques and systems described herein may also be applicable to other codec standards, such as AVC, MPEG, JPEG (or other codec standards for still images), their extensions, or other suitable codec standards that are already available or not yet available or developed. Therefore, although the techniques and systems described herein may be described with reference to a specific video codec standard, it will be understood by those of ordinary skill in the art that the description should not be interpreted as being applicable only to that specific standard.

[0056] refer to Figure 1 , video source 102 may provide video data to encoding device 104. Video source 102 may be part of a source device, or may be part of a device other than a source device. Video source 102 may include a video capture device (e.g., a video camera, a camera phone, a video phone, etc.), a video archive containing stored video, a video server or content provider that provides video data, a video feed interface that receives video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of these sources, or any other suitable video source.

[0057] The video data from the video source 102 may include one or more input pictures or frames. In some cases, the picture or frame is a still image that is part of the video. In some examples, the data from the video source 102 may be a still image that is not part of the video. The encoder engine 106 (or encoder) of the encoding device 104 encodes the video data to generate a coded video bitstream. In some examples, the coded video bitstream (or "video bitstream" or "bitstream") is a series of one or more coded video sequences. The coded video sequence (CVS) includes a series of access units (AUs) that start with an AU having a random access point picture in the base layer and having certain attributes, until but not including the next AU having a random access point picture in the base layer and having certain attributes. For example, certain attributes of the random access point picture that starts the CVS may include a RASL flag equal to 1 (e.g., NoRaslOutputFlag). Otherwise, the random access point picture (with a RASL flag equal to 0) does not start the CVS. The access unit (AU) includes one or more coded pictures and control information corresponding to the coded pictures that share the same output time. The coded slices of the picture are encapsulated into data units called network abstraction layer (NAL) units at the bitstream level. For example, an HEVC video bitstream may include one or more CVSs containing NAL units. Each of the NAL units has a NAL unit header. In one example, the header of H.264 / AVC (except for multi-layer extensions) is one byte and the header of HEVC is two bytes. The syntax elements in the NAL unit header use specified bits, so they are visible to various systems and transport layers, such as transport streams, real-time transport (RTP) protocols, file formats, etc.

[0058] There are two types of NAL units in the HEVC standard, including video codec layer (VCL) NAL units and non-VCL NAL units. The VCL NAL unit includes a slice or slice segment of the coded picture data (described below), while the non-VCL NAL unit includes control information related to one or more coded pictures. In some cases, the NAL unit can be referred to as a packet. The HEVC AU includes a VCL NAL unit containing the coded picture data and a non-VCL NAL unit corresponding to the coded picture data (if any).

[0059] The NAL unit may contain a sequence of bits (e.g., a coded video bitstream, a CVS of a bitstream, etc.) that form a coded representation of video data, such as a coded representation of a picture in a video. The encoder engine 106 generates a coded representation of a picture by dividing each picture into multiple slices. The slice is independent of other slices so that the information in the slice is encoded and decoded without relying on the data of other slices in the same picture. The slice includes one or more slice segments, which include independent slice segments and one or more dependent slice segments that depend on the previous slice segment (if any). The slice is then divided into codec tree blocks (CTBs) of luminance samples and chrominance samples. The CTB of the luminance sample and one or more CTBs of the chrominance sample, together with the syntax of the sample, are called a codec tree unit (CTU). CTU is the basic processing unit of HEVC encoding. CTU can be split into multiple codec units (CUs) of different sizes. CU contains arrays of luminance and chrominance samples called codec blocks (CBs).

[0060] Luma and chroma CBs can be further split into prediction blocks (PBs). A PB is a block of samples of a luminance component or a chroma component that uses the same motion parameters for inter-frame prediction or intra-block copy prediction (when available or enabled). A luminance PB and one or more chroma PBs, together with the associated syntax, form a prediction unit (PU). For inter-frame prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) is signaled in the bitstream of each PU, which is used for inter-frame prediction of the luminance PB and one or more chroma PBs. Motion parameters can also be referred to as motion information. A CB can also be divided into one or more transform blocks (TBs). A TB represents a square block of samples of a color component, to which the same two-dimensional transform is applied to encode and decode the prediction residual signal. A transform unit (TU) represents a TB of luminance and chroma samples, and corresponding syntax elements.

[0061] The size of the CU corresponds to the size of the codec mode and may be square in shape. For example, the size of the CU may be 8×8 samples, 16×16 samples, 32×32 samples, 64×64 samples, or any other appropriate size up to the size of the corresponding CTU. In this document, the phrase "N×N" is used to represent the pixel dimensions of a video block in vertical and horizontal dimensions (e.g., 8 pixels × 8 pixels). The pixels in a block may be arranged in rows and columns. In some embodiments, the number of pixels in the horizontal direction of the block may be different from the number of pixels in the vertical direction. The syntax data associated with the CU may describe, for example, the partitioning of the CU into one or more PUs. Depending on whether the CU is encoded in intra-prediction mode or in inter-prediction mode, the partitioning mode may be different. The PU may be partitioned into a non-square shape. The syntax data associated with the CU may also describe, for example, the partitioning of the CU into one or more TUs according to the CTU. The shape of the TU may be square or non-square.

[0062] According to the HEVC standard, transform units (TUs) may be used for transforming. TUs may vary with different CUs. The size of a TU may be determined based on the size of a PU within a given CU. The size of a TU may be the same as or smaller than the size of a PU. In some examples, a quadtree structure called a residual quadtree (RQT) may be used to subdivide the residual samples corresponding to a CU into smaller units. The leaf nodes of the RQT may correspond to TUs. The pixel difference values ​​associated with the TU may be transformed to generate transform coefficients. The transform coefficients may then be quantized by the encoder engine 106.

[0063] Once the picture of the video data is partitioned into CUs, the encoder engine 106 predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (as described below). For each CU, the prediction mode can be signaled in the bitstream using syntax data. The prediction mode may include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction exploits the correlation between spatially adjacent samples within the picture. For example, in the case of intra-frame prediction, each PU is predicted based on adjacent image data in the same picture using, for example, DC prediction for seeking the average value of the PU, plane prediction for fitting a plane surface to the PU, directional prediction inferred from adjacent data, or any other suitable type of prediction. Inter-frame prediction uses temporal correlation between pictures to derive motion compensated prediction of blocks of image samples. For example, in the case of inter-frame prediction, each PU is predicted using motion compensated prediction based on image data in one or more reference pictures (before or after the current picture in output order). A decision on whether to encode or decode a picture area using inter-picture prediction or intra-picture prediction can be made at, for example, the CU level.

[0064] In some examples, one or more slices of a picture are assigned a slice type. Slice types include I slices, P slices, and B slices. An I slice (intra-frame, independently decodable) is a slice of a picture that is encoded and decoded only by intra-frame prediction and is therefore independently decodable because an I slice only requires data within the frame to predict any prediction unit or prediction block of the slice. A P slice (unidirectional predicted frame) is a slice of a picture that can be encoded and decoded using intra-frame prediction and unidirectional inter-frame prediction. Each prediction unit or prediction block within a P slice is encoded and decoded by intra-frame prediction or inter-frame prediction. When inter-frame prediction is applied, the prediction unit or prediction block is predicted only by one reference picture, so the reference samples come from only one reference area of ​​a frame. A B slice (bidirectional predicted frame) is a slice of a picture that can be encoded and decoded using intra-frame prediction as well as using inter-frame prediction (e.g., bidirectional prediction or unidirectional prediction). The prediction unit or prediction block of a B slice may be bidirectionally predicted based on two reference pictures, where each picture provides a reference region, and the sample sets of the two reference regions are weighted (e.g., with equal weights or with different weights) to generate a prediction signal for the bidirectional prediction block. As described above, the slices of a picture are independently coded. In some cases, a picture may be coded as exactly one slice.

[0065] As described above, intra-picture prediction exploits the correlation between spatially adjacent samples within a picture. Inter-picture prediction uses the temporal correlation between pictures in order to derive motion compensated prediction of blocks of image samples. Using a translational motion model, the block position in a previously decoded picture (reference picture) is indicated by a motion vector (Δx, Δy), where Δx specifies the horizontal displacement of the reference block relative to the position of the current block, and Δy specifies the vertical displacement of the reference block relative to the position of the current block. In some cases, the motion vector (Δx, Δy) may be integer sample precision (also known as integer precision), in which case the motion vector points to an integer pixel grid (or integer pixel sampling grid) of a reference frame. In some cases, the motion vector (Δx, Δy) may have fractional sample precision (also known as fractional pixel precision or non-integer precision) to more accurately capture the motion of a potential object without being limited to the integer pixel grid of a reference frame. The precision of the motion vector may be represented by the quantization level of the motion vector. For example, the quantization level may be integer precision (e.g., 1-pixel) or fractional pixel precision (e.g., 1 / 4-pixel, 1 / 2-pixel, or other sub-pixel values). When the corresponding motion vector has fractional sample precision, interpolation is applied to the reference picture to derive the prediction signal. For example, the samples available at integer positions (e.g., using one or more interpolation filters) can be filtered to estimate the values ​​at the fractional positions. The previously decoded reference picture is indicated by the reference index (refIdx) of the reference picture list. The motion vector and the reference index may be referred to as motion parameters. Two types of inter-picture prediction may be performed, including unidirectional prediction and bidirectional prediction.

[0066] In the case of inter-frame prediction using bi-prediction, two sets of motion parameters (Δx 0 ,y 0 ,refIdx 0 and Δx 1 ,y 1 ,refIdx 1 ) to generate two motion compensated predictions (based on the same reference picture or possibly on different reference pictures). For example, for bi-prediction, two motion compensated prediction signals are used for each prediction block, and a B prediction unit is generated. The two motion compensated predictions are then combined to obtain the final motion compensated prediction. For example, the two motion compensated predictions can be combined by averaging. In another example, weighted prediction can be used, in which case different weights can be applied to each motion compensated prediction. The reference pictures that can be used in bi-prediction are stored in two different lists, which are denoted as list 0 and list 1, respectively. Motion parameters can be derived using a motion estimation process at the encoder.

[0067] In the case of inter-frame prediction using uni-prediction, the motion parameter set (Δx 0 ,y 0 ,refIdx 0 ) to generate motion compensation prediction according to the reference picture. For example, in the case of single prediction, each prediction block uses at most one motion compensation prediction signal and generates a P prediction unit.

[0068] The PU may include data related to the prediction process (e.g., motion parameters or other suitable data). For example, when the PU is encoded using intra prediction, the PU may include data describing the intra prediction mode of the PU. As another example, when the PU is encoded using inter prediction, the PU may include data defining the motion vector of the PU. The data defining the motion vector of the PU may describe, for example, the horizontal component of the motion vector (Δx), the vertical component of the motion vector (Δy), the resolution of the motion vector (e.g., integer precision, quarter-pixel precision, or eighth-pixel precision), the reference picture to which the motion vector points, the reference index, the reference picture list of the motion vector (e.g., list 0, list 1, or list C), or any combination of the above.

[0069] The encoding device 104 may then perform transformation and quantization. For example, after prediction, the encoder engine 106 may calculate a residual value corresponding to the PU. The residual value may include a pixel difference value between a current pixel block (PU) being encoded and a prediction block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a prediction block (e.g., using inter-frame prediction or intra-frame prediction), the encoder engine 106 may generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantize the difference between the pixel values ​​of the current block and the pixel values ​​of the prediction block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or a pixel value array). In this example, the residual block is a two-dimensional representation of a pixel value.

[0070] Any residual data remaining after performing the prediction is transformed using a block transform, which can be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., sizes 32×32, 16×16, 8×8, 4×4, or other suitable sizes) may be applied to the residual data in each CU. In some embodiments, a TU may be used for a transform and quantization process implemented by the encoder engine 106. A given CU with one or more PUs may also include one or more TUs. As described in further detail below, a block transform may be used to transform the residual values ​​into transform coefficients, and the TUs may be quantized and scanned to produce serialized transform coefficients for entropy coding and decoding.

[0071] In some embodiments, after intra-prediction or inter-prediction encoding and decoding using the PU of the CU, the encoder engine 106 can calculate the residual data of the TU of the CU. The PU may include pixel data in the spatial domain (or pixel domain). After applying the block transform, the TU may include coefficients in the transform domain. As described above, the residual data may correspond to the pixel difference between the pixel of the uncoded picture and the predicted value corresponding to the PU. The encoder engine 106 may form a TU including the residual data of the CU, and then the TU may be transformed to generate transform coefficients for the CU.

[0072] The encoder engine 106 may quantize the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient having an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

[0073] Once quantization is performed, the encoded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction mode, motion vector, block vector, etc.), segmentation information, and any other suitable data, such as other syntax data. Then, the different elements of the encoded video bitstream can be entropy encoded by the encoder engine 106. In some examples, the encoder engine 106 can scan the quantized transform coefficients using a predefined scan order to generate a serialized vector that can be entropy encoded. In some examples, the encoder engine 106 can perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoder engine 106 can entropy encode the vector. For example, the encoder engine 106 can use context adaptive variable length coding, context adaptive binary arithmetic coding, grammar-based context adaptive binary arithmetic coding, probability interval segmentation entropy coding, or other suitable entropy coding techniques.

[0074] The output 110 of the encoding device 104 may send the NAL units that make up the encoded video bitstream data to a decoding device 112 of a receiving device via a communication link 120. The input 114 of the decoding device 112 may receive the NAL units. The communication link 120 may include a channel provided by a wireless network, a wired network, or a combination of wired and wireless networks. The wireless network may include any wireless interface or combination of wireless interfaces and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, a WiFi, or a wireless network). TM , Radio Frequency (RF), UWB, WiFi Direct, Cellular, Long Term Evolution (LTE), WiMax TM The wired network may include any wired interface (e.g., optical fiber, Ethernet, power line Ethernet, Ethernet based on coaxial cable, digital signal line (DSL), etc.). The wired and / or wireless network may be implemented by using various equipment, such as base stations, routers, access points, bridges, gateways, switches, etc. The encoded video bitstream data may be modulated according to a communication standard such as a wireless communication protocol and sent to a receiving device.

[0075] In some examples, encoding device 104 may store the encoded video bitstream data in memory 108. Output 110 may retrieve the encoded video bitstream data from encoder engine 106 or from memory 108. Memory 108 may include a variety of distributed or locally accessed data storage media. For example, memory 108 may include a hard drive, a storage disk, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0076] The input 114 of the decoding device 112 receives the encoded video bitstream data and can provide the video bitstream data to the decoder engine 116, or to the storage device 118 for later use by the decoder engine 116. The decoder engine 116 can decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting the elements of one or more encoded and decoded video sequences that make up the encoded video data. The decoder engine 116 can then rescale the encoded video bitstream data and perform an inverse transform. The residual data is then passed to the prediction stage of the decoder engine 116. The decoder engine 116 then predicts the pixel block (e.g., PU). In some examples, the prediction is added to the output (residual data) of the inverse transform.

[0077] The decoding device 112 may output the decoded video to a video destination device 122, which may include a display or other output device for displaying the decoded video data to a consumer of the content. In some aspects, the video destination device 122 may be part of a receiving device that includes the decoding device 112. In some aspects, the video destination device 122 may be part of a separate device other than the receiving device.

[0078] In some embodiments, the video encoding device 104 and / or the video decoding device 112 can be integrated with the audio encoding device and the audio decoding device, respectively. The video encoding device 104 and / or the video decoding device 112 may also include other hardware or software necessary to implement the above-mentioned coding and decoding techniques, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination of the above. The video encoding device 104 and the video decoding device 112 can be integrated as part of a combined encoder / decoder (codec) in the corresponding device. The following reference Figure 7 An example of specific details of the encoding device 104 is described below. Figure 8 An example of specific details of the decoding device 112 is described.

[0079] Extensions to the HEVC standard include a multi-view video codec extension called MV-HEVC and a scalable video codec extension called SHVC. The MV-HEVC and SHVC extensions share the concept of layered coding, where different layers are included in the coded video bitstream. Each layer in the coded video sequence is addressed by a unique layer identifier (ID). The layer ID may be present in the header of the NAL unit to identify the layer associated with the NAL unit. In MV-HEVC, different layers may represent different views of the same scene in the video bitstream. In SHVC, different scalable layers are provided to represent the video bitstream at different spatial resolutions (or picture resolutions) or different reconstruction fidelity. The scalable layer may include a base layer (layer ID = 0) and one or more enhancement layers (layer ID = 1, 2, ..., n). The base layer may conform to the profile of the first version of HEVC and represent the lowest available layer in the bitstream. Compared to the base layer, the enhancement layer has increased spatial resolution, temporal resolution or frame rate and / or reconstruction fidelity (or quality). The enhancement layer is hierarchically constructed and may (or may not) depend on a lower layer. In some examples, different layers may be encoded using a single standard codec (e.g., all layers may be encoded using HEVC, SHVC, or other codec standards). In some examples, different layers may be encoded using multiple standard codecs. For example, the base layer may be encoded using AVC, while one or more enhancement layers may be encoded using SHVC and / or MV-HEVC extensions of the HEVC standard.

[0080] Typically, a layer consists of a set of VCL NAL units and a corresponding set of non-VCL NAL units. NAL units are assigned specific layer ID values. Layers can be hierarchical, which means that layers can depend on lower layers. A layer set refers to a set of self-contained layers represented in a bitstream, which means that layers within a layer set can depend on other layers in the layer set during decoding, but do not depend on any other layers for decoding. Therefore, the layers in a layer set can form an independent bitstream that can represent video content. A set of layers in a layer set can be obtained from another bitstream by running a sub-bitstream extraction process. A layer set can correspond to a set of layers to be decoded when the decoder wants to operate according to certain parameters.

[0081] As mentioned above, the HEVC bitstream includes a set of NAL units, including VCL NAL units and non-VCL NAL units. The VCL NAL unit includes coded and decoded picture data that forms a coded and decoded video bitstream. For example, the bit sequence that forms the coded and decoded video bitstream is present in the VCL NAL unit. In addition to other information, the non-VCL NAL unit may also contain a parameter set with advanced information related to the coded video bitstream. For example, the parameter set may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). Examples of the goals of the parameter set include bit rate efficiency, error resilience, and providing a system layer interface. Each slice references a single active PPS, SPS, and VPS to access the information that the decoding device 112 will use to decode the slice. An identifier (ID) can be encoded and decoded for each parameter set, including a VPS ID, an SPS ID, and a PPS ID. The SPS includes an SPS ID and a VPS ID. The PPS includes a PPS ID and an SPS ID. Each slice header includes a PPS ID. By using the ID, the activation parameter set for a given slice can be identified.

[0082] The PPS includes information applied to all slices in a given picture. Therefore, all slices in the picture refer to the same PPS. Slices in different pictures can also refer to the same PPS. The SPS includes information applied to all pictures in the same coded video sequence (CVS) or bitstream. As mentioned above, the coded video sequence is a series of access units (AUs) that start with an access unit having a random access point picture (e.g., an instantaneous decoding reference (IDR) picture or a broken link access (BLA) picture, or other appropriate random access point picture) and certain attributes (as described above) in the base layer, until but not including the next AU having a random access point picture in the base layer and having certain attributes (or at the end of the bitstream). The information in the SPS may not change with the pictures in the coded video sequence. The pictures in the coded video sequence can use the same SPS. The VPS includes information applied to all layers in the coded video sequence or bitstream. The VPS includes a syntax structure having syntax elements applied to the entire coded video sequence. In some embodiments, the VPS, SPS or PPS can be transmitted in-band with the encoded bitstream. In some embodiments, the VPS, SPS, or PPS may be transmitted out-of-band in a transmission separate from the NAL units containing the encoded video data.

[0083] The video bitstream may also include a supplemental enhancement information (SEI) message. For example, a SEI NAL unit may be part of the video bitstream. In some examples, the SEI message may contain information that is not required for the decoding process. For example, the information in the SEI message may not be necessary for the decoder to decode the video picture of the bitstream, but the decoder may use this information to improve the display or processing of the picture (e.g., decoded output). The information in the SEI message may be embedded metadata. In an illustrative example, the information in the SEI message may be used by a decoder-side entity to improve the visibility of the content. In some cases, certain application standards may require the presence of such SEI messages in the bitstream so that all devices that comply with the application standard can be improved in quality (e.g., the carrying of frame encapsulation SEI messages for frame-compatible planar stereoscopic 3DTV video formats, where SEI messages are carried for each frame of the video, processing recovery point SEI messages, using panoramic scanning to scan rectangular SEI messages in DVB, among many other examples).

[0084] In some embodiments, the video encoding device 104 and / or the video decoding device 112 can be integrated with the audio encoding device and the audio decoding device, respectively. The video encoding device 104 and / or the video decoding device 112 may also include other hardware or software necessary to implement the above-mentioned coding and decoding techniques, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination of the above. The video encoding device 104 and the video decoding device 112 can be integrated as part of a combined encoder / decoder (codec) in the corresponding device. The following reference Figure 7 An example of specific details of the encoding device 104 is described below. Figure 8 An example describing specific details of the decoding device 112.

[0085] Figure 1The example system shown in is an illustrative example that can be used in this article. The technology of processing video data using the technology described in this article can be performed by any digital video encoding and / or decoding device. Although the technology of the present disclosure is usually performed by a video encoding device or a video decoding device, these technologies can also be performed by a combined video encoder-decoder (commonly referred to as a "codec"). Moreover, the technology of the present disclosure can also be performed by a video preprocessor. The source device and the receiving device are only examples of such codec devices, wherein the source device generates coded and decoded video data for transmission to the receiving device. In some examples, the source device and the receiving device can operate in a substantially symmetrical manner so that each of the above-mentioned devices includes a video encoding and decoding component. Thus, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting or video telephony.

[0086] As described above, for each block, a motion information set (also referred to herein as motion parameters) is available. The motion information set contains motion information for a forward prediction direction and a backward prediction direction. The forward prediction direction and the backward prediction direction are the two prediction directions of the bidirectional prediction mode, in which case the terms "forward" and "backward" do not necessarily have a geometric meaning. Instead, "forward" and "backward" correspond to reference picture list 0 (RefPicList0 or L0) and reference picture list 1 (RefPicList1 or L1) of the current picture. In some examples, when only one reference picture list is available for a picture or slice, only RefPicList0 is available and the motion information for each block of the slice is forward.

[0087] In some cases, a motion vector is used in a coding process (e.g., motion compensation) together with its reference index. Such a motion vector with an associated reference index is represented as a unidirectional prediction set of motion information. For each prediction direction, the motion information may contain a reference index and a motion vector. In some cases, for simplicity, the motion vector itself may be referenced in a manner assuming that it has an associated reference index. The reference index is used to identify the reference picture in the current reference picture list (RefPicList0 or RefPicList1). The motion vector has horizontal and vertical components. The horizontal and vertical components provide an offset from the coordinate position in the current picture to the coordinate in the reference picture identified by the reference index. For example, a reference index may indicate a specific reference picture that should be used for a block in the current picture, and a motion vector may indicate where the best matching block in the reference picture (the block that best matches the current block) is in the reference picture.

[0088] The picture order count (POC) can be used in video codec standards to identify the display order of pictures. Although in some cases, two pictures in a coded video sequence can have the same POC value, this usually does not happen in a coded video sequence. When there are multiple coded video sequences in the bitstream, pictures with the same POC value will be closer to each other in decoding order. The POC value of a picture can be used for reference picture list construction, derivation of reference picture sets such as in HEVC, and motion vector scaling.

[0089] In H.264 / AVC, each inter-frame macroblock (MB) can be partitioned in four different ways, including: one 16×16MB partition; two 16×8MB partitions; two 8×16MB partitions; and four 8×8MB partitions. Different MB partitions in one MB can have different reference index values ​​for each direction (RefPicList0 or RefPicList1). In some cases, when the MB is not partitioned into four 8×8MB partitions, it can have only one motion vector for each MB partition in each direction. In some cases, when the MB is partitioned into four 8×8MB partitions, each 8×8MB partition can be further partitioned into sub-blocks, in which case each sub-block can have a different motion vector in each direction. In some examples, there are four different ways to get sub-blocks from an 8×8MB partition, including: one 8×8 sub-block; two 8×4 sub-blocks; two 4×8 sub-blocks; and four 4×4 sub-blocks. Each sub-block can have a different motion vector in each direction. Therefore, the motion vector exists at a level equal to or higher than the sub-block.

[0090] In AVC, temporal direct mode can be enabled at the MB level or at the MB partition level for skip and / or direct mode in B slices. For each MB partition, the motion vector of the block in the RefPicList1[0] of the current block that is co-located with the current MB partition is used to derive the motion vector. Each motion vector in the co-located block is scaled based on the POC distance.

[0091] A spatial direct mode can also be implemented in AVC. For example, in AVC, the direct mode can also predict motion information based on spatially adjacent objects.

[0092] In HEVC, the largest codec unit in a slice is called a codec tree block (CTB). The CTB contains a quadtree whose nodes are codec units. In the HEVC main profile, the size of the CTB can range from 16×16 to 64×64. In some cases, 8×8 CTB size can be supported. The codec unit (CU) can be the same size as the CTB and as small as 8×8. In some cases, each codec unit is encoded and decoded using one mode. When a CU is inter-coded, the CU can be further partitioned into 2 or 4 prediction units (PUs), or it can become just one PU when further partitioning is not applied. When two PUs exist in a CU, it can be a rectangle with a size half of the size of the CU, or it can be two rectangles with sizes of 1 / 4 and 3 / 4 of the size of the CU, respectively.

[0093] When a CU is inter-coded, there is a motion information set for each PU. In addition, each PU is coded with a unique inter prediction mode to derive the motion information set.

[0094] For motion prediction in HEVC, there are two inter prediction modes, including merge mode and advanced motion vector prediction (AMVP) mode for prediction units (PUs). Skipping is considered a special case of merging. In AMVP or merge mode, a motion vector (MV) candidate list is maintained for multiple motion vector predictors. The motion vector of the current PU and the reference index in merge mode are generated by obtaining a candidate from the MV candidate list. In some examples, as described below, one or more stored local illumination compensation (LIC) flags may be included in the MV candidate list along with the stored motion vector.

[0095] In an example where the MV candidate list is used for motion prediction (and, where applicable, illumination compensation) of a block, the MV candidate list may be constructed separately by an encoding device and a decoding device. For example, the encoding device may generate the MV candidate list when encoding the block, and the decoding device may generate the MV candidate list when decoding the block. Information related to the motion information candidates in the MV candidate list (e.g., information related to one or more motion vectors, information related to one or more LIC flags stored in the MV candidate list in some cases, and / or other information) may be signaled between the encoding device and the decoding device. For example, in merge mode, the index value of the stored motion information candidate may be signaled from the encoding device to the decoding device (e.g., in the following syntax structures, such as a picture parameter set (PPS), a sequence parameter set (SPS), a video parameter set (VPS), a slice header, a supplemental enhancement information (SEI) message sent in a video bitstream or sent separately from a video bitstream, and / or other signaling). The decoding device may construct the MV candidate list and use the signaled reference or index to obtain one or more motion information candidates from the constructed MV candidate list for motion compensated prediction. For example, the decoding device 112 may construct an MV candidate list and use the motion vector (and in some cases the LIC flag) according to the indexed position for motion prediction of the block. In the case of the AMVP mode, in addition to the reference or index, the difference or residual value may also be signaled as an increment. For example, for the AMVP mode, the decoding device may construct one or more MV candidate lists and apply the increment value to one or more motion information candidates obtained using the signaled index value when performing motion compensation prediction of the block.

[0096] In some examples, the MV candidate list contains up to five candidates for merge mode and two candidates for AMVP mode. In other examples, the MV candidate list for merge mode and / or AMVP mode may include different numbers of candidates. The merge candidate may include a motion information set. For example, the motion information set may include motion vectors corresponding to both reference picture lists (list 0 and list 1) and reference indices. If the merge candidate is identified by a merge index, the reference picture is used for prediction of the current block, and the associated motion vector is determined. However, in AMVP mode, for each potential prediction direction from list 0 or list 1, because the AMVP candidate only contains motion vectors, the reference index needs to be explicitly signaled to the MV candidate list together with the MVP index. In AMVP mode, the predicted motion vector can be further refined.

[0097] As can be seen above, the merge candidate corresponds to a complete motion information set, while the AMVP candidate contains only one motion vector for a specific prediction direction and reference index. Candidates for both modes are similarly derived from the same spatial and temporal neighboring blocks.

[0098] In some examples, merge mode allows an inter-predicted PU to inherit one or more identical motion vectors, prediction directions, and one or more reference picture indexes from an inter-predicted PU including a motion data location selected from a set of spatially adjacent motion data locations and one of two motion data locations at the same location in time. For AMVP mode, one or more motion vectors of a PU may be predictively encoded and decoded relative to one or more motion vector predictors (MVPs) in an AMVP candidate list constructed by an encoder and / or decoder. In some cases, for unidirectional inter prediction of a PU, an encoder and / or decoder may generate a single AMVP candidate list. In some cases, for bidirectional prediction of a PU, an encoder and / or decoder may generate two AMVP candidate lists, one of which uses motion data from spatially and temporally adjacent PUs in a forward prediction direction and the other uses motion data from spatially and temporally adjacent PUs in a backward prediction direction.

[0099] Candidates for both modes can be derived based on spatial and / or temporal neighboring blocks. Figure 2A and Figure 2B Included is a conceptual diagram showing spatial neighbor candidates in HEVC.

[0100] Figure 2A Spatially neighboring motion vector (MV) candidates for merge mode are shown. Figure 2B 1 and 2. Spatial neighboring motion vector (MV) candidates for AMVP mode are shown. Spatial MV candidates are derived from neighboring blocks of a specific PU (PU0), although the method of generating candidates from blocks is different for merge and AMVP modes.

[0101] In merge mode, the encoder and / or decoder may form a merge candidate list by considering merging candidates of multiple motion data positions. Figure 2A As shown, reference Figure 2A Up to four spatial MV candidates can be derived from the spatially adjacent motion data positions indicated by numbers 0-4 in the merge candidate list. The MV candidates can be sorted in the order indicated by numbers 0-4 in the merge candidate list. For example, the positions and order may include: left position (0), top position (1), top right position (2), bottom left position (3), and top left position (4).

[0102] exist Figure 2BIn the AVMP mode shown in , the neighboring blocks are divided into two groups: the left group including blocks 0 and 1, and the upper group including blocks 2, 3 and 4. For each group, potential candidates in the neighboring blocks that refer to the same reference picture as the reference picture indicated by the signaled reference index have the highest priority to be selected to form the final candidate of the group. It may not be possible that all neighboring blocks contain motion vectors pointing to the same reference picture. Therefore, if no such candidate is found, the first available candidate will be scaled to form the final candidate so that the temporal distance difference can be compensated.

[0103] Some examples described herein may use an adaptive motion vector resolution (AMVR) mode, as described above. AMVR refers to adaptively adjusting the range or resolution of motion vectors during image processing. The use of AMVR enables different levels of accuracy when encoding and decoding motion vector differences, which improves encoding and decoding efficiency and may involve the use of fractional sample units (e.g., half-pixel or other such fractional samples at any resolution level). In some implementations, uniform fractional sample units, integer sample units, etc. may be used for the entire range, or the range of sample units may be adaptively set based on the region. The former may be referred to as linear AMVR, while the latter may be referred to as non-linear AMVR. The use of the AMVR mode may further involve explicit signaling of filter index values ​​(e.g., using syntax elements in the video bitstream) for motion compensation processing of the current image.

[0104] In the above case, reference picture resampling (RPR) is a resampling process that can be used to allow reference pictures with picture sizes that are different from the picture size of the current picture being encoded or decoded. When the reference picture has a different size than the current picture, the picture resampling process can be called to provide an upsampled or downsampled version of one of the pictures to match the size of the other picture (for example, upsampling or downsampling the reference picture to match the current picture size). In some examples, such resampling can be similar to the operations used for spatial scalability used in the scalable extension (SHVC) of the H.265 / HEVC standard. The RPR process can be applied to pictures of a single layer with different sizes and / or can be applied to pictures of different layers with different sizes. Refer to the following Figure 3 An illustrative example of applying RPR to pictures of different sizes from different layers is described.

[0105] Figure 3 1 is a diagram showing various aspects of picture data in a system at different scaling values ​​(e.g., different sizes). A picture or picture data (e.g., access units) may be assigned a layer structure. Each layer may be used for picture frames having different sizes (e.g., resolutions). Figure 3One such example shown in includes two layers, while various other examples may use other numbers of layers (e.g., three layers, four layers, etc.). As described above, the pictures within a single layer may have different sizes, in which case an RPR process may be performed. Figure 3 In the example of , data for three pictures is displayed, where each picture is associated with two different picture sizes. The first picture is associated with a layer 1 unit 314 of a first picture size and a layer 0 unit 312 of a second picture size. Similarly, the second picture is associated with a layer 1 unit 324 of a first picture size and a layer 0 unit 322 of a second picture size, and the third picture is associated with a layer 1 unit 334 of a first picture size and a layer 0 unit 332 of a second picture size. During signaling, a reference picture list may be constructed to identify reference pictures for use in processing a current picture. In some examples, a picture from one layer (e.g., at a first resolution) may be used as a reference picture for a current picture of a second layer (e.g., at a second resolution). In another example, a single layer may have pictures of different sizes, as described above.

[0106] With RPR, pictures (or blocks of pictures) of different sizes (e.g., pictures from a single layer or pictures from different layers) can be used as reference pictures for blocks of the current picture. Such reference pictures can be stored in a decoded picture buffer (DPB) (e.g., using a decoder for a filter unit (e.g., Figure 8 The filter unit 91) has a memory (e.g., Figure 8 The reference picture sampling tool may be used to generate necessary reference data from the reference picture for processing the current picture. In some examples, the reference picture sampling tool may be a filter unit (e.g., Figure 8 , or in other examples may be part of any aspect of an apparatus for encoding or decoding as described herein.

[0107] Figure 4 is a diagram illustrating further aspects of reference picture resampling according to examples described herein. Figure 4A reference picture 410 and a current picture 420 are included. The reference picture 410 has a first horizontal grid spacing 414 and a first vertical grid spacing 412. The current picture 420 has a second horizontal grid spacing 424 different from the first horizontal grid spacing 414. The current picture also has a second vertical grid spacing 422 different from the first vertical grid spacing 412. The first grid spacings 412 and 414 may be associated with a first layer, for example, while the second grid spacings 422 and 424 may be associated with a second layer. The reference picture 410 may be a first layer picture in a video stream that is continuous with the current picture 420 of the second layer, wherein the reference picture 410 is selected as a reference picture for the current picture 420 as part of a prediction operation.

[0108] The reference picture 410 and the current picture 420 each include a pixel grid, including pixels 418 and 419 of the reference picture 410 and pixel 428 of the current picture 420. Each pixel may have associated data, such as color values ​​of the red, green, and blue parts of the pixel. Each color has a data value at each pixel. Each pixel of each picture may be associated with an integer position (e.g., a size or scaling value) in the grid of the picture. A standard motion vector 430 pointing directly from one pixel to another may use the integer position of the pixel. However, as described above, fractional precision may allow the motion vector 430 to point between pixels. Half-pixel precision may, for example, allow the motion vector 430 to point to the middle between pixels 419 and 418. Quarter-pixel precision allows the motion vector 430 to point to three different positions (e.g., one-quarter, one-half, and three-quarters) between pixels and directly to the integer position of the pixel.

[0109] As described above, AMVR allows motion vector precession to be adaptively set at different fractional precision levels. When image processing using the AMVR mode is implemented, MV precision can be adaptively selected with integer pixel precision (e.g., where the MV is located at grid points only for grid spacings 412, 414, 422, and 424, without fractional precision) or with any fractional precision level supported by the implemented AMVR mode (e.g., allowing the MV to point to points between grid points with fractional precision levels for grid spacings 412, 414, 422, and 424). Allowing the system to adaptively determine the resolution level (e.g., with integer resolution at the grid spacing level or fractional resolution for precision between grid spacings) improves codec efficiency. This improved codec efficiency can be signaled using additional syntax elements, which can also explicitly indicate the filter index of the current picture, as described herein for various examples.

[0110] Interpolation is used to select data values ​​(e.g., red, green, and blue data values) at pixel locations given expected motion or other changes, including intra-pixel data (e.g., intra-pixel data based on motion vectors using fractional precision). During processing (e.g., decoding or encoding), processing is performed on each pixel or on each sample of each pixel (e.g., luminance sample and / or chrominance sample). Filtering is performed on each pixel or on each sample of each pixel, including difference and any resampling compensation due to RPR, to generate pixel or sample values ​​of the current picture 420 during video data processing. In another example, interpolation and resampling filtering can be combined into one filtering process. During processing, each pixel or sample will have an attached scaling value and motion vector. As described above, different filters can be used in different situations (e.g., different values ​​of motion vector 430 or different scaling values ​​between reference picture 410 and current picture 420).

[0111] Table 1 below provides an example table of filter coefficients with filter indices for interpolation filter switching between different half-pixel interpolation filters:

[0112]

[0113] Table 1

[0114] Table 1 shows a table of interpolation filter coefficients for filtering at one-sixteenth precision positions. Depending on the fractional precision position (e.g., fractional sample positions 1-15 in the first column, where the 0 position is a non-fractional or integer position), different filter coefficients (e.g., coefficients from the row corresponding to the fractional position) can be used. In addition, at half-pixel positions (e.g., fractional sample position 8 in one-sixteenth fractional precision of Table 1), two interpolation filters are available, and signaling can be used to switch between filters at this level. Although Table 1 describes two filters for interpolation filter switching at the half-pixel level (e.g., selecting between filter index values ​​1 and 2), other examples may use more than two filters for switching, or may include additional filters for switching at any fractional position. Adding additional filters for interpolation filter switching can improve fractional precision during image processing at the expense of additional signaling and processing resources.

[0115] In JVET-N0279 and JVET-O0242, RPR is proposed, and the entire contents of JVET-N0279 and JVET-O0242 are incorporated herein by reference and used for all purposes. In this proposal, the current picture (e.g., the picture currently being encoded or decoded) and the reference picture can have different resolutions (e.g., size, grid spacing, scaling value, etc.) as described above. In this case, during motion compensation, the difference in resolution is taken into account when interpolation filtering is performed by adjusting the interpolation filter phase. For example, in real-time video communication, allowing the resolution to change within the coded and decoded video sequence without inserting an intra-frame prediction picture (I-picture) can not only make the video data seamlessly adapt to dynamic channel conditions or user preferences, but also eliminate the impact caused by the I-picture. When the adaptive resolution changes, the current picture can be predicted based on reference pictures of different sizes. An illustrative example of a reference picture resampling (or scaling) process is described below.

[0116] In JVET-O0057, the concept of switchable interpolation filters is proposed to VVC, and the entire contents of JVET-O0057 are incorporated herein by reference and for all purposes. In this document, one or more interpolation filters can be switched. For example, the filter index can be signaled or inferred (e.g., by a decoder) to indicate that another interpolation filter is used for a block. The filter index can be signaled on a block basis (e.g., explicitly or by half-pixel AMVR mode signaling). For example, in a system with half-pixel AMVR, a syntax element signaled in the bitstream (e.g., a syntax element indicating a specific filter index value of Table 1) can explicitly indicate the filter index to be used when processing the current block associated with the syntax element. In an example where AMVR uses fractional precision other than half-pixel, similar signaling can use syntax elements configured as specific resolution options for AMVR implementations.

[0117] In other implementations, the filter index does not need to be explicitly signaled via a syntax element in the bitstream that specifically indicates the filter index to be used. In such other examples without explicit signaling, the filter index may be signaled via a merge-like mode (e.g., as described above with reference to Figure 2A and 2B The filter index is inferred (e.g., by a decoder) from motion vector candidates in the merge mode described above.

[0118] In one illustrative example, a total of N filters with different characteristics (e.g., such as filter length, cutoff frequency, transition band, or amount of ripple) may be defined. For each block, a subset of the N filters may be defined, and the index of the filter selected in the subset may be explicitly or implicitly signaled to the decoder (e.g., using a flag or filter index value as shown in Table 1). The subset may vary from one block to another based on certain criteria (e.g., hit rate of filters in previously coded blocks, motion vector values, scaling values, etc.).

[0119] When interpolation filter switching occurs, the filter indicated by the filter index may be suitable for interpolation purposes, but may not be used for picture resampling purposes (and other sampling rate conversion processes in some cases). For example, a smoothing filter may not be a good resampling filter. Switching between different filters (e.g., interpolation filter switching) allows image processing operations to create stability between images depending on the characteristics of the pixels being filtered (e.g., the associated motion vectors and scaling values ​​of the pixels in the current picture and the associated reference picture). For example, the filter index may have filter index values ​​for different filters. In some examples, the filter to be used is signaled (e.g., using a flag) to indicate which of the filters in the index is to be used (e.g., switching between interpolation filters). In another example, the filter to be used may be from a spatial candidate (e.g., a reference picture), where the filter index is stored together with the motion vector data and propagated from the reference picture to be used with the current picture.

[0120] The methods, apparatus, and computer-readable media described herein solve the above problems and other problems. In some examples, as described in more detail below, a conditional check can be performed to identify whether the current picture size and the reference picture size are the same size. The conditional check can be performed at different levels, as described in detail below.

[0121] Figure 5 is a flow chart illustrating a method 500 for processing video data. In some examples, the method 500 is performed by a device including a memory and one or more processors. In some examples, the method 500 may be performed by an encoding device such as device 104. In some examples, the method 500 may be performed by a decoding device such as device 112. In some examples, the method 500 is embodied as instructions stored in a computer-readable memory that, when executed by one or more processors, cause the one or more processors to perform the method 500.

[0122] At operation 505, method 500 includes obtaining a current picture and a reference picture. As described above, in various examples, the current picture and the reference picture may be divided and constructed in various ways (eg, various units, slices, etc.).

[0123] Operation 510 of method 500 includes identifying filter index information for a current block of a current picture. In different examples, the filter index information can be inferred from neighboring blocks, or can be signaled as part of AMVR. In some examples, the filter index information includes AMVR mode signaling data, wherein the current filter index is signaled via AMVR mode signaling. In some examples, the filter index information includes neighboring block information, and the current filter index is derived from the neighboring block information during the derivation of the merge mode motion vector candidate list. The filter index information includes motion information such as a filter index or a filter index value. In merge mode, the filter index information can be propagated or implied from neighboring blocks. In AMVR mode, the filter index information can be explicitly signaled using AMVR signaling.

[0124] Operation 515 of method 500 includes determining that a first picture size value of the current picture is different from a second picture size value of the reference picture. In some examples, the determination may involve a width and height comparison. In other examples, any other such comparison may be used. In one example, determining that the first picture size value of the current picture is different from the second picture size value of the reference picture includes determining at least one of: the current picture width is different from the reference picture width; and the current picture height is different from the reference picture height. The picture size check may be used as a condition for changing the filter index. Depending on the picture size check and the associated change in the filter index, the propagation of the original index value (e.g., the current index value before being replaced in the case indicated by the picture size check) may be configured (e.g., by storing the original index value and replacing it for the propagation step, or retaining the original index value if the default value does not replace the original index value).

[0125] At operation 520, method 500 includes performing a sampling rate conversion process by replacing the current filter index identified by the filter index information with a default filter index. In some examples, the sampling rate conversion process includes a resampling process. For example, operation 520 may include performing a resampling process by replacing the current filter index identified by the filter index information with a default filter index. In some examples, the sampling rate conversion process includes an interpolation process. The execution may be based on determining that the first picture size value of the current picture is different from the second picture size value of the reference picture. In various examples, when the current picture and the reference picture have different size values, the sampling rate conversion process (e.g., a resampling process) is performed without using the current filter index. In some such examples, the default value of the current filter index is parsed by the device or includes an indication that an alternative switchable filtering is not applied as part of the sampling rate conversion process (e.g., a resampling process).

[0126] In various examples, when the filter index is derived (e.g., the current index indicated by signaling or derivation), the value is with the block and can be used (e.g., propagated) for subsequent blocks processed by the system. In such examples, if a default filter index value is used for the current picture (e.g., the alternative switchable filter is not used), the default filter index value is not propagated from the current picture, and the original value can be stored or the default value can be replaced with the original value after being used for propagation to subsequent pictures.

[0127] In some examples, the method may also involve operations for storing a current filter index identified by the filter index information with the current block, and using the current filter index in motion vector candidate derivation for a subsequent block based on the current filter index being identified by the filter index information. In some examples, the method may also involve operations for assigning a default value to the current filter index to temporarily replace the original value to create a default filter index when a sampling rate conversion process (e.g., a resampling process) is performed without using the current filter index, and for assigning the original value to the current filter index after the sampling rate conversion process (e.g., the resampling process). This reassignment of the original allows the current filter index value (e.g., the original value) to be propagated to the subsequent picture even if the current filter index value is not used for the current picture.

[0128] Some such examples may be run by determining that the current filter index is not used for a sample rate conversion process (e.g., a resampling process), and replacing the value of the current filter index with a default value in response to determining that the current filter index is not used for a sample rate conversion process (e.g., a resampling process). Similarly, some such examples may be run by processing the default value to determine that an alternative switchable filter is not applied.

[0129] Some examples may operate in the following cases: in response to determining that the current filter index is not used in a sample rate conversion process (e.g., a resampling process), the current filter index value is not signaled. Some examples may operate in the case where the current filter index is selected from a filter set, in which case a subset of the filter set is used to perform the sample rate conversion process (e.g., a resampling process) and the remainder of the filter set is not available for sample rate conversion (e.g., not available for resampling). Some such examples operate in the following cases: the current filter index is selected using a syntax element that indicates that the current filter index is prohibited from being selected from the remainder of the filter set.

[0130] Then, as needed, method 500 can be repeated as part of a decoding or encoding operation. In some operations, method 500 will not be used for some iterations (e.g., interpolation without resampling or involving other sampling rate conversion processes, such as when the reference picture is the same size as the current picture). As described above, in some examples, the sampling rate conversion process includes a resampling process. The operation of method 500 may also include identifying a first filter group for interpolation and a second filter group for resampling, wherein the current filter index is from the second filter group. The operation of method 500 may also include obtaining a second current picture and a second reference picture, determining that the third picture size value of the second current picture is the same size value as the fourth picture size value of the second reference picture. Based on determining that the third picture size value of the second current picture and the fourth picture size value of the second reference picture are the same size values, the operation of method 500 may include performing a second resampling process on the second current block using a second selected filter, wherein the second selected filter is from the first filter group. In some such examples, each filter in the first filter group and the second filter group is identified by an associated filter index, and the associated filter index indicates an associated index value from a filter set for interpolation or a filter set for resampling. Some such examples also include obtaining a current filter index based on the filter index via a lookup operation for a selected filter index value, wherein the selected filter index value is associated with a second filter group, and wherein the association between the selected filter index value and the second filter group is based on determining that the first picture size value is different from the second picture size value.

[0131] In some examples, when the current picture width is different from the reference picture width or the current picture height is different from the reference picture height, the current filter index is not used (e.g., ignored or the resampling process or other sampling rate conversion process does not depend on the value of the current filter index) in the resampling process. The derived or signaled filter index can be stored with the current block and used in the motion vector candidate derivation process in subsequent blocks. When it is determined that the derived or signaled filter index is not used, a default value for the filter index can be temporarily assigned during the resampling process, while the derived or signaled filter index value remains unchanged after the resampling process. In this case, the default value can indicate that the alternative switchable filter is not applied.

[0132] In some such examples, the filter index is derived from neighboring blocks during a motion vector candidate list derivation process for merge mode, as described above. In other examples, the filter index is signaled via an AMVR mode. In some examples, the sampling rate conversion process is a motion compensation process. In some such examples, the reference picture is a reference picture for inter-frame prediction of a current block of a current picture.

[0133] In some examples, when it is determined that the derived or signaled filter index is not used, the filter index value can be replaced with a default value. In some such examples, the default value indicates that an alternative switchable filter is not applied, where in one example the alternative filter can be a smoothing filter. In some other examples, when it is determined that the filter index is not used in a resampling process (or other sampling rate conversion process), the filter index value is not signaled.

[0134] In some implementations, the processes (or methods) described herein may be performed by a computing device or by an apparatus, such as Figure 1 For example, these processes may be performed by Figure 1 and Figure 7 Executed by the encoding device 104 shown in FIG. 1 , executed by another video source side device or a video transmission device, or executed by Figure 1 and Figure 8, and / or by another client device, which is, for example, a player device, a display, or any other client device. In some cases, a computing device or apparatus may include a processor, a microprocessor, a microcomputer, or other components of an apparatus configured to perform the steps of the process described herein. In some examples, a computing device or apparatus may include a camera configured to capture video data (e.g., a video sequence) including video frames. In some examples, a camera or other capture device that captures video data is separated from a computing device, in which case the computing device receives or obtains the captured video data. The computing device may also include a network interface configured to transmit video data. The network interface may be configured to transmit data based on an Internet Protocol (IP) or other types of data. In some examples, a computing device or apparatus may include a display for displaying output video content, such as a sample of a picture of a video bitstream.

[0135] Figure 6 is a flowchart illustrating aspects of reference picture resampling according to some examples, Figure 6 6 is shown as method 600. Method 600 may be performed as part of an image or video processing procedure, such as a processing procedure according to VVC, JEM or other video coding standard procedures as described above.

[0136] During video processing, as part of an implementation of method 600, operation 602 includes identifying a current picture. The current picture may be identified using the POC as described above or using any other operation for identifying data associated with an image in a set of images (eg, a video).

[0137] To process the current picture identified in operation 602, reference pictures are identified in operation 604. As part of this identification, one or more reference pictures or reference picture candidates (e.g., from the DPB) may be selected for a reference picture list (e.g., using the POC) as described in detail above and below. Depending on the implementation, a single reference picture may be selected from the reference picture list for processing various aspects of the current picture. In some implementations, different reference pictures may be used to process different portions of the current picture, but during processing, data from a single reference picture (e.g., image data used to create motion vectors (such as motion vector 430 using reference picture 410)) is used.

[0138] Additionally, various different modes may be used as part of the video processing during video processing according to method 600. These may include different modes for motion prediction as described above, such as AMVR and merge mode. In different implementations, these modes may be changed during processing of a single image, portion of an image, or group of images.

[0139] Different processing modes can signal information about the filter to be used in processing the current image in different ways. For example, as described herein, the AMVR can explicitly signal a syntax element indicating a filter index value that identifies information related to the filter to be used in processing the current image. In contrast, by deriving filter index information based on other pictures (e.g., adjacent pictures, reference pictures, etc.), the merge mode can operate without explicit filter index signaling. In operation 608, an appropriate operation for identifying filter index information is determined, and the processing operation continues with the appropriate operation to collect filter index information. If explicit filter index signaling is used, then in operation 610, the syntax element of the explicit signaling is parsed to collect filter index information (e.g., the value of the filter index). If explicit filter index signaling is not used, then in operation 612, the filter index information is derived (e.g., propagated from adjacent images or image portions).

[0140] Once the filter index information is collected in operation 610 or in operation 612, the filter index information is used to set the current filter index value in operation 614. For example, in an implementation using Table 1, based on the filter index information, the filter index value may be set to one of two filter index values ​​associated with the half-pixel fractional index position.

[0141] Operation 616 includes a conditional check, which is used to select an appropriate filter based on whether resampling is used. The conditional check involves comparing the size of the current picture and the reference picture identified in operation 604. As described herein, the conditional check can involve comparing horizontal dimensions, vertical dimensions, resolution values, or any other such information associated with the size of the current picture and the reference picture. In some examples, the conditional check can be performed on a block basis (e.g., comparing reference and current picture sizes or scaling). For example, a conditional check can be performed to check whether the current picture containing the block and the reference picture indicated by the reference index in the block have the same picture size or whether they do not have the same size. In some examples, a conditional check can be performed on a slice, tile, sub-picture, or other block group level. For example, a conditional check can check whether the current picture and the reference picture that can be used for inter-frame prediction have the same size.

[0142] In some examples, the condition may be checked at the sequence level, such as checking whether the possibility of having different picture sizes is possible. In one example, it may be indicated in a parameter set (e.g., by a flag in a PPS, SPS, and / or VPS) whether picture resampling may be applied.

[0143] In some cases, when the picture size is derived (e.g., for both the current picture and the reference picture), the size can be the decoded picture size or the cropped picture size. For example, a conformance window can be signaled in the bitstream and can be used by a decoder or a media player to crop the decoded picture to the cropped picture size. In some cases, the picture size for the conditional check can be derived based on the decoded picture size and various offsets (e.g., left offset, right offset, top offset, and / or bottom offset). The decoded picture size and / or offset can be signaled in a parameter set (e.g., PPS, SPS, and / or VPS) or elsewhere (e.g., in an SEI message, a slice header, or other signaling mechanism). In this case, the scaling ratio for resampling can be derived by considering the picture size of the current picture and the size of the reference picture with the applied offset. In some implementations, the offset can be similar to the scaling offset used in the HEVC standard (e.g., in a scalable extension). This scaling ratio and / or size value for the current picture and the reference picture may then be used in determining which filters are applied to process the current picture.The techniques described herein may use any method of checking whether the current picture size and the reference picture size are the same size.

[0144] The current image may then be processed in a different manner (e.g., using a different filter) depending on the result of operation 616. If the current picture and the reference picture are identified as being the same size, then in operation 618, the current index value (e.g., as signaled or derived above) is used in processing the current picture.

[0145] In the event that the current picture and the reference picture sizes are not the same, then, in operation 620, the current filter index value set in operation 614 is ignored. This may involve the use of a default filter value set in the system for filtering that involves resampling of the reference picture or a default filter loop for the operation for which the check condition is true. In some examples, if the check condition is identified as true (e.g., the picture sizes of the current picture and the reference, or one or both reference pictures in bidirectional prediction are different), the use of one or more switchable interpolation filters is disabled at the sequence level in the SPS, at the picture level signaled by the PPS, at the slice and / or tile level, and / or at the block level. This disabling of switchable interpolation filtering (e.g., using the set current filter index value when switchable interpolation filtering is used in the system) can be considered as a selection of a default filter or any alternative filter for the switchable interpolation filter set in operation 614.

[0146] Regardless of which filter is selected, the current picture is processed using the selected filter in operation 622. This processing can be part of a codec loop that filters different portions of the current picture, so depending on the implementation, the operations of method 600 can be repeated and looped in different ways to process the current picture or portions of the current picture using different filters. In the example of method 600, operation 624 includes propagating the current filter index value so that this information is available to subsequent blocks if needed. This can involve generating explicit signaling, or can involve storing the current filter index value so that it can be accessed during subsequent iterations of operation 612 to derive the filter index value (e.g., when the current picture or a portion of the current picture becomes an adjacent or reference picture to the subsequent current picture being processed). Method 600 can then end and can be looped to process the video stream. It will be apparent that method 600 may be implemented with additional repetitions or intervening operations, depending on how the use of switchable interpolation filtering is used (e.g., where the structure for switchable interpolation filtering is to be disabled at the sequence level in the SPS, at the picture level via PPS signaling, at the slice and / or tile level, and / or at the block level). In addition, during certain modes or processing operations, switchable interpolation filtering may not be used (e.g., in the case of switchable interpolation filtering, selection of a single quarter-pel or other value of Table 1 other than the half-pel index value), and method 600 may not be used to process a portion of a video when it is used for other portions of the same video.

[0147] In some cases, when use of one or more switchable interpolation filters is disabled at the block level (e.g., when a check condition is identified as true), a syntax element indicating a filter index is not signaled for the block (e.g., half-pixel AMVR mode signaling, where different half-pixel interpolation is implicitly implied). In this case, the syntax element associated with the switchable filtering (e.g., filter index) for half-pixel AMVR for indicating different interpolation filters depends on the picture size check (e.g., whether the current picture size and the reference picture size are the same size).

[0148] In some cases, when the use of one or more switchable interpolation filters is disabled at the block level (e.g., when a check condition is identified as true), if the filter index or the use of a different filter (e.g., by, for example, half-pixel AMVR signaling or in a merge mode or other merge-like mode) is signaled or inferred from a neighboring block, the filter index is modified and set to a default filter index value for the derived motion vector candidate from the neighboring block. In one example, the default filter may indicate that the switchable filter is not applied. According to various examples, the selected filter may be indicated explicitly or implicitly. In some examples, the explicit syntax element is signaled via AMVR mode signaling. In other examples, the filter is implied, derived, or propagated from a neighboring block or a reference block. When the filter index that is signaled (e.g., explicitly or implicitly) is not used because the switchable interpolation filtering is disabled, the default filter index value may be used. The default filter may indicate that the switchable filtering is not used (e.g., as part of a process for disabling the switchable interpolation filtering when it is not allowed).

[0149] In some cases, when the use of switchable interpolation filters is disabled at the block level (when the check condition is identified as true), if the filter index (or use of a different filter by, for example, half-pixel AMVR signaling) is inferred from neighboring blocks, for example in merge mode or other merge-like modes, then the filter index (or implicit signaling by, for example, half-pixel AMVR mode) is derived from neighboring blocks, and during the resampling process, the derived filter index is ignored and the resampling filter is applied even though it may be indicated that switchable filtering should be used. The derived filter index for the current block can be stored with the block and can be used for subsequent blocks (after the current block), in which case the filter index is propagated.

[0150] In some examples, the above idea is extended in the following cases: filters can be selected from a filter set (e.g., the two half-pixel filters of Table 1), only a subset of the filters can be used for resampling, and other filters that are not available for resampling are also restricted from selection (in one example, a syntax element indicating that such selection can be prohibited based on different conditions of picture size) or such selection is ignored when performing the resampling process.

[0151] In some examples, two filter banks or filter sets may be provided, one of which is used for interpolation (e.g., for processing operation 622 using the filter selected in operation 618) and the other filter bank is used for resampling (e.g., for processing operation 622 using the filter selected in operation 620). In such an example, if switching of filters is indicated, a filter is selected from the set based on the condition that the picture size is different, i.e., resampling is performed. For example, the filter index may indicate the use of a filter with a filter index from a filter set (or group), where the set (or group) is identified based on whether interpolation or resampling is performed only on the block.

[0152] In a more detailed example, it is assumed that there is a filter set for interpolation (denoted as interpolationSet) and another filter set for resampling (denoted as resamplingSet). The filter set can have one filter or multiple filters (the size of the filter set can be 1 or greater than 1). A filter index with a value filterIndex can be selected for use in a block. The filter used in the block can be obtained from the filter set by looking up the filter with index filterIndex. The filter can be from the interpolationSet or resamplingSet set, and the filter set is identified based on the condition of whether the current picture and the reference picture size are the same. If the current picture and the reference picture size are the same (in this case no resampling is required), the filter set is identified as interpolationSet. If the current picture and the reference picture size are different (in this case resampling is performed), the filter set is identified as resamplingSet. The advantage of this technique is that there is no need to change the filter index signaling and filter index inference (for example, the filter index value can be propagated between adjacent blocks that are interpolated or resampled), and only the final filter is selected from the set based on the picture size condition.

[0153] An illustrative example of a reference picture resampling (or scaling) RPR process will now be described. As described above, when there is a resolution change (e.g., within a coded video sequence (CVS)), a picture may have a size different from one or more of its reference pictures. The RPR process may be performed to normalize all motion vectors to the current picture grid, rather than their corresponding reference picture grids. Such a process may be beneficial in maintaining design consistency and making resolution changes transparent to the motion vector prediction process. Otherwise, adjacent motion vectors pointing to reference pictures of different sizes may not be used directly for spatial motion vector prediction due to different scales.

[0154] When a resolution change occurs, both the motion vector and the reference block can be scaled when performing motion compensated prediction. In some cases, the scaling range is limited to [1 / 8, 2] (e.g., enlargement is limited to 1:8, and reduction is limited to 2:1). Here, enlargement (or upsampling) refers to the case where the reference picture is smaller than the current picture, while reduction (or downsampling) refers to the case where the reference picture is larger than the current picture. The scaling process is described in more detail below.

[0155] For the luma block, the scaling factor and its fixed-point representation can be defined as:

[0156]

[0157]

[0158] The RPR process may include two parts, including: (1) mapping the upper left pixel of the current block to the reference picture; and (2) using horizontal and vertical steps to process the reference positions of other pixels of the current block. In some cases, if the coordinates of the upper left pixel of the current block are (x, y), the sub-pixel position (x', y') in the reference picture pointed to by the motion vector (mvX, mvY) in units of 1 / 16 pixels is specified, where the horizontal position in the reference picture is:

[0159] x′=((x<<4)+mvX)·hori_scale_fp, equation (3)

[0160] And, x′ is further reduced to keep only 10 fractional digits:

[0161] x′=Sign(x′)·((Abs(x′)+(1<<7))>>8), equation (4)

[0162] Similarly, the vertical position in the reference picture is:

[0163] y′=((y<<4)+mvY)·vert_scale_fp, equation (5)

[0164] And, y′ is further reduced to:

[0165] y′=Sign(y′)·((Abs(y′)+(1<<7))>>8), equation (6)

[0166] At this point, the reference position of the top left pixel of the current block is at (x', y'). Other reference sub-pixel / pixel positions can be calculated with horizontal and vertical steps relative to (x', y'). Those steps can be derived with 1 / 1024 pixel precision (or other pixel precision) based on the above horizontal and vertical scaling factors, as follows:

[0167] x_step=(hori_scale_fp+8)>>4, equation (7)

[0168] y_step=(vert_scale_fp+8)>>4, equation (8)

[0169] In one example, if a pixel in the current block is i columns and j rows away from the upper left corner pixel (or sample), the horizontal and vertical coordinates of its corresponding reference pixel are derived as follows:

[0170] x′ i =x′+i*x_step, equation (9)

[0171] y′ j =y′+j*y_step, equation (10)

[0172] In sub-pixel interpolation, x′ i and y′ j It may need to be decomposed into an integer pixel part and a fractional pixel part as follows, where the integer pixel part used to address the reference block is equal to:

[0173] (x′ i +32)>>10, equation (11)

[0174] (y′ j +32)>>10, equation (12)

[0175] The fractional pixel portion used to select the interpolation filter is equal to:

[0176] Δx=((x′ i +32)>>6)&15, equation (13)

[0177] Δy=((y′ j +32)>>6)&15Equation (14)

[0178] Once the integer pixel and fractional pixel positions within the reference picture are determined, the existing motion compensation interpolator can be used without any further changes. The integer pixel positions will be used to obtain the reference patch from the reference picture, while the fractional pixel positions will be used to select the appropriate interpolation filter.

[0179] An example of the RPR process for a chroma block is now described. For example, when the chroma format is 4:2:0, the accuracy of the chroma motion vector is 1 / 32 pixel. In this example, the scaling process of the chroma motion vector and the chroma reference block is similar to that of the luminance block, except that adjustments related to the chroma format are introduced.

[0180] When the upper left pixel coordinates of the current chroma block are (xc, yc), the initial horizontal and vertical positions in the reference chroma picture are:

[0181] x c ′=((x c <<5)+mvX)·hori_scale_fp, equation (15)

[0182] y c ′=((y c <<5)+mvY)·vert_scale_fp, equation (16)

[0183] Where mvX and mvY are the original luma motion vectors, but should now be detected with 1 / 32 pixel accuracy. c ′ and y c ' is further reduced to maintain 1 / 1024 pixel accuracy as follows:

[0184] x c ′=Sign(x c ′)·((Abs(x c ')+(1<<8))>>9), equation (17)

[0185] y c ′=Sign(y c ′)·((Abs(y c ')+(1<<8))>>9), equation (18)

[0186] The above right shift adds one extra bit compared to the associated brightness equation.

[0187] The step size used can be the same as that used for luma. For a chroma pixel at (i, j) relative to the top left pixel, the horizontal and vertical coordinates of its reference pixel are derived as follows:

[0188]

[0189]

[0190] In sub-pixel interpolation, and is also decomposed into an integer-pixel part and a fractional-pixel part, and the integer-pixel part used to address the reference block is equal to:

[0191]

[0192]

[0193] The fractional pixel portion used to select the interpolation filter is equal to:

[0194]

[0195]

[0196] The processes may be described with reference to a logic flow diagram, the operations of which represent a sequence of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the enumerated operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the processes.

[0197] In addition, these processes can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as codes (e.g., executable instructions, one or more computer programs, or one or more applications) that are executed together on one or more processors. These processes can also be implemented by hardware, or a combination of the above. As described above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0198] The coding and decoding techniques discussed herein can be implemented in an example video coding and decoding system (e.g., system 100). In some examples, the system includes a source device that provides encoded video data to be decoded by a target device later. Specifically, the source device provides video data to the target device via a computer-readable medium. The source device and the target device can include any of a variety of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, so-called "smart" tablets, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, the source device and the target device can be equipped for wireless communication.

[0199] The target device can receive the encoded video data to be decoded via a computer-readable medium. The computer-readable medium may include any type of medium or device capable of moving the encoded video data from the source device to the target device. In one example, the computer-readable medium may include a communication medium so that the source device directly transmits the encoded video data to the target device in real time. The encoded video data may be modulated according to a communication standard such as a wireless communication protocol and transmitted to the target device. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form a part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other device that may be useful for facilitating communication from the source device to the target device.

[0200] In some examples, the encoded data can be output from the output interface to the storage device. Similarly, the encoded data can be accessed from the storage device through the input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a DVD, a CD-ROM, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device may correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device. The target device can access the stored video data in the storage device via streaming or downloading. The file server can be any type of server capable of storing encoded video data and transmitting the encoded video data to the target device. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The target device can access the encoded video data through any standard data connection, including an Internet connection. The target device may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a DSL, a cable modem, etc.) suitable for accessing the encoded video data stored on the file server, or a combination of both. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0201] In one example, the source device includes a video source, a video encoder and an output interface. The target device may include an input interface, a video decoder and a display device. The video encoder of the source device may be configured to apply the technology disclosed herein. In other examples, the source device and the target device may include other components or arrangements. For example, the source device may receive video data from an external video source (e.g., an external camera). Similarly, the target device may be connected to an external display device, rather than including an integrated display device.

[0202] The above example system is only an example. The techniques for processing video data in parallel can be performed by any digital video encoding and / or decoding device. Although the techniques of the present disclosure are generally performed by video encoding devices, these techniques can also be performed by video encoders / decoders, which are generally referred to as "codecs". In addition, the techniques of the present disclosure can also be performed by a video preprocessor. The source device and the target device are only examples of codec devices in which the source device generates encoded video data for transmission to the target device. In some examples, the source device and the target device can operate in a substantially symmetrical manner so that each of the devices includes a video encoding and decoding component. Thus, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0203] The video source can include a video capture device, such as a video camera, a video archive comprising previously captured video, and / or a video feed interface for receiving video from a video content provider. As another alternative, the video source can generate computer graphics-based data as the source video, or a combination of live video, archived video and computer-generated video. In some cases, if the video source is a video camera, the source device and the target device can form a so-called camera phone or video phone. However, as mentioned above, the technology described in the present disclosure is generally applicable to video coding and decoding, and can be applied to wireless and / or wired applications. In each case, the captured, pre-captured or computer-generated video can be encoded by a video encoder. Then the encoded video information can be output to the computer-readable medium by the output interface.

[0204] As described above, computer-readable media may include transient media (e.g., wireless broadcast or wired network transmission) or storage media (i.e., non-temporary storage media) (e.g., hard disk, flash drive, compact disc, digital video disc, Blu-ray disc, or other computer-readable media). In some examples, a network server (not shown) may receive encoded video data from a source device and provide the encoded video data to a target device, for example, via a network transmission. Similarly, a computing device of a media production facility such as a disc stamping mechanism may receive encoded video data from a source device and produce a disc containing the encoded video data. Therefore, in various examples, a computer-readable medium may be understood to include one or more computer-readable media in various forms.

[0205] The input interface of the target device receives information from the computer-readable medium. The information of the computer-readable medium may include grammatical information defined by the video encoder and used by the video decoder. The information includes grammatical elements describing the characteristics and / or processing of blocks and other encoding and decoding units (e.g., group of pictures (GOP)). The display device displays the decoded video data to the user, and may include any of various display devices, including, for example, cathode ray tubes (CRTs), liquid crystal displays (LCDs), plasma displays, organic light emitting diode (OLED) displays, or another type of display device. Various embodiments of the present application have been described.

[0206] The details of the encoding device 104 and the decoding device 112 are described in Figure 7 and Figure 8 are shown separately in. Figure 7is a block diagram illustrating an example encoding device 104 that can implement one or more of the techniques described in the present disclosure. The encoding device 104 can, for example, generate the syntax structures described herein (e.g., syntax structures of VPS, SPS, PPS, or other syntax elements). The encoding device 104 can perform intra-frame prediction and inter-frame prediction coding and decoding on video blocks within a video slice. As previously described, intra-frame coding and decoding rely at least in part on spatial prediction to reduce or remove spatial redundancy within a given video frame or picture. Inter-frame coding and decoding rely at least in part on temporal prediction to reduce or remove temporal redundancy within adjacent or surrounding frames of a video sequence. Intra-frame mode (I mode) can refer to any of several spatial-based compression modes. Inter-frame mode, such as unidirectional prediction (P mode) or bidirectional prediction (B mode), can refer to any of several temporal-based compression modes.

[0207] The encoding device 104 includes a segmentation unit 35, a prediction processing unit 41, a filter unit 63, a picture memory 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 includes a motion estimation unit 42, a motion compensation unit 44, and an intra-frame prediction processing unit 46. For video block reconstruction, the encoding device 104 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62. The filter unit 63 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although the filter unit 63 is Figure 7 6 is shown as a loop filter, but in other configurations, filter unit 63 may be implemented as a post-loop filter. Post-processing device 57 may perform additional processing on the encoded video data generated by encoding device 104. Techniques of the present disclosure may be implemented by encoding device 104 in some examples. However, in other examples, one or more of the techniques of the present disclosure may be implemented by post-processing device 57.

[0208] like Figure 7As shown, the encoding device 104 receives video data, and the segmentation unit 35 segments the data into video blocks. Segmentation may also include segmentation into slices, slice segments, tiles, or other larger units, and video block segmentation according to, for example, a quadtree structure of LCUs and CUs. The encoding device 104 generally illustrates a component for encoding a video block within a video slice to be encoded. A slice may be divided into a plurality of video blocks (and may be divided into sets of video blocks referred to as tiles). The prediction processing unit 41 may select one of a plurality of possible codec modes for the current video block based on error results (e.g., codec rate and distortion, etc.), for example, one of a plurality of intra-frame prediction codec modes or one of a plurality of inter-frame prediction codec modes. The prediction processing unit 41 may provide the resulting intra-frame or inter-frame coded block to the adder 50 to generate residual block data, and to the adder 62 to reconstruct the coded block for use as a reference picture.

[0209] Intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction encoding and decoding of the current video block relative to one or more neighboring blocks within the same frame or slice as the current block to be encoded and decoded to provide spatial compression. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction encoding and decoding of the current video block relative to one or more prediction blocks in one or more reference pictures to provide temporal compression.

[0210] Motion estimation unit 42 may be configured to determine an inter-frame prediction mode for a video slice according to a predetermined pattern for a video sequence. The predetermined pattern may specify a video slice in a sequence as a P slice, a B slice, or a GPB slice. Motion estimation unit 42 and motion compensation unit 44 may be highly integrated but are shown separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates the motion of a video block. A motion vector may, for example, indicate the displacement of a prediction unit (PU) of a video block within a current video frame or picture relative to a prediction block within a reference picture.

[0211] A prediction block is a block that is found to closely match a PU of a video block to be coded in terms of pixel difference, which may be determined by the sum of absolute differences (SAD), the sum of squared differences (SSD), or other difference metrics. In some examples, encoding device 104 may calculate values ​​for sub-integer pixel positions of a reference picture stored in picture memory 64. For example, encoding device 104 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of a reference picture. Thus, motion estimation unit 42 may perform motion search relative to full pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

[0212] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU to the position of a prediction block of a reference picture. The reference picture may be selected from a first reference picture list (list 0) or a second reference picture list (list 1), where each list identifies one or more reference pictures stored in picture memory 64. Motion estimation unit 42 sends the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44.

[0213] Motion compensation performed by the motion compensation unit 44 may involve obtaining or generating a prediction block based on a motion vector determined by motion estimation, possibly performing interpolation to sub-pixel precision. Upon receiving the motion vector of the PU of the current video block, the motion compensation unit 44 may locate the prediction block pointed to by the motion vector in the reference picture list. The encoding device 104 forms a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded and decoded to form pixel difference values. The pixel difference values ​​form the residual data of the block and may include both a luminance difference component and a chrominance difference component. The adder 50 represents one or more components that perform this subtraction operation. The motion compensation unit 44 may also generate syntax elements associated with the video block and the video slice for use by the decoding device 112 during decoding of the video block of the video slice.

[0214] The intra-prediction processing unit 46 may perform intra-prediction on the current block as an alternative to the inter-prediction performed by the motion estimation unit 42 and the motion compensation unit 44, as described above. Specifically, the intra-prediction processing unit 46 may determine an intra-prediction mode for encoding the current block. In some examples, the intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example during separate encoding passes, and the intra-prediction processing unit 46 may select an appropriate intra-prediction mode from the tested modes to use. For example, the intra-prediction processing unit 46 may calculate rate-distortion values ​​for various tested intra-prediction modes using rate-distortion analysis, and may select the intra-prediction mode with the best rate-distortion characteristics from the tested modes. The rate-distortion analysis typically determines the amount of distortion (or error) between the encoded block and the original unencoded block that has been encoded and produced the encoded block, as well as the bit rate (i.e., the number of bits) used to produce the encoded block. Intra-prediction processing unit 46 may calculate scales from the distortions and rates for the various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0215] In any case, after selecting an intra-prediction mode for a block, intra-prediction processing unit 46 may provide information indicating the intra-prediction mode selected for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode. Encoding device 104 may include in the transmitted bitstream configuration data a definition of the coding contexts for the various blocks and an indication of the most probable intra-prediction mode, an indication of an intra-prediction mode index table, and an indication of a modified intra-prediction mode index table for each of the contexts. The bitstream configuration data may include a plurality of intra-prediction mode index tables and a plurality of modified intra-prediction mode index tables (also referred to as codeword mapping tables).

[0216] After the prediction processing unit 41 generates a prediction block for the current video block via inter-frame prediction or intra-frame prediction, the encoding device 104 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be contained in one or more TUs and applied to the transform processing unit 52. The transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform. The transform processing unit 52 may transform the residual video data from a pixel domain to a transform domain, such as a frequency domain.

[0217] Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of the matrix including the quantized transform coefficients. Alternatively, entropy coding unit 56 may perform the scan.

[0218] After quantization, the entropy coding unit 56 performs entropy coding on the quantized transform coefficients. For example, the entropy coding unit 56 may perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique. After entropy coding unit 56 performs entropy coding, the encoded bitstream may be transmitted to the decoding device 112, or archived for later transmission or retrieval by the decoding device 112. The entropy coding unit 56 may also perform entropy coding on motion vectors and other syntax elements for the current video slice being encoded or decoded.

[0219] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain for later use as a reference block of a reference picture. Motion compensation unit 44 may calculate a reference block by adding the residual block to a prediction block of one of the reference pictures in the reference picture list. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for use in motion estimation. Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in picture memory 64. The reference block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block for inter-frame prediction of a block in a subsequent video frame or picture.

[0220] In this way, Figure 7 The encoding device 104 is configured to perform any of the techniques described herein, including the above referenced Figure 5 Described method 500. In some cases, some of the techniques of the present disclosure may also be implemented by post-processing device 57.

[0221] Figure 7 1 is a block diagram illustrating an example decoding device 112. The decoding device 112 includes an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, a filter unit 91, and a picture memory 92. The prediction processing unit 81 includes a motion compensation unit 82 and an intra-frame prediction processing unit 84. In some examples, the decoding device 112 may perform the same as the reference. Figure 7 The encoding channel described in the encoding device 104 is roughly the opposite of the decoding channel.

[0222] During the decoding process, the decoding device 112 receives an encoded video bitstream representing video blocks and associated syntax elements of an encoded video slice sent by the encoding device 104. In some embodiments, the decoding device 112 may receive the encoded video bitstream from the encoding device 104. In some embodiments, the decoding device 112 may receive the encoded video bitstream from the network entity 79, wherein the network entity 79 includes, for example, a server, a media-aware network element (MANE), a video editor / splicer, or other such device configured to implement one or more of the above-mentioned techniques. The network entity 79 may or may not include the encoding device 104. Some of the techniques described in this disclosure may be implemented by the network entity 79 before the network entity 79 sends the encoded video bitstream to the decoding device 112. In some video decoding systems, the network entity 79 and the decoding device 112 may be parts of separate devices, while in other examples, the functions described with reference to the network entity 79 may be performed by the same device including the decoding device 112.

[0223] The entropy decoding unit 80 of the decoding device 112 performs entropy decoding on the bitstream to generate quantized coefficients, motion vectors and other syntax elements. The entropy decoding unit 80 forwards the motion vectors and other syntax elements to the prediction processing unit 81. The decoding device 112 can receive syntax elements at the video slice level and / or the video block level. The entropy decoding unit 80 can process and parse both fixed-length syntax elements and variable-length syntax elements in one or more parameter sets (e.g., VPS, SPS, and PPS).

[0224] When the video slice is encoded and decoded as an intra-frame coded (I) slice, the intra-frame prediction processing unit 84 of the prediction processing unit 81 can generate prediction data for the video block of the current video slice based on the signaled intra-frame prediction mode and data from the previously decoded block of the current frame or picture. When the video frame is encoded and decoded as an inter-frame coded (i.e., B, P or GPB) slice, the motion compensation unit 82 of the prediction processing unit 81 generates a prediction block for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 80. The prediction block can be generated from one of the reference pictures in the reference picture list. The decoding device 112 can construct the reference frame lists, list 0 and list 1, using a default construction technique based on the reference pictures stored in the picture memory 92.

[0225] Motion compensation unit 82 determines prediction information for a video block of the current video slice by parsing the motion vector and other syntax elements, and uses the prediction information to generate a prediction block for the video block currently being decoded. For example, motion compensation unit 82 may use one or more syntax elements in a parameter set to determine: a prediction mode (e.g., intra-frame or inter-frame prediction) used to encode or decode a video block of a video slice, an inter-frame prediction slice type (e.g., a B slice, a P slice, or a GPB slice), construction information for one or more reference picture lists for the slice, a motion vector for each inter-frame encoded video block of the slice, an inter-frame prediction state for each inter-frame encoded video block of the slice, and other information for decoding a video block in the current video slice.

[0226] The motion compensation unit 82 may also perform interpolation based on an interpolation filter. The motion compensation unit 82 may calculate interpolated values ​​of sub-integer pixels of the reference block using an interpolation filter such as that used by the encoding device 104 during encoding of the video block. In this case, the motion compensation unit 82 may determine the interpolation filter used by the encoding device 104 based on the received syntax elements and may use the interpolation filter to generate a prediction block.

[0227] The inverse quantization unit 86 inverse quantizes or dequantizes the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 80. The inverse quantization process may include using the quantization parameters calculated by the encoding device 104 for each video block in the video slice to determine the degree of quantization and the degree of inverse quantization that should be applied. The inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT or other suitable inverse transform), an inverse integer transform, or a conceptually similar inverse transform process to the transform coefficients in order to produce a residual block in the pixel domain.

[0228] After the motion compensation unit 82 generates a prediction block for the current video block based on the motion vector and other syntax elements, the decoding device 112 forms a decoded video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82. Adder 90 represents one or more components that perform this summation operation. If desired, a loop filter (in the codec loop or after the codec loop) can also be used to smooth pixel transitions or improve video quality. Filter unit 91 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although Figure 8 Filter unit 91 is shown as a loop filter in FIG. 1 , but in other configurations, filter unit 91 may be implemented as a post-loop filter. The decoded video blocks in a given frame or picture are then stored in a picture memory 92, which stores reference pictures for subsequent motion compensation. The picture memory 92 also stores the decoded video for later use in, for example, Figure 1 The video shown in FIG. 1 is presented on a display device of target device 122 .

[0229] in this case, Figure 8 The decoding device 112 represents a device configured to perform any of the techniques described herein (including those described above with reference to Figure 3 An example of a video decoder of the method 300).

[0230] The techniques of the present invention are not necessarily limited to wireless applications or settings. These techniques can be applied to support video encoding and decoding of any of the following multimedia applications: for example, over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission (e.g., dynamic adaptive streaming over HTTP (DASH)), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, the system can be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0231] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media and / or data that can store, contain, or carry instructions. Computer-readable media may include non-temporary media in which data can be stored and do not include carrier waves and / or transient electronic signals that are propagated wirelessly or by wired connections. Examples of non-temporary media may include, but are not limited to, disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or storage devices. Computer-readable media may store codes and / or machine-executable instructions thereon, which may represent any combination of processes, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Code segments may be coupled to another code segment or hardware circuit by transmitting and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be transmitted, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0232] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0233] Specific details are provided in the above description to provide a comprehensive understanding of the embodiments and examples provided herein. However, it should be understood by those of ordinary skill in the art that these embodiments can be practiced without these specific details. For the sake of clarity of explanation, in some examples, the present technology can be presented as including independent functional blocks, which include functional blocks including steps or routines in methods including devices, device components, software or a combination of hardware and software. Additional components other than the components shown in the figure and / or described herein can be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form, so as not to confuse the embodiments in terms of unnecessary details. In other examples, in order to avoid making the embodiments obscure, known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary details.

[0234] The above various embodiments can be described as processes or methods, which are depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams or block diagrams. Although the flow chart can describe the operation as a sequential process, many operations can be performed in parallel or simultaneously. In addition, the order of the operations can be rearranged. The process is terminated when its operation is completed, but may have other steps not included in the figure. The process may correspond to a method, function, process, subroutine, subprogram, etc. When the process corresponds to a function, its termination may correspond to the function returning to the calling function or main function.

[0235] The processes and methods according to the above examples can be implemented using computer executable instructions stored or available from a computer readable medium. Such instructions may include, for example, instructions and data that cause or configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or group of functions. Parts of the computer resources used may be easily accessible via a network. Computer executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the above examples include disks or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, etc.

[0236] The equipment implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may adopt any of a variety of form factors. When implemented in the form of software, firmware, middleware, or microcode, the program code or code segments that perform the necessary tasks (e.g., computer program products) may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or plug-in cards. As another example, such functions may also be implemented on circuit boards between different chips or between different processes executed in a single device.

[0237] Instructions, media for transmitting such instructions, computing resources for executing the same, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0238] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but it should be appreciated by those skilled in the art that the present application is not limited thereto. Therefore, although the illustrative embodiments of the present application are described in detail herein, it should be understood that the inventive concept can be embodied and adopted in many aspects, and the appended claims are intended to be interpreted as including these variations, unless limited to the prior art. The various features and aspects of the above-mentioned application can be used individually or jointly. Moreover, without departing from the broader spirit and scope of this specification, the embodiment can be used in any number of environments and applications beyond the environment and application described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that in an alternative embodiment, these methods can be executed in an order different from the described order.

[0239] It should be understood by those of ordinary skill that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of this specification.

[0240] Where a component is described as being “configured to” perform certain operations, such configuration may be accomplished by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor, or other suitable electronic circuitry) to perform the operations, or any combination thereof.

[0241] The phrase "coupled to" refers to any component that is physically connected to another component, directly or indirectly, and / or any component that communicates directly and / or indirectly with another component (e.g., connected to another component via a wired or wireless connection, and / or other suitable communication interface).

[0242] Claim language or other language stating "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0243] Various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above in terms of their functions. Whether this function is implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. Technicians can implement the described functions in different ways for each specific application, but this implementation decision should not be interpreted as causing deviations from the scope of the application.

[0244] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. These techniques may be implemented in any of a variety of devices. Various devices include, for example, general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses (including applications in wireless communication device mobile phones and other devices). Any features described as modules or components may be implemented together in an integrated logic device or separately implemented in discrete interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including program code. The program code includes instructions for executing one or more of these methods when executed. A computer-readable data storage medium may form part of a computer program product, which may include grouped materials. A computer-readable medium may include a memory or a data storage medium, for example, a random access memory (RAM) such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, etc. Additionally or alternatively, these techniques may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures, such as a propagated signal or wave, that can be accessed, read, and / or executed by a computer.

[0245] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or may be included in a combined video encoder-decoder (codec).

[0246] Illustrative examples of the present disclosure include:

[0247] Example 1: A method for processing video data, the method comprising: obtaining a current picture of the video data; determining that a size of the current picture is different from a size of at least one reference picture; determining that use of a switchable interpolation filter is disabled based on the size of the current picture being different from a size of at least one reference picture; and performing reference picture resampling of at least one reference picture based on the size of the current picture being different from a size of at least one reference picture.

[0248] Example 2. The method of Example 1, wherein the sizing is performed on a block basis.

[0249] Example 3. The method of Example 1, wherein the sizing is performed on a slice basis.

[0250] Example 4. The method of example 1, wherein the sizing is performed on a tile basis.

[0251] Example 5. The method of example 1, wherein the sizing is performed on a sprite basis.

[0252] Example 6. The method of Example 1, wherein the sizing is performed on a sequence level.

[0253] Example 7. The method of any one of Examples 1 to 6, wherein the size of the current picture is a decoded picture size of the current picture.

[0254] Example 8. The method of any one of Examples 1 to 6, wherein the size of the current picture is a cropped picture size of the current picture after applying one or more consistency windows.

[0255] Example 9. The method of any one of Examples 1 to 8, wherein the size of the reference picture is determined based on a decoded picture size of the current picture and one or more offsets.

[0256] Example 10. The method of any one of Examples 1 to 9, further comprising determining a scaling ratio for performing reference picture resampling based on a size of the current picture and a size of the reference picture with the applied offset.

[0257] Example 11. The method of any one of Examples 1 to 10, wherein for a block of the encoded video bitstream having switchable interpolation filtering disabled at a block level, one or more syntax elements associated with the switchable interpolation filtering are not signaled.

[0258] Example 12. The method of Example 11, wherein the one or more syntax elements associated with switchable interpolation filtering include a syntax element indicating a filter index for a block of the current picture.

[0259] Example 13. A method according to any one of Examples 1 to 12, wherein when switchable interpolation filtering is disabled at the block level, and when the filter index of a block is inferred based on one or more neighboring blocks of the block, the filter index is modified and set to a default filter index of the motion vector candidates derived from the neighboring blocks of the block.

[0260] Example 14. The method of Example 13, wherein the default filter index indicates that switchable interpolation filtering is not applied.

[0261] Example 15. The method of any of Examples 13 or 14, wherein a default filter index is ignored for the block and stored with the block for use in one or more subsequent blocks.

[0262] Example 16. The method of any one of Examples 1 to 15, further comprising generating an encoded video bitstream including the current picture.

[0263] Example 17. The method according to any one of Examples 1 to 15 further includes: decoding the current picture.

[0264] Example 18. An apparatus comprises a memory configured to store video data and a processor configured to process the video data according to any of Examples 1 to 17.

[0265] Example 19. The apparatus of Example 18, wherein the apparatus comprises an encoder.

[0266] Example 20. The apparatus of Example 18, wherein the apparatus comprises a decoder.

[0267] Example 21. The apparatus of any one of Examples 18 to 20, wherein the apparatus is a mobile device.

[0268] Example 22. The apparatus of any one of Examples 18 to 21, further comprising a display configured to display video data.

[0269] Example 23. The apparatus of any one of Examples 18 to 22, further comprising a camera configured to capture one or more frames.

[0270] Example 24. A computer readable medium having instructions stored thereon, which instructions, when executed by a processor, perform the method of any one of Examples 1 to 17.

[0271] Example 24. A method for processing video data, the method comprising: obtaining a current picture of the video data; determining whether a size of the current picture is different from a size of at least one reference picture; obtaining a filter index indicating a filter to be selected from a first filter set or a second filter set, the first filter set comprising one or more interpolation filters, and the second set comprising one or more filters for performing reference picture resampling of at least one reference picture; and determining whether to select a filter from the first filter set or the second filter set based on whether the size of the current picture is different from a size of at least one reference picture.

[0272] Example 25. The method according to Example 24 also includes: determining that the size of the current picture and the size of at least one reference picture are the same size; and selecting a filter from the first filter set based on determining that the size of the current picture and the size of at least one reference picture are the same size.

[0273] Example 26. The method according to Example 24 also includes: determining that the size of the current picture is different from the size of at least one reference picture; and selecting a filter from the second filter set based on determining that the size of the current picture is different from the size of at least one reference picture.

[0274] Example 27. An apparatus comprising a memory configured to store video data and a processor configured to process the video data according to any of Examples 24 to 26.

[0275] Example 28. An apparatus according to Example 27, wherein the apparatus comprises an encoder.

[0276] Example 29. The apparatus of Example 27, wherein the apparatus comprises a decoder.

[0277] Example 30. The apparatus of any one of Examples 27 to 29, wherein the apparatus is a mobile device.

[0278] Example 31. The apparatus of any one of Examples 27 to 30, further comprising a display configured to display video data.

[0279] Example 32. The apparatus of any one of Examples 27 to 31, further comprising a camera configured to capture one or more frames.

[0280] Example 33. A computer readable medium having stored thereon instructions that, when executed by a processor, perform the method of any one of Examples 24 to 26.

[0281] Example 34. A device for processing video data, the device comprising: one or more memory units storing instructions; and one or more processors executing the instructions, wherein execution of the instructions by the one or more processors causes the one or more processors to: obtain a current picture and a reference picture; identify filter index information for a current block of the current picture; determine that a first picture size value of the current picture is different from a second picture size value of the reference picture; and based on determining that the first picture size value of the current picture is different from the second picture size value of the reference picture, perform a resampling process without using the current filter index identified by the filter index information.

[0282] Example 35. A device for processing video data, the device comprising: one or more memory units for storing instructions; and one or more processors for executing the instructions, wherein execution of the instructions by the one or more processors causes the one or more processors to: obtain a current picture and a reference picture; identify filter index information for a current block of the current picture; determine that a first picture size value of the current picture is different from a second picture size value of the reference picture; and based on determining that the first picture size value of the current picture is different from the second picture size value of the reference picture, use a default filter index (e.g., a default index value) to replace the current filter index identified by the filter index information to perform a sampling rate conversion process.

[0283] Example 36. An apparatus according to any one of Examples 34 or 35, wherein execution of the instruction by one or more processors further causes the one or more processors to: store a current filter index identified by the filter index information together with the current block; and use the current filter index in motion vector candidate derivation for a subsequent block based on the current filter index identified by the filter index information.

[0284] Example 37. An apparatus according to any one of Examples 34 or 35, wherein execution of the instruction by the one or more processors further causes the one or more processors to: assign a default value to the current filter index to temporarily replace the original value to create a default filter index when the current filter index is not used to perform a sampling rate conversion process; and assign the original value to the current filter index after the sampling rate conversion process.

[0285] Example 38. The apparatus of Example 37, wherein a default value of the current filter index indicates that alternative switchable filtering is not applied as part of the resampling process.

[0286] Example 39. An apparatus according to any one of Examples 34 to 38, wherein the filter index information includes neighboring block information, and wherein the current filter index is derived from the neighboring block information during merge mode motion vector candidate list derivation.

[0287] Example 40. An apparatus according to any one of Examples 34 to 39, wherein the filter index information includes adaptive motion vector resolution (AMVR) mode signaling, and wherein the current filter index is signaled by MVR mode signaling.

[0288] Example 41. An apparatus according to any one of Examples 34 to 40, wherein the sampling rate conversion process is a motion compensation process.

[0289] Example 42. An apparatus according to any one of Examples 34 to 41, wherein the sampling rate conversion process includes a resampling process.

[0290] Example 43. An apparatus according to any one of Examples 34 to 42, wherein the sampling rate conversion process includes an interpolation process.

[0291] Example 44. An apparatus according to any one of Examples 34 to 43, wherein execution of the instructions by the one or more processors further causes the one or more processors to perform inter-frame prediction of a current block of a current picture using a reference picture.

[0292] Example 45. An apparatus according to any one of Examples 34 to 44, wherein execution of the instruction by one or more processors further causes the one or more processors to: determine that a current filter index is not used for a sampling rate conversion process; and replace the value of the current filter index with a default value in response to determining that the current filter index is not used for a sampling rate conversion process.

[0293] Example 46. The apparatus of any of Examples 34 to 45, wherein execution of the instructions by the one or more processors further causes the one or more processors to process a default value to determine that the alternative switchable filter is not applied.

[0294] Example 47. The apparatus of Example 46, wherein the alternative switchable filter is a smoothing filter.

[0295] Example 48. An apparatus according to any one of Examples 34 to 47, wherein in response to determining that the current filter index is not used in a sampling rate conversion process, the current filter index value is not signaled.

[0296] Example 49. An apparatus according to any one of Examples 34 to 48, wherein the current filter index is selected from a filter set, wherein a subset of the filter set is used to perform a sampling rate conversion process, and wherein the remainder of the filter set is not available for sampling rate conversion.

[0297] Example 50. An apparatus according to any of Example 49, wherein the current filter index is selected using a syntax element indicating that the current filter index is prohibited from being selected from the rest of the filter set.

[0298] Example 51. An apparatus according to any one of Examples 34 to 50, wherein the sampling rate conversion process includes a resampling process, and wherein execution of the instruction by one or more processors also causes the one or more processors to: identify a first filter group for interpolation and a second filter group for resampling, wherein the current filter index is from the second filter group.

[0299] Example 52. An apparatus according to Example 51, wherein execution of the instruction by one or more processors further causes the one or more processors to: obtain a second current picture and a second reference picture; determine that a third picture size value of the second current picture and a fourth picture size value of the second reference picture are the same size value; and based on determining that the third picture size value of the second current picture and the fourth picture size value of the second reference picture are the same size value, perform a second resampling process of the second current block using a second selected filter, wherein the second selected filter is from the first filter group.

[0300] Example 53. An apparatus according to any of Examples 51 or 52, wherein each filter in the first filter group and the second filter group is identified by an associated filter index, the associated filter index indicating an associated index value from a filter set for interpolation or a filter set for resampling.

[0301] Example 54. An apparatus according to any one of Examples 51 to 53, wherein execution of the instruction by one or more processors further causes the one or more processors to: obtain a current filter index from the filter index by a lookup operation for a selected filter index value; wherein the selected filter index value is associated with a second filter group; and wherein the association between the selected filter index value and the second filter group is based on determining that the first picture size value is different from the second picture size value.

[0302] Example 55. An apparatus according to any one of Examples 34 to 54, wherein determining that a first picture size value of a current picture is different from a second picture size value of a reference picture comprises determining at least one of: a current picture width is different from a reference picture width; and a current picture height is different from a reference picture height.

[0303] Example 56. The apparatus of any one of Examples 34 to 55, wherein execution of the instructions by the one or more processors further causes the one or more processors to: generate an encoded video bitstream comprising a current picture and a reference picture.

[0304] Example 57. An apparatus according to Example 56, wherein execution of the instruction by one or more processors further causes the one or more processors to: send an encoded video bitstream to a decoding device, the encoded video bitstream being sent together with signaling information, the signaling information including a default filter index.

[0305] Example 58. The apparatus of any of Examples 56 or 57, wherein execution of the instructions by the one or more processors further causes the one or more processors to: store the encoded video bitstream.

[0306] Example 59. An apparatus according to any one of Examples 56 to 58, wherein execution of the instruction by one or more processors further causes the one or more processors to: obtain an encoded video bitstream comprising a current picture and a reference picture; identify signaling information associated with the encoded video bitstream, the signaling information comprising filter index information; and decode a current block of the current picture from the encoded video bitstream.

[0307] Example 60. An apparatus according to any one of Examples 34 to 55, wherein decoding a current block of a current picture from an encoded video bitstream comprises reconstructing the current block based on a default filter index.

[0308] Example 61. An apparatus according to any one of Examples 34 to 60, wherein the apparatus is a mobile device.

[0309] Example 62. An apparatus according to any one of Examples 34 to 61, wherein the apparatus comprises a display configured to display video data.

[0310] Example 63. An apparatus according to any of Examples 34 to 62, wherein the apparatus comprises a camera configured to capture one or more video frames of video data.

[0311] Example 64. A method according to any of Examples 34 to 63 above.

[0312] Example 65. A non-transitory computer-readable storage medium comprising instructions that, when executed by a device, cause the device to perform operations according to any one of Examples 34 to 63 above.

[0313] Example 66. An apparatus comprising means for performing any of the operations according to Examples 34 to 63 above.

Claims

1. A method for processing video data, the method comprising: Get the current picture and the reference picture; identifying filter index information for a current block of the current picture; Determining that a first picture size value of the current picture is different from a second picture size value of the reference picture; as well as Based on determining that the first picture size value of the current picture is different from the second picture size value of the reference picture, a sampling rate conversion process is performed by replacing the current filter index identified by the filter index information with a default filter index.

2. The method according to claim 1, further comprising: storing the current filter index identified by the filter index information together with the current block; as well as The current filter index is used in motion vector candidate derivation of a subsequent block based on the current filter index identified by the filter index information.

3. The method according to claim 1, further comprising: When the current filter index is not used to perform the sampling rate conversion process, assigning a default value to the current filter index to temporarily replace the original value to create the default filter index; as well as The current filter index is assigned the original value after the sampling rate conversion process.

4. The method according to claim 3, wherein: The default value of the current filter index indicates that alternative switchable filtering is not applied as part of the sampling rate conversion process.

5. The method according to claim 1, wherein: The filter index information includes neighboring block information, and wherein the current filter index is derived according to the neighboring block information during derivation of a merge mode motion vector candidate list.

6. The method according to claim 1, wherein: The filter index information includes adaptive motion vector resolution (AMVR) mode signaling, and wherein the current filter index is signaled through the AMVR mode signaling.

7. The method according to claim 1, wherein: The sampling rate conversion process is a motion compensated process.

8. The method according to claim 1, wherein: The sampling rate conversion process includes a resampling process.

9. The method according to claim 1, wherein: The sampling rate conversion process includes an interpolation process.

10. The method of claim 1, further comprising using the reference picture to perform inter-frame prediction of the current block of the current picture.

11. The method according to claim 1, further comprising: determining that the current filter index is not used in the sampling rate conversion process; as well as In response to determining that the current filter index is not used in the sample rate conversion process, a value of the current filter index is replaced with a default value.

12. The method according to claim 11, further comprising: The default value is processed to determine that the alternative switchable filter is not applied.

13. The method according to claim 12, wherein: The alternative switchable filter is a smoothing filter.

14. The method according to claim 1, wherein: In response to determining that the current filter index is not used in the sampling rate conversion process, a current filter index value is not signaled.

15. The method according to claim 1, wherein: The current filter index is selected from a filter set, wherein a subset of the filter set is used to perform the sample rate conversion process, and wherein a remainder of the filter set is unavailable for sample rate conversion.

16. The method according to claim 15, wherein: The current filter index is selected using a syntax element indicating that selection of the current filter index from the remaining portion of the filter set is prohibited.

17. The method according to claim 1, wherein: The sampling rate conversion process includes a resampling process, and further includes: A first filter bank for interpolation and a second filter bank for resampling are identified, wherein the current filter index is from the second filter bank.

18. The method according to claim 17, further comprising: Obtain a second current picture and a second reference picture; Determining that a third picture size value of the second current picture and a fourth picture size value of the second reference picture are the same size value; as well as Based on determining that the third picture size value of the second current picture and the fourth picture size value of the second reference picture are the same size value, a second resampling process of the second current block is performed using a second selected filter, wherein the second selected filter is from the first filter group.

19. The method according to claim 17, wherein: Each filter in the first filter bank and the second filter bank is identified by an associated filter index indicating an associated index value from a filter set for interpolation or a filter set for resampling.

20. The method of claim 17, further comprising: Obtaining the current filter index from the filter index by a lookup operation for the selected filter index value; wherein the selected filter index value is associated with the second filter bank; and Wherein the association between the selected filter index value and the second filter bank is based on a determination that the first picture size value is different from the second picture size value.

21. The method according to claim 1, wherein: Determining that the first picture size value of the current picture is different from the second picture size value of the reference picture includes determining at least one of: The current picture width is different from the reference picture width; and The current picture height is different from the reference picture height.

22. The method of claim 1, further comprising: An encoded video bitstream including the current picture and the reference picture is generated.

23. The method according to claim 22, further comprising: The encoded video bitstream is sent to a decoding device, the encoded video bitstream is sent together with signaling information, the signaling information includes the default filter index.

24. The method of claim 22, further comprising: The encoded video bitstream is stored.

25. The method of claim 1, further comprising: Obtaining a coded video bitstream including the current picture and the reference picture; identifying signaling information associated with the encoded video bitstream, the signaling information comprising the filter index information; as well as The current block of the current picture is decoded from the encoded video bitstream.

26. The method according to claim 25, wherein: Decoding the current block of the current picture from the encoded video bitstream includes reconstructing the current block based on the default filter index.

27. An apparatus for processing video data, the apparatus comprising: one or more memory cells for storing instructions; as well as one or more processors executing the instructions, wherein execution of the instructions by the one or more processors causes the one or more processors to: Get the current picture and the reference picture; identifying filter index information for a current block of the current picture; Determining that a first picture size value of the current picture is different from a second picture size value of the reference picture; as well as Based on determining that the first picture size value of the current picture is different from the second picture size value of the reference picture, a sampling rate conversion process is performed by replacing a current filter index identified by the filter index information with a default filter index.

28. The device according to claim 27, wherein Execution of the instructions by the one or more processors further causes the one or more processors to: storing the current filter index identified by the filter index information together with the current block; and The current filter index is used in motion vector candidate derivation of a subsequent block based on the current filter index identified by the filter index information.

29. The device according to claim 27, wherein: Execution of the instructions by the one or more processors further causes the one or more processors to: When the current filter index is not used to perform the sampling rate conversion process, assigning a default value to the current filter index to temporarily replace the original value to create the default filter index; as well as The current filter index is assigned the original value after the sampling rate conversion process.

30. The device according to claim 29, wherein: The default value of the current filter index indicates that alternative switchable filtering is not applied as part of the sampling rate conversion process.

31. The apparatus of claim 27, wherein: The filter index information includes neighboring block information, and wherein the current filter index is derived according to the neighboring block information during derivation of a merge mode motion vector candidate list.

32. The apparatus of claim 27, wherein: The filter index information includes adaptive motion vector resolution (AMVR) mode signaling, and wherein the current filter index is signaled through the AMVR mode signaling.

33. The apparatus of claim 27, wherein: The sampling rate conversion process is a motion compensated process.

34. The apparatus of claim 27, wherein: The sampling rate conversion process includes a resampling process.

35. The apparatus of claim 27, wherein: The sampling rate conversion process includes an interpolation process.

36. The apparatus of claim 27, wherein: Execution of the instructions by the one or more processors further causes the one or more processors to use the reference picture to perform inter-frame prediction of the current block of the current picture.

37. The apparatus of claim 27, wherein: Execution of the instructions by the one or more processors further causes the one or more processors to: determining that the current filter index is not used in the sample rate conversion process; and In response to the determination that the current filter index is not used in the sample rate conversion process, a value of the current filter index is replaced with a default value.

38. The device according to claim 37, wherein Execution of the instructions by the one or more processors further causes the one or more processors to process the default value to determine that the alternative switchable filter is not applied.

39. The device according to claim 38, wherein The alternative switchable filter is a smoothing filter.

40. The apparatus of claim 27, wherein: In response to determining that the current filter index is not used in the sampling rate conversion process, a current filter index value is not signaled.

41. The apparatus of claim 27, wherein: The current filter index is selected from a filter set, wherein a subset of the filter set is used to perform the sample rate conversion process, and wherein a remainder of the filter set is unavailable for sample rate conversion.

42. The device according to claim 41, wherein The current filter index is selected using a syntax element indicating that selection of the current filter index from the remaining portion of the filter set is prohibited.

43. The apparatus of claim 27, wherein: The sampling rate conversion process includes a resampling process, and wherein execution of the instructions by the one or more processors further causes the one or more processors to: A first filter bank for interpolation and a second filter bank for resampling are identified, wherein the current filter index is from the second filter bank.

44. The device according to claim 43, wherein Execution of the instructions by the one or more processors further causes the one or more processors to: Obtain a second current picture and a second reference picture; Determining that a third picture size value of the second current picture and a fourth picture size value of the second reference picture are the same size value; as well as Based on determining that the third picture size value of the second current picture and the fourth picture size value of the second reference picture are the same size value, a second resampling process of the second current block is performed using a second selected filter, wherein the second selected filter is from the first filter group.

45. The apparatus of claim 43, wherein: Each filter in the first filter bank and the second filter bank is identified by an associated filter index indicating an associated index value from a filter set for interpolation or a filter set for resampling.

46. ​​The apparatus of claim 27, wherein: Execution of the instructions by the one or more processors further causes the one or more processors to: Obtaining the current filter index from the filter index by a lookup operation for the selected filter index value; wherein the selected filter index value is associated with a second filter bank; and Wherein the association between the selected filter index value and the second filter bank is based on a determination that the first picture size value is different from the second picture size value.

47. The apparatus of claim 27, wherein: Determining that the first picture size value of the current picture is different from the second picture size value of the reference picture includes determining at least one of: The current picture width is different from the reference picture width; and The current picture height is different from the reference picture height.

48. The apparatus of claim 27, wherein: Execution of the instructions by the one or more processors causes the one or more processors to: An encoded video bitstream including the current picture and the reference picture is generated.

49. The apparatus of claim 48, wherein: Execution of the instructions by the one or more processors causes the one or more processors to: The encoded video bitstream is sent to a decoding device, the encoded video bitstream is sent together with signaling information, the signaling information includes the default filter index.

50. The apparatus of claim 48, wherein: Execution of the instructions by the one or more processors causes the one or more processors to: The encoded video bitstream is stored.

51. The apparatus of claim 27, wherein: Execution of the instructions by the one or more processors causes the one or more processors to: Obtaining a coded video bitstream including the current picture and the reference picture; identifying signaling information associated with the encoded video bitstream, the signaling information comprising the filter index information; as well as The current block of the current picture is decoded from the encoded video bitstream.

52. The apparatus of claim 51, wherein: Decoding the current block of the current picture from the encoded video bitstream includes reconstructing the current block based on the default filter index.

53. The apparatus of claim 27, wherein: The device is a mobile device.

54. The apparatus of claim 27, wherein: The apparatus comprises a display configured to display the video data.

55. The apparatus of claim 27, wherein: The apparatus includes a camera configured to capture one or more video frames of the video data.

Images