Adaptive resolution change in video processing

Abstract

To provide systems and methods for performing adaptive resolution change during video encoding and decoding.SOLUTION: The methods include: comparing resolutions of a target picture and a first reference picture; in response to the target picture and first reference picture having different resolutions, resampling the first reference picture to generate a second reference picture; and encoding or decoding the target picture using the second reference picture.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This disclosure claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 865,927, filed June 24, 2019, and U.S. Provisional Patent Application No. 62 / 900,439, filed September 13, 2019, both of which are incorporated herein by reference in their entireties.

[0002] Technical Field FIELD OF THE DISCLOSURE

[0002] The present disclosure relates generally to video processing, and more particularly, to a method and system for adaptive resolution change in video coding. [Background technology]

[0003] background

[0003] A video is a sequence of still pictures (or "frames") that capture visual information. To reduce storage memory and transmission bandwidth, a video may be compressed before storage or transmission and decompressed before display. The compression process is usually called encoding, and the decompression process is usually called decoding. There are various video coding formats that use standardized video coding techniques, most commonly based on prediction, transformation, quantization, entropy coding, and in-loop filtering. Video coding standards, such as the High Efficiency Video Coding (HEVC) / H.265 standard, the Versatile Video Coding (VVC) / H.266 standard, and the AVS standard, that specify specific video coding formats, are developed by standardization organizations. As more and more advanced video coding techniques are adopted into video standards, the coding efficiency of the new video coding standards becomes higher and higher. Summary of the Invention [Means for solving the problem]

[0004] Disclosure Summary

[0004] An embodiment of the present disclosure provides a method for adaptive resolution change during video encoding or decoding. In an example embodiment, the method includes: comparing resolutions of a target picture and a first reference picture; in response to the target picture and the first reference picture having different resolutions, resampling the first reference picture to generate a second reference picture; and encoding or decoding the target picture using the second reference picture.

[0005]

[0005] The present disclosure also provides a device for adaptive resolution change during video encoding or decoding. In an exemplary embodiment, the device includes one or more memories that store computer instructions, and one or more processors configured to execute the computer instructions to cause the device to: compare resolutions of a target picture and a first reference picture; in response to the target picture and the first reference picture having different resolutions, resample the first reference picture to generate a second reference picture; and use the second reference picture to encode or decode the target picture.

[0006]

[0006] An embodiment of the present disclosure provides a non-transitory computer-readable medium storing a set of instructions, the set of instructions being executable by at least one processor of a computer system to cause the computer system to perform a method of adaptive resolution change. In an example embodiment, the method includes: comparing resolutions of a target picture and a first reference picture; in response to the target picture and the first reference picture having different resolutions, resampling the first reference picture to generate a second reference picture; and encoding or decoding the target picture using the second reference picture.

[0007] BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments and various aspects of the present disclosure are set forth in the following detailed description and the accompanying drawings, in which various features are not drawn to scale. [Brief description of the drawings]

[0008] [Figure 1] 1 is a schematic diagram illustrating an example video encoder consistent with disclosed embodiments. [Diagram 2]

[0009] 1 is a schematic diagram illustrating an exemplary video decoder consistent with disclosed embodiments. [Diagram 3]

[0010] 1 illustrates an example where the resolution of a reference picture is different from that of a current picture, consistent with disclosed embodiments. [Figure 4]

[0011] 1 is a table illustrating sub-pel motion compensation interpolation filters used for the luma component in Versatile Video Coding (VVC), consistent with disclosed embodiments. [Diagram 5]

[0012] 1 is a table illustrating sub-pel motion compensation interpolation filters used for chroma components in VVC, consistent with disclosed embodiments. [Figure 6]

[0013] 1 illustrates an example reference picture buffer consistent with disclosed embodiments. [Figure 7]

[0014] 1 is a table illustrating an example of a supported resolution set that includes three different resolutions, consistent with disclosed embodiments. [Figure 8]

[0015] 1 illustrates an example decoded picture buffer (DPB) where both resampled and original reference pictures are stored, consistent with disclosed embodiments. [Figure 9]

[0016] 1 illustrates progressive downsampling, consistent with disclosed embodiments. [Figure 10]

[0017] 1 illustrates an example video encoding process with resolution conversion consistent with disclosed embodiments. [Figure 11]

[0018] 1 is a table illustrating example downsampling filters consistent with disclosed embodiments. [Figure 12]

[0019] 1 illustrates an example video encoding process using a peak signal-to-noise ratio (PSNR) calculation consistent with disclosed embodiments. [Figure 13]

[0020] 1 illustrates a frequency response of an example low pass filter consistent with disclosed embodiments. [Figure 14]

[0021] 1 is a table illustrating a 6-tap filter, consistent with an embodiment of the disclosure. [Figure 15]

[0022] 1 is a table illustrating an 8-tap filter, consistent with an embodiment of the disclosure. [Figure 16]

[0023] 1 is a table illustrating a 4-tap filter, consistent with an embodiment of the disclosure. [Figure 17]

[0024] 11 is a table showing interpolation filter coefficients for 2:1 downsampling, consistent with disclosed embodiments. [Figure 18]

[0025] 11 is a table showing interpolation filter coefficients for 1.5:1 downsampling, consistent with disclosed embodiments. [Figure 19]

[0026] 1 illustrates an example luma sample interpolation filtering process for reference downsampling, consistent with disclosed embodiments. [Figure 20]

[0027] 1 illustrates an example chroma sample interpolation filtering process for reference downsampling, consistent with disclosed embodiments. [Figure 21]

[0028] 1 illustrates an example luma sample interpolation filtering process for reference downsampling, consistent with disclosed embodiments. [Figure 22]

[0029] 1 illustrates an example chroma sample interpolation filtering process for reference downsampling, consistent with disclosed embodiments. [Figure 23]

[0030] 1 illustrates an example luma sample interpolation filtering process for reference downsampling, consistent with disclosed embodiments. [Figure 24]

[0031] 1 illustrates an example chroma sample interpolation filtering process for reference downsampling, consistent with disclosed embodiments. [Diagram 25]

[0032] 1 illustrates an example chroma fractional sample position calculation for reference downsampling, consistent with disclosed embodiments. [Figure 26] 1 is a table illustrating an 8-tap filter for MC interpolation using 2:1 ratio reference downsampling, consistent with disclosed embodiments. [Figure 27] 2 is a table illustrating an 8-tap filter for MC interpolation using reference downsampling at a rate of 1.5:1, consistent with an embodiment of the disclosure. [Figure 28] 3 is a table illustrating an 8-tap filter for MC interpolation using 2:1 ratio reference downsampling, consistent with an embodiment of the disclosure. [Figure 29] 4 is a table illustrating an 8-tap filter for MC interpolation using reference downsampling at a rate of 1.5:1, consistent with an embodiment of the disclosure. [Diagram 30] 5 is a table showing 6-tap filter coefficients for luma 4×4 block MC interpolation with 2:1 ratio reference downsampling, consistent with disclosed embodiments. [Diagram 31] 1 is a table showing 6-tap filter coefficients for luma 4×4 block MC interpolation using 1.5:1 ratio reference downsampling, consistent with disclosed embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Detailed Description

[0007] Reference will now be made in detail to the exemplary embodiments shown in the accompanying drawings. The following description will refer to the accompanying drawings, in which the same numbers in different drawings represent the same or similar elements unless otherwise stated. The implementations described in the following description of the exemplary embodiments do not represent all implementations consistent with the present invention. Instead, they are merely examples of apparatuses and methods consistent with aspects related to the present invention as described in the appended claims. Certain aspects of the present disclosure will now be described in more detail. In the event of a conflict with the incorporated terms and / or definitions, the terms and definitions provided herein will take precedence.

[0010]

[0008] A video is a series of still pictures (or "frames") arranged in time sequence to preserve visual information. A video capture device (e.g., a camera) can be used to capture and store these pictures in time sequence, and a video playback device (e.g., a television, a computer, a smartphone, a tablet computer, a video player, or any end-user terminal with a display function) can be used to display such pictures in time sequence. In some applications, the video capture device can also transmit the captured video to a video playback device (e.g., a computer with a monitor) in real time, such as for surveillance, conference hosting, or live broadcasting.

[0011]

[0009] To reduce the storage space and transmission bandwidth required in such applications, the video may be compressed. For example, the video may be compressed before storage and transmission and decompressed before display. The compression and decompression may be performed by software executed by a processor (e.g., a processor of a general-purpose computer) or dedicated hardware. The module for compression is generally called an "encoder" and the module for decompression is generally called a "decoder". The encoders and decoders may be collectively referred to as a "codec". The encoders and decoders may be implemented as any of a variety of suitable hardware, software, or combinations thereof. For example, hardware implementations of the encoders and decoders may include circuitry such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, or any combination thereof. Software implementations of the encoders and decoders may include program code, computer executable instructions, firmware, or any suitable computer-implemented algorithm or process fixed on a computer-readable medium. Video compression and decompression may be performed by a variety of algorithms or standards, such as MPEG-1, MPEG-2, MPEG-4, the H.26x family, etc. In some applications, a codec may decompress video from a first encoding standard and recompress the decompressed video using a second encoding standard, in which case the codec may be referred to as a "transcoder."

[0012]

[0010] The video encoding process can identify and retain useful information that can be used for picture reconstruction. If the information ignored in the video encoding process cannot be completely reconstructed, the encoding process may be called "lossy". Otherwise, it may be called "lossless". Most encoding processes are lossy, which is a tradeoff to reduce the required storage space and transmission bandwidth.

[0013]

[0011] In many cases, useful information of a picture being encoded (called the "current picture") may include changes with respect to a reference picture (e.g., a previously encoded or reconstructed picture). Such changes may include pixel position changes, luminance changes, or color changes, among which position changes are of most concern. Position changes of pixels representing an object may reflect the object's motion between the reference picture and the current picture.

[0014]

[0012] To achieve the same subjective quality as HEVC / H.265 at half the bandwidth, JVET is developing a technology that goes beyond HEVC using the joint exploration model ("JEM") reference software. As the coding technology is incorporated into JEM, JEM has achieved significantly higher coding performance than HEVC. VCEG and MPEG have also officially started development of the next-generation video compression standard (Versatile Video Coding (VVC / H.266) standard) that will exceed HEVC.

[0015]

[0013] The VVC standard continues to add more encoding techniques that provide better compression performance. VVC can be implemented in the same video encoding system that has been used in modern video compression standards such as HEVC, H.264 / AVC, MPEG2, H.263, etc. FIG. 1 is a schematic diagram illustrating an exemplary video encoder 100 consistent with disclosed embodiments. For example, the video encoder 100 may perform intra- or inter-coding of blocks in a video frame, including video blocks, or partitions or sub-partitions of video blocks. Intra-coding may rely on spatial prediction to reduce or remove spatial redundancy of video in a given video frame. Inter-coding may rely on temporal prediction to reduce or remove temporal redundancy of video in adjacent frames of a video sequence. Intra-modes may refer to several spatial-based compression modes, and inter-modes (such as uni-predictive or bi-predictive) may refer to several temporal-based compression modes.

[0016]

[0014] Referring to FIG. 1, the input video signal 102 may be processed block by block. For example, a video block unit may be a 16x16 pixel block (e.g., macroblock (MB)). In HEVC, extended block sizes (e.g., coding units (CUs)) may be used to compress video signals with resolutions of, for example, 1080p or higher. In HEVC, a CU may include up to 64x64 luma samples and corresponding chroma samples. In VVC, the size of a CU may be further increased to include 128x128 luma samples and corresponding chroma samples. The CU may be partitioned into prediction units (PUs) to which separate prediction methods may be applied. Each input video block (e.g., MB, CU, PU, ​​etc.) may be processed by using a spatial prediction unit 160 or a temporal prediction unit 162.

[0017]

[0015] Spatial prediction unit 160 performs spatial prediction (e.g., intra prediction) for the current CU using information about the same picture / slice that contains the current CU. Spatial prediction may use pixels from already coded neighboring blocks in the same video picture / slice to predict the current video block. Spatial prediction can reduce spatial redundancy inherent in video signals. Temporal prediction (e.g., inter prediction or motion compensated prediction) may use samples from already coded video pictures to predict the current video block. Temporal prediction can reduce temporal redundancy inherent in video signals.

[0018]

[0016] Temporal prediction unit 162 performs temporal prediction (e.g., inter prediction) on the current CU using information from one or more pictures / slices different from the picture / slice containing the current CU. The temporal prediction for the video block may be signaled by one or more motion vectors. The motion vector may indicate the amount and direction of motion between the current block and one or more of the current block's predictive blocks in the reference frame. If multiple reference pictures are supported, one or more reference picture indexes may be sent for the video block. The one or more reference indexes may be used to identify which one or more reference pictures in decoded picture buffer (DPB) 164 (also referred to as reference picture store 164) the temporal prediction signal may come from. After spatial prediction or temporal prediction, mode decision and encoder control unit 180 in the encoder may choose a prediction mode based on, for example, a rate-distortion optimization method. The predictive block may be subtracted from the current video block at adder 116. The prediction residual may be transformed by transform unit 104 and quantized by quantization unit 106. The quantized residual coefficients may be inverse quantized in inverse quantization unit 110 and inverse transformed in inverse transform unit 112 to form a reconstructed residual. The reconstructed block may be added to the prediction block at summer 126 to form a reconstructed video block. In-loop filtering, such as a deblocking filter and adaptive loop filter 166, may be applied to the reconstructed video block before it is added to reference picture store 164 and used to encode future video blocks. To form output video bitstream 120, the coding mode (e.g., inter or intra), prediction mode information, motion information, and the quantized residual coefficients may be sent to entropy coding unit 108 to be compressed and packed to form bitstream 120.

[0019]

[0017] Consistent with the disclosed embodiments, the above-mentioned units of the video encoder 100 may be implemented as software modules (e.g., computer programs implementing different functions), hardware components (e.g., different circuitry blocks for performing respective functions), or a hybrid of software and hardware.

[0020]

[0018] Figure 2 is a schematic diagram illustrating an exemplary video decoder 200 consistent with embodiments of the disclosure. With reference to Figure 2, a video bitstream 202 may be unpacked or entropy decoded in an entropy decoding unit 208. Coding mode or prediction information may be sent to a spatial prediction unit 260 (e.g., in the case of intra-coding) or a temporal prediction unit 262 (e.g., in the case of inter-coding) to form a prediction block. If inter-coding, the prediction information may include a prediction block size, one or more motion vectors (e.g., which may indicate the direction and amount of motion), or one or more reference indexes (e.g., which may indicate from which reference picture the prediction signal is obtained).

[0021]

[0019] Motion compensation prediction may be applied by the temporal prediction unit 262 to form a temporal prediction block. The residual transform coefficients may be sent to the inverse quantization unit 210 and the inverse transform unit 212 to reconstruct a residual block. The prediction block and the residual block may be summed at 226. The reconstructed block may undergo in-loop filtering (by a loop filter 266) before being stored in a decoded picture buffer (DPB) 264 (also called reference picture storage 264). The reconstructed image in the DPB 264 may be used to drive a display device or to predict future video blocks. The decoded image 220 may be displayed on a display.

[0022]

[0020] Consistent with the disclosed embodiments, the above-mentioned units of the video decoder 200 may be implemented as software modules (e.g., computer programs implementing different functions), hardware components (e.g., different circuitry blocks for performing respective functions), or a hybrid of software and hardware.

[0023]

[0021] One of the objectives of the VVC standard is to provide videoconferencing applications with the ability to tolerate network and device diversity. In particular, the VVC standard needs to provide the ability to rapidly adapt to changing network environments, including rapidly decreasing the encoding bit rate when network conditions deteriorate and rapidly increasing the video quality when network conditions improve. In addition, for adaptive streaming services that provide multiple representations of the same content, each of the multiple representations may have different characteristics (e.g., spatial resolution or sample bit depth), and the video quality may vary from low to high. Therefore, the VVC standard needs to support fast representation switching for adaptive streaming services. During switching from one representation to another (such as switching from one resolution to another), the VVC standard needs to enable the use of an efficient prediction structure without compromising the fast and seamless switching capability.

[0024] In some embodiments consistent with this disclosure, to change resolution, an encoder (e.g., encoder 100 of FIG. 1) sends an instantaneous-decoder-refresh (IDR) coded picture to flush the contents of a reference picture buffer (e.g., DPB 164 of FIG. 1 and DPB 264 of FIG. 2). Upon receiving an IDR coded picture, a decoder (e.g., decoder 200 of FIG. 2) marks all pictures in the reference buffer as “not used for reference”. All subsequent transmitted pictures can be decoded without referencing any frames decoded prior to the IDR picture. The first picture in a coded video sequence is always an IDR picture.

[0025]

[0023] In some embodiments consistent with this disclosure, adaptive resolution change (ARC) techniques may be used to enable a video stream to change spatial resolution between coded pictures in the same video sequence without requiring new IDR pictures and without requiring multiple layers as in the case of scalable video codecs. According to the ARC technique, at a resolution switch point, the currently coded picture is predicted from a reference picture of the same resolution (if available) or from a reference picture of a different resolution by resampling the reference picture. A diagram of adaptive resolution change is shown in FIG. 3, where the resolution of reference picture "Ref0" is the same as the resolution of the currently coded picture. However, the resolutions of reference pictures "Ref1" and "Ref2" are different from the resolution of the current picture. To generate a motion compensated prediction signal for the current picture, both "Ref1" and "Ref2" are resampled to the resolution of the current picture.

[0026]

[0024] In accordance with the present disclosure, when the resolution of the reference frame is different from that of the current frame, one way to generate a motion-compensated prediction signal is picture-based resampling, in which the reference picture is first resampled to the same resolution as the current picture, and the existing motion compensation process using motion vectors can be applied. The motion vectors can be scaled (if the motion vectors are sent in units before the resampling is applied) or not scaled (if the motion vectors are sent in units after the resampling is applied). When using picture-based resampling, especially for reference picture downsampling (i.e., when the resolution of the reference picture is larger than that of the current picture), information can be lost in the reference resampling step before motion compensation interpolation, because downsampling is usually achieved by low-pass filtering followed by decimation.

[0027]

[0025] Another method is block-based resampling, where resampling is done at the block level by looking at one or more reference pictures used by the current block, and if one or both of them have a different resolution than the current picture, resampling is done in combination with a sub-pel motion compensation interpolation process.

[0028]

[0026] This disclosure provides both picture-based and block-based resampling methods for use with ARC. The following description first refers to the disclosed picture-based resampling method, and then refers to the disclosed block-based resampling method.

[0029]

[0027] The disclosed picture-based resampling method can solve some problems caused by conventional block-based resampling methods. First, in conventional block-level resampling methods, on-the-fly block-level resampling may be performed when it is determined that the resolution of the reference picture is different from that of the current picture. However, the block-level resampling process may complicate the encoder design, since the encoder may have to resample the block at each search point during motion search. Motion search is generally a time-consuming process for the encoder, and therefore the on-the-fly requirement during motion search may complicate the motion search process. As an alternative to on-the-fly block-level resampling, the encoder may resample the reference picture in advance, so that the resampled reference picture has the same resolution as the current picture. However, this may reduce the coding efficiency, since the prediction signal during motion estimation and motion compensation may be different.

[0030]

[0028] Secondly, in the conventional block-based resampling method, the design of block-level resampling is incompatible with some other useful coding tools, such as subblock-based temporal motion vector prediction (SbTMVP), affine motion compensation prediction, and decoder side motion vector refinement (DMVR). When ARC is enabled, these coding tools can be disabled. However, such disabling significantly reduces coding performance.

[0031]

[0029] Third, any sub-pel motion compensation interpolation filter known in the art may be used for block resampling, but it may not be equally applicable to block upsampling and block downsampling. For example, when the reference picture has a lower resolution than the current picture, the interpolation filter used for motion compensation may be used for upsampling. However, when the reference picture has a higher resolution than the current picture, the interpolation filter is not suitable for downsampling because the interpolation filter cannot filter integer positions and therefore may cause aliasing. For example, Table 1 (FIG. 4) shows exemplary interpolation filter coefficients used for luma component integer positions with various values ​​of fractional sample positions. Table 2 (FIG. 5) shows exemplary interpolation filter coefficients used for chroma components with various values ​​of fractional sample positions. As shown in Table 1 (FIG. 4) and Table 2 (FIG. 5), at fractional sample position 0 (i.e., integer positions), no interpolation filter is applied.

[0032]

[0030] To avoid the above problems associated with conventional block-based resampling methods, the present disclosure provides a picture-level resampling method that can be used for ARC. The picture resampling process may involve upsampling or downsampling. Upsampling is to increase the spatial resolution while maintaining a two-dimensional (2D) representation of an image. In the upsampling process, the resolution of a reference picture is increased by interpolating unavailable samples from neighboring available samples. In the downsampling process, the resolution of a reference picture is decreased.

[0033]

[0031] According to the disclosed embodiment, resampling can be performed at the picture level. In the picture level resampling, if the resolution of the reference picture is different from that of the current picture, the reference picture is resampled to the resolution of the current picture. The motion estimation and / or compensation of the current picture can be performed based on the resampled reference picture. In this way, ARC can be implemented in a "transparent" manner in the encoder and the decoder, since the block level operations are independent of the resolution change.

[0034]

[0032] Picture-level resampling may be performed on the fly, i.e., while the current picture is predicted. In some exemplary embodiments, only original (non-resampled) reference pictures are stored in a decoded picture buffer (DPB), e.g., DPB 164 in the encoder 100 (FIG. 1) and DBP 264 in the decoder 200 (FIG. 2). The DPB may be managed in the same manner as in the current version of VVC. In the disclosed picture-level resampling method, before encoding or decoding of a picture, the encoder or decoder resamples the reference picture if the resolution of the reference picture in the DPB is different from the resolution of the current picture. In some embodiments, a resampled picture buffer may be used to store all resampled reference pictures for the current picture. The resampled pictures of the reference pictures are stored in the resampled picture buffer, and the motion search / compensation of the current picture is performed using pictures from the resampled picture buffer. After the encoding or decoding is completed, the resampled picture buffer is removed. FIG. 6 shows an example of on-the-fly picture-level resampling, in which a low-resolution reference picture is resampled and stored in a resampled picture buffer. As shown in FIG. 6, the DPB of an encoder or decoder contains three reference pictures. The resolutions of reference pictures "Ref0" and "Ref2" are the same as the resolution of the current picture. Therefore, "Ref0" and "Ref2" do not need to be resampled. However, the resolution of reference picture "Ref1" is different from the resolution of the current picture, and therefore needs to be resampled. Thus, only the resampled "Ref1" is stored in the reference picture buffer, and "Ref0" and "Ref2" are not stored in the reference picture buffer.

[0035]

[0033] In accordance with some exemplary embodiments, in on-the-fly picture-level resampling, the number of resampled reference pictures that a picture can use is constrained. For example, the maximum number of resampled reference pictures for a given current picture may be preset. For example, the maximum number may be set to be 1. In this case, a bitstream constraint may be imposed such that an encoder or decoder can tolerate at most one of the reference pictures having a different resolution than the resolution of the current picture, and all of the other reference pictures must have the same resolution. This maximum number is directly related to the worst-case decoder complexity, since it indicates the size of the resampled picture buffer and the maximum number of resamplings that can be performed on any picture of the current video sequence. Therefore, this maximum number may be signaled as part of the sequence parameter set (SPS) or picture parameter set (PPS), or may be specified as part of the profile / level definition.

[0036]

[0034] In some exemplary embodiments, two versions of a reference picture are stored in the DPB. One version has the original resolution, and the other version has the full resolution. If the resolution of the current picture is different from the original resolution or the full resolution, the encoder or decoder may perform on-the-fly downsampling from the stored full resolution picture. For picture output, the DPB always outputs the original (non-resampled) reference picture.

[0037]

[0035] In some exemplary embodiments, the resampling ratio can be selected arbitrarily and the vertical and horizontal scaling ratios may be different. Because picture-level resampling is applied, the block-level operations of the encoding / decoder are made independent of the resolution of the reference picture, thereby allowing arbitrary resampling ratios to be possible without further complicating the block-level design logic of the encoder / decoder.

[0038]

[0036] In some exemplary embodiments, the signaling of the coding resolution and the maximum resolution can be done as follows: The maximum resolution of any picture of the video sequence is signaled in the SPS. The coding resolution of a picture can be signaled in the PPS or in the slice header. In both cases, one flag is signaled indicating whether the coding resolution is the same as the maximum resolution or not. If the coding resolution is not the same as the maximum resolution, the coding width and height of the current picture are further signaled. If PPS signaling is used, the signaled coding resolution applies to all pictures that refer to this PPS. If slice header signaling is used, the signaled coding resolution applies only to the current picture itself. In some embodiments, the difference between the current coding resolution and the signaled maximum resolution can be signaled in the SPS.

[0039]

[0037] In some exemplary embodiments, instead of using any resampling ratio, the resolution is limited to a predefined set of N supported resolutions (N is the number of supported resolutions in the sequence). The values ​​of N and the supported resolutions can be signaled in the SPS. The coding resolution of a picture is signaled using the PPS or slice header. In both cases, instead of signaling the actual coding resolution, the corresponding resolution index is signaled. Table 3 (Figure 7) shows an example of a supported resolution set where a coding sequence allows three different resolutions. If the coding resolution of the current picture is 1440x816, the corresponding index value (=1) is signaled using the PPS or slice header.

[0040]

[0038] In some exemplary embodiments, both resampled and non-resampled reference pictures are stored in the DPB. For each reference picture, one original (i.e., non-resampled) picture and N-1 resampled copies are stored. If the resolution of the reference picture is different from the resolution of the current picture, the resampled picture in the DPB is used to encode the current picture. For picture output, the DPB outputs the original (non-resampled) reference picture. As an example, FIG. 8 shows the occupancy rate of the DPB of the support resolution set shown in Table 3 (FIG. 7). As shown in FIG. 8, the DPB contains N (e.g., N=3) copies of each picture used as a reference picture.

[0041]

[0039] The choice of the value of N depends on the application. Larger values ​​of N provide more flexibility in resolution selection, but increase complexity and memory requirements. Small values ​​of N are suitable for less complex devices, but limit the resolutions that can be selected. Therefore, in some embodiments, flexibility may be given to the encoder to determine the value of N signaled by the SPS based on the application and device capabilities.

[0042]

[0040] Consistent with the present disclosure, downsampling of the reference picture may be performed progressively (i.e., gradually). In conventional (i.e., direct) downsampling, downsampling of the input image is performed in a single step. If the downsampling ratio is high, single-step downsampling requires a longer tap downsampling filter to avoid severe aliasing problems. However, longer tap filters are computationally expensive. In some exemplary embodiments, in order to maintain the quality of the downsampled image, progressive downsampling is used, in which downsampling is performed gradually when the downsampling ratio is greater than a threshold (e.g., greater than 2:1 downsampling). For example, to achieve a downsampling ratio greater than 2:1, a downsampling filter sufficient for 2:1 downsampling may be repeatedly applied. Figure 9 shows an example of progressive downsampling, in which a quarter downsampling is performed across two pictures in both horizontal and vertical dimensions. The first picture is downsampled by half (in both directions), and the second picture is downsampled by half (in both directions).

[0043]

[0041] Consistent with this disclosure, the disclosed picture-level resampling method can be used with other coding tools. For example, in some exemplary embodiments, scaled motion vectors based on resampled reference pictures can be used in the temporal motion vector predictor (TMVP) or the advanced temporal motion vector predictor (ATMVP).

[0044]

[0042] Next, a test methodology for evaluating the encoding tool for ARC is described. Since the adaptive resolution change is mainly used to adapt to the network bandwidth, the following test conditions can be considered to compare the encoding efficiency of different ARC schemes during the network bandwidth change.

[0045]

[0043] In some embodiments, the resolution is changed to half (both vertically and horizontally) at a particular time instance, and then restored to the original resolution after a period of time. Figure 10 shows an example of resolution change. At time t1, the resolution is changed to half, and then restored to full resolution at time t2.

[0046] In some embodiments, when the resolution is reduced, if the downsampling ratio is too large (e.g., the downsampling ratio for a given dimension is greater than 2:1), then progressive downsampling is used over a period of time. However, when upsampling is used, progressive upsampling is not used.

[0047] In some embodiments, a Scalable HEVC Test Model (SMH) downsampling filter may be used for source resampling. In Table 4 (FIG. 11), the detailed filter coefficients are shown with different fractional sample positions.

[0048]

[0046] In some exemplary embodiments, two peak signal-to-noise ratios (PSNRs) are calculated to measure video quality. A diagram of PSNR calculation is shown in Figure 12. The first PSNR is calculated between the resampled source and the decoded picture. The second PSNR is calculated between the original source and the upsampled decoded source.

[0049] In some embodiments, the number of pictures encoded is the same as the current test condition for VVC.

[0050]

[0048] Next, the disclosed block-based resampling method is described. In block-based resampling, the above-mentioned information loss can be reduced by integrating resampling and motion compensation interpolation into one filtering operation. Take the following case as an example: the motion vector of the current block has half-pel precision in one dimension (e.g., in the horizontal dimension), and the width of the reference picture is twice the width of the current picture. In this case, compared with the picture-level resampling, which reduces the width of the reference picture by half to match the width of the current picture, and then performs half-pel motion interpolation, the block-based resampling method directly fetches the odd positions of the reference picture as reference blocks with half-pel precision. In the 15th JVET meeting, a block-based resampling method for ARC, in which motion compensation (MC) interpolation and reference resampling are integrated and performed with a single-step filter, was adopted for VVC. In VVC draft 6, the existing filter for MC interpolation without reference resampling is reused for MC interpolation with reference resampling. The same filter is used for both reference upsampling and downsampling. The details of filter selection are explained below.

[0051] For the luma components, if half-pel AMVR mode is selected and the interpolation position is half-pel, a 6-tap filter [3,9,20,20,9,3] is used, and if the motion compensation block size is 4×4, the following 6-tap filter as shown in Table 5 (FIG. 13) is used, otherwise an 8-tap filter as shown in Table 6 (FIG. 14) is used. For the chroma components, a 4-tap filter as shown in Table 7 (FIG. 15) is used.

[0052]

[0050] In VVC, the same filter is used for MC interpolation without reference resampling and MC interpolation with reference resampling. Although the MCIF of VVC is designed based on DCT upsampling, it may not be appropriate to use it as a single-step filter that integrates reference downsampling and MC interpolation. For example, in the case of phase 0 filtering (i.e., scaled mv is an integer), the VVC 8-tap MCIF coefficient is [0,0,0,64,0,0,0,0], which means that the predicted sample is directly copied from the reference sample. This may not be a problem for MC interpolation without reference downsampling or with reference upsampling, but for the reference downsampling case, it may cause aliasing artifacts due to the lack of a low-pass filter before decimation.

[0053]

[0051] This disclosure provides a method for using a cosine windowed sinc filter for MC interpolation with reference downsampling.

[0054]

[0052] A windowed sinc filter is a bandpass filter that separates one band of frequencies from another. A windowed sinc filter is a lowpass filter with a frequency response that passes all frequencies below the cutoff frequency with an amplitude of 1 and cuts all frequencies above the cutoff frequency with an amplitude of zero, as shown in Figure 16.

[0055]

[0053] The filter kernel, also known as the impulse response of the filter, is obtained by taking the inverse Fourier transform of the frequency response of an ideal low-pass filter: The impulse response of a low-pass filter is the general form of a sinc function:

number

number

[0056]

[0054] The sinc function is infinite. To make the filter kernel of finite length, a window function is applied to truncate the filter kernel to L points. To obtain a smooth tapered curve, a cosine windowing function is used, which is given by:

number

[0057]

[0055] The kernel of the cosine windowed sinc filter is the product of the ideal response function h(n) and a cosine window function w(n).

number

[0058]

[0056] Two parameters chosen for the windowed sink kernel are the cutoff frequency fc and the kernel length L. For the downsampling filter used in the Scalable HEVC Test Model (SHM), fc=0.9 and L=13.

[0059]

[0057] The filter coefficients obtained in (4) are real numbers. Applying the filter is equivalent to calculating the weighted average of the reference samples, with the weights being the filter coefficients. For efficient calculation in digital computers or hardware, the coefficients are normalized, multiplied by a scaler, and rounded to integers so that the sum of the coefficients is equal to 2 to the power of N (N is an integer). The filtered sample is divided by 2 to the power of N (equivalent to a right shift of N bits). For example, in VVC Draft 6, the sum of the interpolation filter coefficients is 64.

[0060] In some disclosed embodiments, a downsampling filter is used in SHM for VVC motion compensation interpolation using reference downsampling for both luma and chroma components, and an existing MCIF is used for motion compensation interpolation using reference upsampling. If the kernel length L=13 and the first coefficient is small and rounded to zero, the filter length can be reduced to 12 without affecting the filter performance.

[0061] As an example, the filter coefficients for 2:1 downsampling and 1.5:1 downsampling are shown in Table 8 (FIG. 17) and Table 9 (FIG. 18), respectively.

[0062] Besides the coefficient values, there are several other differences between the SHM filter design and existing MCIF designs.

[0063]

[0061] First, the SHM filter requires filtering at integer and fractional sample positions, while the MCIF only requires filtering at fractional sample positions. An example of the modifications to the luma sample interpolation filtering process for the reference downsampling case of VVC Draft 6 is listed in Table 10 (Figure 19).

[0064]

[0062] An example of modifications to the chroma sample interpolation filtering process for the reference downsampling case of VVC Draft 6 is listed in Table 11 (Figure 20).

[0065]

[0063] As a second difference, for the SHM filter, the total number of filter coefficients is 128, while for the existing MCIF, the total number of filter coefficients is 64. In VVC draft 6, the intermediate prediction signal is maintained at a higher precision (represented by a larger bit depth) than the output signal to reduce losses caused by rounding errors. The precision of the intermediate signal is called the internal precision. In one embodiment, to maintain the same internal precision as VVC draft 6, the output of the SHM filter needs to be further right-shifted by one bit compared to the case when the existing MCIF is used. An example of the change to the luma sample interpolation filtering process of the reference downsampling case of VVC draft 6 is shown in Table 12 (Figure 21).

[0066]

[0064] An example of modifications to the chroma sample interpolation filtering process for the reference downsampling case of VVC Draft 6 is shown in Table 13 (Figure 22).

[0067] According to some embodiments, the internal precision can be increased by 1 bit and an additional 1-bit right shift can be used to convert the internal precision to the output precision. An example of the modifications to the luma sample interpolation filtering process for the reference downsampling case of VVC Draft 6 is shown in Table 14 (Figure 23).

[0068]

[0066] An example of modifications to the chroma sample interpolation filtering process for the reference downsampling case of VVC Draft 6 is shown in Table 15 (Figure 24).

[0069]

[0067] As a third difference, the SHM filter has 12 taps. Therefore, 11 adjacent samples (5 left, 6 right, or 5 up, 6 down) are needed to generate an interpolated sample. Compared to MCIF, additional adjacent samples are fetched. In VVC draft 6, the chroma mv precision is 1 / 32. However, the SHM filter has only 16 phases. Therefore, the chroma mv can be rounded to 1 / 16 with respect to the reference downsampling. This can be done by right shifting the last 5 bits of the chroma mv by 1 bit. An example of the change to the chroma fractional sample position calculation for the reference downsampling case in VVC draft 6 is shown in Table 16 (Figure 25).

[0070] In some embodiments, to match the existing MCIF design of the VVC draft, we propose to use an 8-tap cosine windowed sinc filter. The filter coefficients can be derived by setting L=9 in the cosine windowed sinc function of equation (4). To further match the existing MCIF filter, the sum of the filter coefficients can be set to 64. Exemplary filter coefficients for ratios of 2:1 and 1.5:1 are shown in Table 17 (FIG. 26) and Table 18 (FIG. 27), respectively.

[0071] According to some embodiments, a 32-phase cosine windowed sinc filter set may be used in chroma motion compensation interpolation with reference downsampling to accommodate 1 / 32 sample precision for chroma components. Example filter coefficients for ratios of 2:1 and 1.5:1 are shown in Table 19 (FIG. 28) and Table 20 (FIG. 29), respectively.

[0072] According to some embodiments, for a 4x4 luma block, a 6-tap cosine windowed sinc filter may be used for MC interpolation with reference downsampling. Example filter coefficients for ratios of 2:1 and 1.5:1 are shown below in Table 21 (Figure 30) and Table 22 (Figure 31), respectively.

[0073]

[0071] In some embodiments, a non-transitory computer-readable storage medium is also provided that includes instructions, which can be executed by a device (such as the disclosed encoder and decoder) to perform the above method. Common forms of non-transitory media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tapes or other magnetic data storage media, CD-ROMs, other optical data storage media, any physical media with a pattern of holes, RAM, PROMs, and EPROMs, FLASH-EPROMs or other flash memories, NVRAMs, caches, registers, other memory chips or cartridges, and networked versions of the above. A device may include one or more processors (CPUs), input / output interfaces, network interfaces, and / or memory.

[0074]

[0072] It should be noted that relative terms herein, such as "first" and "second," are used only to distinguish one entity or operation from another, and do not require or imply an actual relationship or order between those entities or operations. Also, the words "comprising," "having," "containing," and "including," and other similar forms, are intended to be equivalent in meaning and to be open-ended in that the term or terms following any one of these terms is not an exhaustive enumeration of such term or terms, or limited to only the enumerated term or terms.

[0075]

[0073] In this specification, unless otherwise specified, the term "or" covers all possible combinations unless not feasible. For example, if it is stated that a database may include A or B, the database may include A, or B, or A and B, unless otherwise specified or feasible. As a second example, if it is stated that a database may include A, B, or C, the database may include A, or B, or C, or A and B, or A and C, or B and C, or A and B and C, unless otherwise specified or feasible.

[0076]

[0074] It is understood that the above embodiments can be implemented by hardware, or software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above computer-readable medium. The software, when executed by a processor, can perform the disclosed method. The computing unit and other functional units described in this disclosure can be implemented by hardware, or software, or a combination of hardware and software. Those skilled in the art will also understand that more than one of the above modules / units can be integrated into one module / unit, and each of the above modules / units can be further divided into multiple sub-modules / sub-units.

[0077]

[0075] In the above specification, the embodiments have been described with respect to numerous specific details that may vary from implementation to implementation. Certain adaptations and modifications of the described embodiments may be made. Other embodiments may become apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the above specification and examples be considered as merely exemplary, with the true scope and spirit of the invention being indicated by the following claims. Also, the order of steps shown in the drawings is intended to be merely illustrative, and is not intended to be limited to any particular order of steps. Thus, one skilled in the art will appreciate that these steps may be performed in different orders while performing the same method.

[0078]

[0076] The embodiments can be further described using the following clauses. 1. A computer-implemented image processing method, comprising: Comparing resolutions of the target picture and the first reference picture; in response to the target picture and the first reference picture having different resolutions, resampling the first reference picture to generate a second reference picture; encoding or decoding the target picture using the second reference picture; The method includes: 2. storing a second reference picture in a first buffer, the first buffer being different from a second buffer that stores a decoded picture used for predicting a future picture; removing the second reference picture from the first buffer after the encoding or decoding of the target picture is completed; 2. The method of claim 1, further comprising: 3. Encoding or decoding the target picture; 3. The method of any one of clauses 1 and 2, comprising encoding or decoding the target picture by using no more than a predetermined number of resampled reference pictures. 4. Encoding or decoding the target picture 4. The method of claim 3, comprising signalling the predefined number as part of a sequence parameter set or a picture parameter set. 5. Resampling the first reference picture to generate the second reference picture, storing a first version and a second version of a first reference picture, the first version having an original resolution and the second version having a maximum resolution to which the reference picture can be resampled; resampling the maximum version to generate a second reference picture; 2. The method according to claim 1, comprising: 6. Storing the first version and the second version of the first reference picture in a decoded picture buffer; outputting the first version for encoding future pictures; 6. The method of claim 5, further comprising: 7. Resampling the first reference picture to generate the second reference picture, 2. The method of claim 1, comprising generating a second reference picture at a supporting resolution. 8. Number of supported resolutions and The pixel dimensions corresponding to the supported resolutions, 8. The method of claim 7, further comprising signaling information indicative of 9. Information, 9. The method of claim 8, including an index corresponding to at least one of the supported resolutions. 10. The method of any one of clauses 8 and 9, further comprising setting the number of supported resolutions based on a configuration of a video application or a video device. 11. Resampling the first reference picture to generate a second reference picture; 11. The method according to any one of clauses 1 to 10, comprising progressive downsampling of the first reference picture to generate the second reference picture. 12. A device comprising: one or more memories for storing computer instructions; Execute computer instructions to cause the device to: Comparing a resolution of the target picture with a resolution of the first reference picture; in response to the target picture and the first reference picture having different resolutions, resampling the first reference picture to generate a second reference picture; encoding or decoding the target picture using the second reference picture; one or more processors configured to cause Including, the device. 13. A non-transitory computer-readable medium storing a set of instructions, the set of instructions being executable by at least one processor of a computer system to cause the computer system to perform a method for processing video content, the method comprising: Comparing resolutions of the target picture and the first reference picture; in response to the target picture and the first reference picture having different resolutions, resampling the first reference picture to generate a second reference picture; encoding or decoding the target picture using the second reference picture; A non-transitory computer readable medium comprising: 14. In response to the target picture and the reference picture having different resolutions, applying a band pass filter to the reference picture to perform motion compensated interpolation and to generate the reference block; encoding or decoding a block of the target picture using the reference block; 23. A computer-implemented image processing method comprising: 15. The method of claim 14, wherein the bandpass filter is a cosine windowed sinc filter. 16. Cosine windowed sinc filter with kernel function

number

number

[0079]

[0077] In the drawings and specification, exemplary embodiments have been disclosed. However, many variations and modifications can be made to these embodiments. Thus, although specific terms are employed, they are used in a generic and descriptive sense only and are not intended to be limiting.

Claims

1. 1. A computer-implemented image processing method, comprising: comparing resolutions of the target picture and the first reference picture; in response to the target picture and the first reference picture having different resolutions, resampling the first reference picture to generate a second reference picture; encoding or decoding the target picture using the second reference picture; The method includes:

2. storing the second reference picture in a first buffer, the first buffer being different from a second buffer that stores decoded pictures used for predicting future pictures; removing the second reference picture from the first buffer after the encoding or decoding of the target picture is completed; The method of claim 1 further comprising:

3. encoding or decoding the target picture, The method of claim 1 , comprising encoding or decoding the target picture by using no more than a predetermined number of resampled reference pictures.

4. encoding or decoding the target picture, The method of claim 3 , comprising signaling the predetermined number as part of a sequence parameter set or a picture parameter set.

5. resampling the first reference picture to generate the second reference picture; storing a first version and a second version of the first reference picture, the first version having an original resolution and the second version having a maximum resolution to which the reference picture can be resampled; resampling the maximum version to generate the second reference picture; The method of claim 1 , comprising:

6. storing the first version and the second version of the first reference picture in a decoded picture buffer; outputting the first version for encoding future pictures; and The method of claim 5 further comprising:

7. resampling the first reference picture to generate the second reference picture; The method of claim 1 , comprising generating the second reference picture at a supporting resolution.

8. Number of supported resolutions, pixel dimensions corresponding to said supported resolutions; The method of claim 7 , further comprising signaling information indicative of:

9. The information is The method of claim 8 including an index corresponding to at least one of the supported resolutions.

10. The method of claim 8 , further comprising setting the number of supported resolutions based on a configuration of a video application or a video device.

11. resampling the first reference picture to generate the second reference picture; The method of claim 1 , comprising progressively downsampling the first reference picture to generate the second reference picture.

12. A device, comprising: one or more memories for storing computer instructions; Executing the computer instructions to cause the device to: comparing a resolution of the target picture with a resolution of a first reference picture; in response to the target picture and the first reference picture having different resolutions, resampling the first reference picture to generate a second reference picture; encoding or decoding the target picture using the second reference picture; one or more processors configured to cause Including, the device.

13. The one or more processors execute the computer instructions to cause the device to: storing the second reference picture in a first buffer, the first buffer being different from a second buffer that stores decoded pictures used for predicting future pictures; removing the second reference picture from the first buffer after the encoding or decoding of the target picture is completed; The device of claim 12 , further configured to:

14. The one or more processors execute the computer instructions to cause the device to: The device of claim 12 , further configured to cause encoding or decoding of the target picture by using no more than a predetermined number of resampled reference pictures.

15. The one or more processors execute the computer instructions to cause the device to: The device of claim 14 , further configured to cause signaling of the predetermined number as part of a sequence parameter set or a picture parameter set.

16. The one or more processors execute the computer instructions to cause the device to: storing a first version and a second version of the first reference picture in the one or more memories, the first version having an original resolution and the second version having a maximum resolution to which the reference picture can be resampled; resampling the maximum version to generate the second reference picture; The device of claim 12 , further configured to:

17. The one or more processors execute the computer instructions to cause the device to: storing the first version and the second version of the first reference picture in a decoded picture buffer; outputting the first version for encoding future pictures; and The device of claim 16 , further configured to:

18. The one or more processors execute the computer instructions to cause the device to: The device of claim 12 , further configured to generate the second reference picture at a supporting resolution.

19. The one or more processors execute the computer instructions to cause the device to: Number of supported resolutions, pixel dimensions corresponding to said supported resolutions; The device of claim 18 , further configured to signal information indicative of:

20. 1. A non-transitory computer-readable medium storing a set of instructions, the set of instructions being executable by at least one processor of a computer system to cause the computer system to perform a method for processing video content, the method comprising: comparing resolutions of the target picture and the first reference picture; in response to the target picture and the first reference picture having different resolutions, resampling the first reference picture to generate a second reference picture; encoding or decoding the target picture using the second reference picture; A non-transitory computer readable medium comprising:

Images