Adaptive resolution change in video processing

Abstract

The present disclosure provides systems and methods for performing adaptive resolution changes during video encoding and decoding. These methods include: comparing the resolution of a target image and a first reference image; in response to the target image having a different resolution than the first reference image, resampling the first reference image to generate a second reference image; and encoding or decoding the target image using the second reference image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to U.S. Provisional Patent Application No. 62 / 865,927, filed June 24, 2019, and U.S. Provisional Patent Application No. 62 / 900,439, filed September 13, 2019, both of which are incorporated herein by reference in their entireties. Technical Field

[0003] The present disclosure relates generally to video processing and, more particularly, to methods and systems for performing adaptive resolution changes in video encoding. Background Art

[0004] A video is a set of static images (or "frames") that capture visual information. To reduce storage memory and transmission bandwidth, videos can be compressed before storage or transmission and decompressed before display. The compression process is usually called encoding, and the decompression process is usually called decoding. There are many video coding formats that use standardized video coding techniques, the most common of which are based on prediction, transform, quantization, entropy coding, and loop filtering. Video codec standards, such as the High Efficiency Video Codec (HEVC / H.265) standard and the Versatile Video Codec (VVC / H.266) standard (AVS), specify specific video codec formats and are developed by standardization organizations. As more and more video standards adopt advanced video codec technologies, the coding and decoding efficiency of new video codec standards is also increasing. Summary of the Invention

[0005] Embodiments of the present invention provide a method for adaptively changing resolution during video encoding and decoding. In one exemplary embodiment, the method includes: comparing the resolutions of a target image and a first reference image; in response to the target image and the first reference image having different resolutions, resampling the first reference image to generate a second reference image; and encoding or decoding the target image using the second reference image.

[0006] An embodiment of the present invention also provides an apparatus for adaptive resolution change during video encoding and decoding. In one exemplary embodiment, the apparatus includes: one or more memories storing computer instructions; one or more processors configured to execute the computer instructions to cause the apparatus to: compare the resolution of a target image and a first reference image; in response to the target image having a different resolution than the first reference image, resample the first reference image to generate a second reference image; and encode or decode the target image using the second reference image.

[0007] Embodiments of the present disclosure also provide a non-transitory computer-readable medium storing a set of instructions, the set of instructions being executable by at least one processor of a computer system to cause the computer system to perform a method for adaptive resolution change. In one exemplary embodiment, the method includes: comparing the resolutions of a target image and a first reference image; in response to the target image having a different resolution than the first reference image, resampling the first reference image to generate a second reference image; and encoding or decoding the target image using the second reference image. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Embodiments and aspects of the present disclosure are illustrated in the following detailed description and accompanying drawings.The various features shown in the drawings are not drawn to scale.

[0009] Figure 1 is a schematic diagram illustrating an exemplary video encoder consistent with an embodiment of the present disclosure.

[0010] Figure 2 is a schematic diagram illustrating an exemplary video decoder consistent with an embodiment of the present invention.

[0011] Figure 3 An example is shown in which the resolution of a reference image is different from that of a current image, consistent with an embodiment of the present disclosure.

[0012] Figure 4 is a table illustrating sub-pixel motion compensation interpolation filters for luma components in Versatile Video Coding (VVC) consistent with an embodiment of the present disclosure.

[0013] Figure 5 This is a table of sub-pixel motion compensation interpolation filters for chroma components in VVC consistent with an embodiment of the present invention.

[0014] Figure 6 An exemplary reference picture buffer consistent with embodiments of the present disclosure is shown.

[0015] Figure 7 is an example table illustrating a supported resolution set including three different resolutions consistent with an embodiment of the present disclosure.

[0016] Figure 8 An exemplary decoded picture buffer (DPB) is shown when both resampled reference pictures and original reference pictures are stored, consistent with embodiments of the present disclosure.

[0017] Figure 9 Progressive downsampling consistent with embodiments of the present disclosure is shown.

[0018] Figure 10An exemplary video encoding process with resolution change consistent with embodiments of the present disclosure is shown.

[0019] Figure 11 is a table illustrating exemplary downsampling filters consistent with embodiments of the present disclosure.

[0020] Figure 12 An exemplary video encoding process with peak signal-to-noise ratio (PSNR) calculation consistent with embodiments of the present disclosure is shown.

[0021] Figure 13 The frequency response of an exemplary low-pass filter consistent with embodiments of the present disclosure is shown.

[0022] Figure 14 is a table illustrating a 6-tap filter consistent with an embodiment of the present disclosure.

[0023] Figure 15 is a table illustrating an 8-tap filter consistent with an embodiment of the present disclosure.

[0024] Figure 16 is a table illustrating a 4-tap filter consistent with an embodiment of the present disclosure.

[0025] Figure 17 is a table showing interpolation filter coefficients for 2:1 downsampling consistent with an embodiment of the present disclosure.

[0026] Figure 18 is a table showing interpolation filter coefficients for 1.5:1 downsampling consistent with an embodiment of the present disclosure.

[0027] Figure 19 An exemplary luma sample interpolation filtering process for reference downsampling consistent with embodiments of the present disclosure is shown.

[0028] Figure 20 An exemplary chroma sample interpolation filtering process for reference downsampling consistent with embodiments of the present disclosure is shown.

[0029] Figure 21 An exemplary luma sample interpolation filtering process for reference downsampling consistent with embodiments of the present disclosure is shown.

[0030] Figure 22 An exemplary chroma sample interpolation filtering process for reference downsampling consistent with embodiments of the present disclosure is shown.

[0031] Figure 23 An exemplary luma sample interpolation filtering process for reference downsampling consistent with embodiments of the present disclosure is shown.

[0032] Figure 24An exemplary chroma sample interpolation filtering process for reference downsampling consistent with embodiments of the present disclosure is shown.

[0033] Figure 25 Exemplary chroma fractional sample position calculations for reference downsampling consistent with embodiments of the present disclosure are shown.

[0034] Figure 26 is a table showing an 8-tap filter for MC interpolation consistent with an embodiment of the present disclosure, where the reference downsampling ratio is 2:1.

[0035] Figure 27 is a table showing an 8-tap filter for MC interpolation consistent with an embodiment of the present disclosure, where the reference downsampling ratio is 1.5:1.

[0036] Figure 28 is a table showing an 8-tap filter for MC interpolation consistent with an embodiment of the present disclosure, where the ratio of reference downsampling is 2:1.

[0037] Figure 29 is a table showing an 8-tap filter for MC interpolation consistent with an embodiment of the present disclosure, where the reference downsampling ratio is 1.5:1.

[0038] Figure 30 is a table showing 6-tap filter coefficients for luma 4x4 block MC interpolation with reference downsampling at a ratio of 2:1, consistent with an embodiment of the present disclosure.

[0039] Figure 31 is a table showing 6-tap filter coefficients for luma 4x4 block MC interpolation with reference downsampling at a ratio of 1.5:1, consistent with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, and unless otherwise specified, the same numbers in different drawings represent the same or similar elements. The embodiments set forth in the following description of exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with the aspects related to the present invention as described in the appended claims. Specific aspects of the present disclosure are described in more detail below. In the event of a conflict with terms and / or definitions incorporated by reference, the terms and definitions provided herein shall prevail.

[0041] A video is a set of static images (or "frames") arranged in a temporal sequence to store visual information. Video capture devices (such as cameras) can be used to capture and store these images in chronological order, and video playback devices (such as televisions, computers, smartphones, tablets, video players, or any end-user device with a display) can be used to display these images in chronological order. Furthermore, in some applications, video capture devices can transmit captured video in real time to video playback devices (such as computers with monitors) for use in monitoring, conferencing, and live broadcasts.

[0042] To reduce the storage space and transmission bandwidth required for such applications, video can be compressed. For example, video can be compressed before storage and transmission and decompressed before display. Compression and decompression can be implemented by software executed by a processor (e.g., a general-purpose computer processor) or dedicated hardware. The compression module is generally referred to as an "encoder," and the decompression module is generally referred to as a "decoder." Encoders and decoders can be collectively referred to as "codecs." Encoders and decoders can be implemented in any of a variety of suitable hardware, software, or combinations thereof. For example, hardware implementations of encoders and decoders can include circuits such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, or any combination thereof. Software implementations of encoders and decoders can include program code, computer-executable instructions, firmware, or any suitable computer-implemented algorithm or process fixed in a computer-readable medium. Video compression and decompression can be implemented using various algorithms or standards, such as MPEG-1, MPEG-2, MPEG-4, the H.26x series, and others. In some applications, a codec can decompress video using one coding standard and recompress the decompressed video using a second coding standard. In such cases, the codec can be referred to as a "transcoder."

[0043] The video encoding process identifies and preserves useful information that can be used to reconstruct the image. If the information ignored during the video encoding process cannot be fully reconstructed, the encoding process can be called "lossy." Otherwise, it can be called "lossless." Most encoding processes are lossy as a trade-off to reduce required storage space and transmission bandwidth.

[0044] In many cases, useful information about the image being coded (referred to as the "current image") can include changes relative to a reference image (e.g., a previously coded or reconstructed image). Such changes can include changes in pixel position, brightness, or color, with position changes being of primary interest. A change in the position of a group of pixels representing an object can reflect the object's motion between the reference image and the current image.

[0045] To achieve the same subjective quality as HEVC / H.265 using half the bandwidth, JVET has been developing technologies beyond HEVC using the Joint Exploration Model ("JEM") reference software. By incorporating coding technologies into JEM, JEM has achieved higher coding performance than HEVC. VCEG and MPEG have also officially begun developing the Versatile Video Coding (VVC / H.266) standard, the next-generation video compression standard that will surpass HEVC.

[0046] The VVC standard continues to include more coding techniques that provide better compression performance. VVC can be implemented in the same video coding system used in modern video compression standards (such as HEVC, H.264 / AVC, MPEG2, H.263, etc.). Figure 1 is a schematic diagram illustrating an exemplary video encoder 100 consistent with the disclosed embodiments. For example, the video encoder 100 may perform intra-frame or inter-frame coding of blocks within a video frame, including video blocks, or partitions or subpartitions of video blocks. Intra-frame coding may rely on spatial prediction to reduce or remove spatial redundancy in video within a given video frame. Inter-frame coding may rely on temporal prediction to reduce or remove temporal redundancy in video in adjacent frames of a video sequence. Intra-frame modes may refer to multiple spatially-based compression modes, while inter-frame modes (e.g., uni-prediction or bi-prediction) may refer to multiple temporally-based compression modes.

[0047] Please refer to Figure 1 , the input video signal 102 may be processed block by block. For example, a video block unit may be a 16×16 pixel block (e.g., a macroblock (MB)). In HEVC, extended block sizes (e.g., coding units (CUs)) may be used to compress video signals of resolutions such as 1080p and higher. In HEVC, a CU may include up to 64×64 luminance samples and corresponding chrominance samples. In VVC, the size of the CU may be further increased to include 128×128 luminance samples and corresponding chrominance samples. The CU may be divided into prediction units (PUs), to which separate prediction methods may be applied. Each input video block (e.g., MB, CU, PU, ​​etc.) may be processed by using a spatial prediction unit 160 or a temporal prediction unit 162.

[0048] The spatial prediction unit 160 performs spatial prediction (e.g., intra-frame prediction) on the current CU using information about the same picture / slice that contains the current CU. Spatial prediction can use pixels from already coded neighboring blocks in the same video picture / slice to predict the current video block. Spatial prediction can reduce spatial redundancy inherent in video signals. Temporal prediction (e.g., inter-frame prediction or motion-compensated prediction) can use samples from already coded video pictures to predict the current video block. Temporal prediction can reduce temporal redundancy inherent in video signals.

[0049] Temporal prediction unit 162 performs temporal prediction (e.g., inter-frame prediction) on the current CU using information from a different picture / slice than the one containing the current CU. Temporal prediction for a video block may be signaled by one or more motion vectors. These motion vectors may indicate the amount and direction of motion between the current block and one or more prediction blocks in a reference frame. If multiple reference pictures are supported, one or more reference picture indexes may be sent for the video block. These one or more reference indexes may be used to identify which reference picture(s) in the decoded picture buffer (DPB) 164 (also referred to as reference picture storage 164) the temporal prediction signal may come from. After spatial or temporal prediction, mode decision and encoder control unit 180 in the encoder may select a prediction mode, for example, based on a rate-distortion optimization approach. The prediction block may be subtracted from the current video block at adder 116. The prediction residual may be transformed by transform unit 104 and quantized by quantization unit 106. The quantized residual coefficients may be inverse quantized by inverse quantization unit 110 and inverse transformed by inverse transform unit 112 to form a reconstructed residual. The reconstructed block may be added to the predicted block at adder 126 to form a reconstructed video block. Loop filtering, such as a deblocking filter and an adaptive loop filter 166, may be applied to the reconstructed video block before it is placed in reference picture memory 164 and used to encode future video blocks. To form the output video bitstream 120, the coding mode (e.g., inter or intra), prediction mode information, motion information, and quantized residual coefficients may be sent to entropy coding unit 108 to be compressed and packed to form the bitstream 120.

[0050] Consistent with the disclosed embodiments, the above-mentioned units of the video encoder 100 can be implemented as software modules (e.g., computer programs that implement different functions), hardware components (e.g., different circuit blocks for performing corresponding functions), or a mixture of software and hardware.

[0051] Figure 2 is a schematic diagram illustrating an exemplary video decoder 200 consistent with the disclosed embodiments. Figure 2 , the video bitstream 202 may be unpacked or entropy decoded at the entropy decoding unit 208. The coding mode or prediction information may be sent to the spatial prediction unit 260 (e.g., if intra-coded) or the temporal prediction unit 262 (e.g., if inter-coded) to form a prediction block. If inter-coded, the prediction information may include the prediction block size, one or more motion vectors (e.g., which may indicate the direction and amount of motion), or one or more reference indexes (e.g., which may indicate which reference image the prediction signal is to be obtained from).

[0052] Motion-compensated prediction may be performed by temporal prediction unit 262 to form a temporal prediction block. The residual transform coefficients may be sent to inverse quantization unit 210 and inverse transform unit 212 to reconstruct the residual block. The prediction block and the residual block may be added together at 226. The reconstructed block may be loop-filtered (via loop filter 266) before being stored in decoded picture buffer (DPB) 264 (also referred to as reference picture storage 264). The reconstructed video in DPB 264 may be used to drive a display device or to predict future video blocks. The decoded video 220 may be displayed on a display.

[0053] Consistent with the disclosed embodiments, the above-mentioned units of the video decoder 200 can be implemented as software modules (e.g., computer programs that implement different functions), hardware components (e.g., different circuit blocks for performing corresponding functions), or a mixture of software and hardware.

[0054] One of the goals of the VVC standard is to provide video conferencing applications with the ability to tolerate network and device diversity. In particular, the VVC standard needs to provide the ability to quickly adapt to changes in the network environment, including quickly reducing the encoding bit rate when network conditions deteriorate and quickly improving video quality when network conditions improve. In addition, for adaptive streaming services that provide multiple representations of the same content, each of the multiple representations may have different properties (such as spatial resolution or sample bit depth), and the video quality may vary from low to high. Therefore, the VVC standard needs to support fast representation switching for adaptive streaming services. During switching from one representation to another (for example, from one resolution to another), the VVC standard needs to enable an efficient prediction structure without affecting the ability to switch quickly and seamlessly.

[0055] In some embodiments consistent with the present disclosure, to change the resolution, an encoder (e.g., Figure 1 The encoder 100 in FIG. 100 ) sends an instantaneous decoder refresh (IDR) coded picture to clear the contents of the reference picture buffer (e.g., Figure 1 DPB164 and Figure 2 Upon receiving an IDR-coded picture, the decoder (e.g., Figure 2 The decoder 200 in FIG. 2 marks all pictures in the reference buffer as "unused for reference". All subsequently transmitted pictures can be decoded without reference to any frames decoded before the IDR picture. The first picture in an encoded video sequence is always an IDR picture.

[0056] In some embodiments consistent with the present disclosure, an Adaptive Resolution Change (ARC) technique may be used to allow a video stream to change spatial resolution between coded pictures within the same video sequence without requiring new IDR pictures and without requiring multiple layers as in scalable video codecs. According to the ARC technique, at a resolution switching point, the currently coded picture is either predicted from a reference picture of the same resolution (if available) or from a reference picture of a different resolution by resampling the reference picture. Figure 3 Adaptive resolution change is shown in Figure 1, where the resolution of the reference picture "Ref 0" is the same as the resolution of the current coded picture. However, the resolutions of the reference pictures "Ref 1" and "Ref 2" are different from the resolution of the current picture. To generate the motion compensated prediction signal for the current picture, both "Ref 1" and "Ref 2" are resampled to the resolution of the current picture.

[0057] Consistent with the present disclosure, when the resolution of the reference frame is different from the resolution of the current frame, one way to generate a motion compensated prediction signal is based on image resampling, where the reference image is first resampled to the same resolution as the current image and an existing motion compensation process with motion vectors can be applied. The motion vectors can be scaled (if they are sent in units before the resampling is applied) or not (if they are sent in units after the resampling is applied). For image-based resampling, especially for downsampling of the reference image (i.e., the resolution of the reference image is greater than the resolution of the current image), information may be lost in the reference resampling step before motion compensated interpolation, because downsampling is typically achieved by low-pass filtering and decimation).

[0058] Another approach is block-based resampling, where resampling is performed at the block level. This is done by checking one or more reference images used by the current block, and if one or both of them have a different resolution than the current image, resampling is performed in conjunction with a sub-pixel motion compensated interpolation process.

[0059] The present disclosure provides an image-based resampling method and a block-based resampling method for use with ARC.The following description first addresses the disclosed image-based resampling method and then addresses the disclosed block-based resampling method.

[0060] The disclosed image-based resampling method can solve some problems caused by traditional block-based resampling methods. First, in traditional block-level resampling methods, when it is determined that the resolution of the reference image is different from the resolution of the current image, immediate block-level resampling can be performed. However, the block-level resampling process may complicate the encoder design because the encoder may have to resample the block at each search point during motion search. Motion search is generally a time-consuming process for the encoder, so the immediate requirements during motion search may complicate the motion search process. As an alternative to immediate block-level resampling, the encoder can resample the reference image in advance so that the resampled reference image has the same resolution as the current image. However, this may reduce coding efficiency because it may result in different prediction signals during motion estimation and motion compensation.

[0061] Secondly, in traditional block-based resampling methods, the design of block-level resampling is incompatible with some other useful coding tools, such as sub-block-based temporal motion vector prediction (SbTMVP), affine motion compensation prediction, and decoder-side motion vector refinement (DMVR). When ARC is enabled, these coding tools may be disabled. However, this disabling will significantly degrade encoding performance.

[0062] Third, while any sub-pixel motion compensation interpolation filter known in the art can be used for block resampling, it may not be equally applicable to block upsampling and block downsampling. For example, in the case where the resolution of the reference image is lower than the current image, the interpolation filter used for motion compensation can be used for upsampling. However, in the case where the resolution of the reference image is higher than the current image, the interpolation filter is not suitable for downsampling because the interpolation filter cannot filter integer positions, which may cause aliasing. For example, Table 1 ( Figure 4 ) shows exemplary interpolation filter coefficients for integer positions of the luma component with various fractional sample position values. Table 2( Figure 5 ) shows exemplary interpolation filter coefficients for chroma components with various fractional sample position values. As shown in Table 1 ( Figure 4 ) and Table 2( Figure 5 ), at the fractional sample position 0 (i.e., the integer position), no interpolation filter is applied.

[0063] To avoid the above-mentioned problems associated with traditional block-based resampling methods, the present disclosure provides an image-level resampling method that can be used for ARC. The image resampling process can involve upsampling or downsampling. Upsampling is to increase the spatial resolution while maintaining the two-dimensional (2D) representation of the image. In the upsampling process, the resolution of the reference image is increased by inserting unavailable samples from adjacent available samples. In the downsampling process, the resolution of the reference image is reduced.

[0064] According to the disclosed embodiments, resampling can be performed at the picture level. In picture-level resampling, if the resolution of a reference picture differs from the resolution of the current picture, the reference picture is resampled to the resolution of the current picture. Motion estimation and / or compensation for the current picture can be performed based on the resampled reference picture. In this way, ARC can be implemented in a "transparent" manner at the encoder and decoder because block-level operations are independent of resolution changes.

[0065] Picture-level resampling may be performed on the fly, i.e., while the current picture is being predicted. In some exemplary embodiments, only the original (non-resampled) reference pictures are stored in the decoded picture buffer (DPB), e.g., encoder 100 ( Figure 1 ) in the DPB 164 and the decoder 200 ( Figure 2 ) in DPB264. The DPB is managed in the same manner as the current version of VVC. In the disclosed picture-level resampling method, before performing encoding or decoding of an image, if the resolution of the reference image in the DPB is different from the resolution of the current image, the encoder or decoder resamples the reference image. In some embodiments, a resampled picture buffer can be used to store all resampled reference images of the current image. The resampled images of the reference images are stored in the resampled picture buffer, and motion search / compensation for the current image is performed using images from the resampled picture buffer. After encoding or decoding is completed, the resampled picture buffer is removed. Figure 6 An example of on-the-fly image-level resampling is shown, where a low-resolution reference image is resampled and stored in a resampled image buffer. Figure 6 As shown in the figure, the encoder or decoder's DPB contains three reference pictures. The resolution of reference pictures "Ref 0" and "Ref 2" is the same as that of the current picture. Therefore, "Ref 0" and "Ref 2" do not need to be resampled. However, the resolution of reference picture "Ref 1" is different from that of the current picture and therefore needs to be resampled. Therefore, only the resampled "Ref 1" is stored in the reference picture buffer, while "Ref 0" and "Ref 2" are not stored in the reference picture buffer.

[0066] Consistent with some exemplary embodiments, during on-the-fly picture-level resampling, the number of resampled reference pictures that a picture can use is limited. For example, a maximum number of resampled reference pictures for a given current picture can be preset. For example, the maximum number can be set to one. In this case, a bitstream constraint can be imposed such that an encoder or decoder can allow at most one of the reference pictures to have a resolution different from that of the current picture, and all other reference pictures must have the same resolution. Because this maximum number dictates the size of the resampled picture buffer and the maximum number of resamplings that can be performed for any picture in the current video sequence, the maximum number is directly related to the worst-case decoder complexity. Therefore, this maximum number can be signaled as part of a sequence parameter set (SPS) or picture parameter set (PPS), and it can be specified as part of a profile / level definition.

[0067] In some exemplary embodiments, two versions of the reference image are stored in the DPB. One version has the original resolution, and the other has the maximum resolution. If the resolution of the current image differs from the original resolution or the maximum resolution, the encoder or decoder can perform on-the-fly downsampling from the stored maximum resolution image. For image output, the DPB always outputs the original (unsampled) reference image.

[0068] In some exemplary embodiments, the resampling ratio can be arbitrarily selected, and different vertical and horizontal scaling ratios are allowed. Since picture-level resampling is applied, the block-level operation of the encoder / decoder is independent of the resolution of the reference picture, allowing the use of arbitrary resampling rates without further complicating the block-level design logic of the encoder / decoder.

[0069] In some exemplary embodiments, the signaling of the coding resolution and the maximum resolution can be performed as follows. The maximum resolution of any image in the video sequence is signaled in the SPS. The coding resolution of an image can be signaled in the PPS or in the slice header. In either case, a flag is signaled whether the coding resolution is the same as the maximum resolution. If the coding resolution is different from the maximum resolution, the coded width and height of the current image are additionally signaled. If a PPS signal is used, the signaled coding resolution is applied to all images that reference that PPS. If a slice header signal is used, the signaled coding resolution is applied only to the current image itself. In some embodiments, the difference between the current coding resolution and the signaled maximum resolution can be signaled in the SPS.

[0070] In some exemplary embodiments, instead of using an arbitrary resampling rate, the resolution is restricted to a predefined set of supported N resolutions, where N is the number of supported resolutions within the sequence. The value of N and the supported resolutions can be signaled in the SPS. The coded resolution of the picture is signaled via the PPS or slice header. In either case, instead of signaling the actual coded resolution, the corresponding resolution index is signaled. Table 3 ( Figure 7 ) shows an example of a supported resolution set, where the coding sequence allows 3 different resolutions. If the coding resolution of the current picture is 1440×816, the corresponding index value (=1) is signaled via the PPS or slice header.

[0071] In some exemplary embodiments, both resampled and un-resampled reference images are stored in the DPB. For each reference image, one original (i.e., un-resampled) image and N-1 resampled copies are stored. If the resolution of the reference image is different from the resolution of the current image, the resampled image in the DPB is used to encode the current image. For image output, the DPB outputs the original (un-resampled) reference image. As an example, Figure 8 Table 3 shows Figure 7 ) is given in the DPB occupancy of the supported resolution set. Figure 8 As shown, the DPB contains N (eg, N=3) copies of each picture used as a reference picture.

[0072] The choice of N value depends on the application. Larger values ​​of N provide more flexibility in resolution selection, but increase complexity and memory requirements. Smaller values ​​of N are suitable for devices with limited complexity, but they limit the available resolutions. Therefore, in some embodiments, the encoder can be given the flexibility to decide the value of N based on the application and device capabilities and signal this through the SPS.

[0073] Consistent with the present disclosure, downsampling of the reference image can be performed progressively (i.e., gradually). In conventional (i.e., direct) downsampling, downsampling of the input image is performed in one step. If the downsampling rate is high, single-step downsampling requires a longer tap downsampling filter to avoid severe aliasing problems. However, longer tap filters are computationally expensive. In some exemplary embodiments, to maintain the quality of the downsampled image, if the downsampling rate is above a threshold (e.g., above 2:1 downsampling), progressive downsampling is used where the downsampling is performed stepwise. For example, a downsampling filter sufficient for 2:1 downsampling can be repeatedly applied to achieve a downsampling ratio greater than 2:1. Figure 9An example of progressive downsampling is shown, where 4x downsampling is achieved on 2 images in both horizontal and vertical dimensions: the first image is 2x downsampled by 2 (bidirectional), and the second image is 2x downsampled by 2 (bidirectional).

[0074] Consistent with the present disclosure, the disclosed picture-level resampling methods can be used with other encoding tools. For example, in some exemplary embodiments, scaled motion vectors based on resampled reference pictures can be used during a temporal motion vector predictor (TMVP) or an advanced temporal motion vector predictor (ATMVP).

[0075] Next, we describe the test method for evaluating coding tools for ARC. Since adaptive resolution change is mainly used to adapt to network bandwidth, the following test conditions can be considered to compare the coding efficiency of different ARC schemes when network bandwidth changes.

[0076] In some embodiments, the resolution is changed to half (in both vertical and horizontal directions) at a particular time instance and then restored to the original resolution after a certain period of time. Figure 10 An example of resolution change is shown. At time t1, the resolution is reduced to half, and then restored to full resolution at time t2.

[0077] In some embodiments, when the resolution is reduced, if the downsampling ratio is too large (e.g., the downsampling ratio in a given dimension is greater than 2:1), then progressive downsampling is used for a certain period of time. However, when upsampling is used, progressive upsampling is not used.

[0078] In some embodiments, the Scalable HEVC Test Model (SMH) downsampling filter may be used for source resampling. Table 4 ( Figure 11 ) shows the detailed filter coefficients with different fractional sampling positions.

[0079] In some exemplary embodiments, two peak signal-to-noise ratios (PSNRs) are calculated to measure video quality. Figure 12 An illustration of the PSNR calculation is shown in . The first PSNR is calculated between the resampled source image and the decoded image. The second PSNR is calculated between the original source and the upsampled decoded source.

[0080] In some embodiments, the number of pictures to be encoded is the same as the current test condition of the VVC.

[0081] Next, the disclosed block-based resampling method is described. In block-based resampling, combining resampling and motion-compensated interpolation into a single filtering operation can reduce the aforementioned information loss. For example, consider the case where the motion vector of the current block has half-pixel precision in one dimension (e.g., the horizontal dimension) and the width of the reference image is twice the width of the current image. In this case, compared to image-level resampling, which halves the width of the reference image to match the width of the current image and then performs half-pixel motion interpolation, the block-based resampling method directly extracts odd-numbered locations in the reference image as reference blocks with half-pixel precision. At the 15th JVET meeting, VVC adopted a block-based ARC resampling method, in which motion-compensated (MC) interpolation and reference resampling are combined and performed in a single filter step. In VVC Draft 6, the existing filters for MC interpolation without reference resampling are reused for MC interpolation with reference resampling. The same filters are used for reference upsampling and downsampling. Details of the filter selection are described below.

[0082] For the luma component: if the half-pixel AMVR mode is selected and the interpolation position is half-pixel, use the 6-tap filter [3,9,20,20,9,3]; if the motion compensation block size is 4x4, use Table 5 ( Figure 13 ) as shown below; otherwise use Table 6 ( Figure 14 ) as shown in Table 7. Figure 15 ) as shown in the 4-tap filter.

[0083] In VVC, the same filter is used for MC interpolation without reference resampling and MC interpolation with reference resampling. Although the VVC MCIF is designed based on DCT upsampling, it may not be appropriate to use it as a one-step filter for combining reference downsampling and MC interpolation. For example, for 0-phase filtering, that is, the scaled mv is an integer, the VVC 8-tap MCIF coefficients are [0,0,0,64,0,0,0,0], which means that the prediction samples are copied directly from the reference samples. Although this may not be a problem for MC interpolation without reference downsampling or with reference upsampling, it may cause aliasing in the case of reference downsampling due to the lack of a low-pass filter before decimation.

[0084] The present disclosure provides a method for MC interpolation with reference downsampling using a cosine windowed sine filter.

[0085] A windowed sinc filter is a bandpass filter that separates one frequency band from other frequency bands. A windowed sinc filter is a low-pass filter whose frequency response allows all frequencies below the cutoff frequency to pass with an amplitude of 1 and blocks all frequencies above the cutoff frequency with zero amplitude, as shown in Figure 16 shown.

[0086] The filter kernel, also known as the filter's impulse response, is obtained by taking the inverse Fourier transform of the frequency response of an ideal low-pass filter. The impulse response of a low-pass filter has the general form of a sinc function:

[0087]

[0088] where f c is the cutoff frequency, which is in the range [0,1], and r is the downsampling ratio, for example, 1.5 means 1.5:1 downsampling, and 2 means 2:1 downsampling. The sinc function is defined as follows:

[0089]

[0090] The sinc function is infinite. In order to make the filter kernel length finite, a window function should be used to truncate the filter kernel into L points. In order to obtain a smooth conical curve, the cosine window function is used, which is given by the following formula:

[0091]

[0092] The kernel of the cosine windowed sinc filter is the product of the ideal response function h(n) and the cosine window function w(n):

[0093]

[0094] Two parameters are chosen for the windowed sinc kernel, the cutoff frequency f c and kernel length L. For the downsampling filter used in the Scalable HEVC Test Model (SHM), f c =0.9 and L=13.

[0095] The filter coefficients obtained in (4) are real numbers. Applying the filter is equivalent to calculating a weighted average of the reference samples, with the weights being the filter coefficients. For efficient computation in a digital computer or hardware, the coefficients are normalized, multiplied by a scaler, and rounded to integers so that the sum of the coefficients is equal to 2^N, where N is an integer. The filtered samples are divided by 2^N (equivalent to a right shift by N bits). For example, in VVC draft 6, the sum of the interpolation filter coefficients is 64.

[0096] In some disclosed embodiments, the downsampling filter is used for VVC motion compensated interpolation with reference downsampling for luma and chroma components in SHM, and the existing MCIF is used for motion compensated interpolation with reference upsampling. In the case of kernel length L = 13, the first coefficient is small and rounded to zero, and the filter length can be reduced to 12 without affecting the filter performance.

[0097] As an example, in Table 8 ( Figure 17 ) and Table 9( Figure 18 ) show the filter coefficients for 2:1 downsampling and 1.5:1 downsampling respectively.

[0098] Besides the values ​​of the coefficients, there are several other differences between the design of the SHM filter and the existing MCIF.

[0099] As a first difference, the SHM filter needs to filter at both integer and fractional sample positions, while the MCIF filter only needs to filter at fractional sample positions. Table 10 ( Figure 19 An example of a modification to the VVC draft 6 luma sample interpolation filtering process for the reference downsampling case is described in .

[0100] Table 11 Figure 20 An example of a modification to the VVC draft 6 chroma sample interpolation filtering process for the reference downsampling case is described in .

[0101] As a second difference, the sum of the filter coefficients of the SHM filter is 128, while the sum of the filter coefficients of the existing MCIF is 64. In VVC draft 6, in order to reduce the loss caused by rounding errors, the intermediate prediction signal is kept at a higher precision (represented by a higher bit depth) than the output signal. The precision of the intermediate signal is called internal precision. In one embodiment, in order to keep the internal precision the same as in VVC draft 6, the output of the SHM filter needs to be right shifted by 1 additional bit compared to using the existing MCIF. Table 12 ( Figure 21 ) shows an example of modification of the VVC draft 6 luma sample interpolation filtering process for the reference downsampling case.

[0102] Table 13 Figure 22 An example of a modification to the VVC draft 6 chroma sample interpolation filtering process for the reference downsampling case is shown in .

[0103] According to some embodiments, the internal precision may be increased by 1 bit, and a 1-bit right shift may be added to convert the internal precision to the output precision. Table 14 ( Figure 23 An example of a modification to the VVC draft 6 luma sample interpolation filtering process for the reference downsampling case is shown in FIG.

[0104] Table 15 Figure 24 An example of a modification to the VVC draft 6 chroma sample interpolation filtering process for the reference downsampling case is shown in .

[0105] As a third difference, the SHM filter has 12 taps. Therefore, to generate the interpolated sample, 11 adjacent samples are required (5 to the left, 6 to the right, or 5 above or 6 below). Compared to MCIF, additional adjacent samples are obtained. In VVC draft 6, the chroma mv accuracy is 1 / 32. However, the SHM filter has only 16 phases. Therefore, the chroma mv can be rounded to 1 / 16 for reference downsampling. This can be done by shifting the last 5 bits of the chroma mv right by 1 bit. Table 16 ( Figure 25 An example of a modification to the VVC draft 6 chroma fractional sample position calculation for the reference downsampling case is shown in .

[0106] According to some embodiments, in order to align with the existing MCIF design in the VVC draft, we propose to use an 8-tap cosine windowed sine filter. The filter coefficients can be derived by setting L=9 in the cosine windowed sine filter function in equation (4). The sum of the filter coefficients can be set to 64 to further align with the existing MCIF filter. In the following Table 17 ( Figure 26 ) and Table 18( Figure 27 ) show example filter coefficients for ratios of 2:1 and 1.5:1, respectively.

[0107] According to some embodiments, to accommodate the 1 / 32 sampling accuracy of the chroma components, a 32-phase cosine-windowed sine filter bank may be used in chroma motion compensation interpolation with reference downsampling. Examples of filter coefficients for 2:1 and 1.5:1 ratios are shown in Table 19 ( Figure 28 ) and Table 20( Figure 29 ) is shown in ).

[0108] According to some embodiments, for a 4x4 luma block, a 6-tap cosine windowed sine filter may be used for MC interpolation with reference downsampling. Examples of filter coefficients for 2:1 and 1.5:1 ratios are shown in Table 21 below ( Figure 30 ) and Table 22( Figure 31 ) is shown in ).

[0109] In some embodiments, a non-transitory computer-readable storage medium comprising instructions is also provided, and the instructions can be executed by a device for performing the above method (e.g., the disclosed encoder and decoder). Common forms of non-transitory media include, for example, floppy disks, disks, hard disks, solid-state drives, tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with hole patterns, RAM, PROM and EPROM, FLASH-EPROM or any other flash memory, NVRAM, caches, registers, any other memory chips or cassettes, and network versions thereof. The device may include one or more processors (CPUs), input / output interfaces, network interfaces, and / or memories.

[0110] It should be noted that the relational terms such as "first" and "second" in this document are used only to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or order between these entities or operations. In addition, the words "include", "have", "include" and "including" and other similar forms are equivalent in meaning and are open-ended, in that the one or more items following any of these words are not intended to be an exhaustive list of such items or items, or to be limited to the listed items.

[0111] As used herein, unless specifically stated otherwise, the term "or" encompasses all possible combinations unless not feasible. For example, if a database is stated to contain either A or B, then, unless explicitly stated otherwise or not feasible, the database may contain A, or B, or A and B. As a second example, if a database is stated to contain either A, B, or C, then, unless explicitly stated otherwise or not feasible, the database may contain A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0112] It will be appreciated that the above embodiments may be implemented by hardware, or software (program code), or a combination of hardware and software. If implemented by software, it may be stored in the above-mentioned computer-readable medium. The software may execute the disclosed method when executed by a processor. The computing units and other functional units described in the present disclosure may be implemented by hardware, or software, or a combination of hardware and software. It will also be appreciated by those skilled in the art that multiple of the above-mentioned modules / units may be combined into one module / unit, and each of the above-mentioned modules / units may be further divided into multiple submodules / subunits.

[0113] In the foregoing description, embodiments have been described with reference to numerous specific details that may vary depending on the implementation. Certain modifications and variations may be made to the described embodiments. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. The description and embodiments are to be considered exemplary only, with the true scope and spirit of the invention being indicated by the claims. The order of steps shown in the figures is also intended to be illustrative only and is not intended to be limiting to any particular order of steps. Therefore, those skilled in the art will appreciate that the steps may be performed in a different order while implementing the same method.

[0114] The embodiments may be further described using the following terms:

[0115] 1. A computer-implemented video processing method, comprising:

[0116] comparing the resolution of the target image and the first reference image;

[0117] In response to the target image and the first reference image having different resolutions, resampling the first reference image to generate a second reference image; and

[0118] The target image is encoded or decoded using the second reference image.

[0119] 2. The method according to clause 1, further comprising:

[0120] storing the second reference picture in a first buffer that is different from a second buffer that stores decoded pictures used to predict future pictures; and

[0121] After encoding or decoding of the target image is completed, the second reference image is removed from the first buffer.

[0122] 3. The method according to any one of clauses 1 and 2, wherein encoding or decoding the target image comprises:

[0123] The target image is encoded or decoded by using no more than a predetermined number of resampled reference images.

[0124] 4. The method of clause 3, wherein encoding or decoding the target image comprises:

[0125] The predetermined number is signaled as part of a sequence parameter set or a picture parameter set.

[0126] 5. The method of clause 1, wherein resampling the first reference image to generate a second reference image comprises:

[0127] storing a first version and a second version of the first reference image, the first version having an original resolution and the second version having a maximum resolution allowed for resampling the reference image; and

[0128] Resample the maximum resolution version to generate the second reference image.

[0129] 6. The method according to clause 5, further comprising:

[0130] storing the first version and the second version of the first reference picture in a decoded picture buffer; and

[0131] The first version is output for encoding future images.

[0132] 7. The method of clause 1, wherein resampling the first reference image to generate a second reference image comprises:

[0133] A second reference image is generated at a supported resolution.

[0134] 8. The method according to clause 7, further comprising:

[0135] The information signaled as part of a sequence parameter set or picture parameter set indicates:

[0136] Supported resolutions, and

[0137] Pixel dimensions corresponding to supported resolutions.

[0138] 9. The method of clause 8, wherein the information comprises:

[0139] An index corresponding to at least one supported resolution.

[0140] 10. The method according to any one of clauses 8 and 9, further comprising:

[0141] Set the number of supported resolutions based on the configuration of your video application or video device.

[0142] 11. A method according to any one of clauses 1 to 10, wherein said resampling the first reference image to generate a second reference image:

[0143] The first reference image is progressively downsampled to generate a second reference image.

[0144] 12. A device comprising:

[0145] one or more memories storing computer instructions; and

[0146] one or more processors configured to execute the computer instructions to cause the device to:

[0147] comparing the resolution of the target image with the first reference image;

[0148] In response to the target image and the first reference image having different resolutions, resampling the first reference image to generate a second reference image; and

[0149] The target image is encoded or decoded using the second reference image.

[0150] 13. A non-transitory computer-readable medium storing a set of instructions executable by at least one processor of a computer system to cause the computer system to perform a method for processing video content, the method comprising:

[0151] comparing the resolution of the target image and the first reference image;

[0152] In response to the target image and the first reference image having different resolutions, resampling the first reference image to generate a second reference image; and

[0153] The target image is encoded or decoded using the second reference image.

[0154] 14. A computer-implemented video processing method comprising:

[0155] In response to a target image and a reference image having different resolutions, applying a bandpass filter to the reference image to perform motion compensated interpolation and generate a reference block; and

[0156] A block of a target image is encoded or decoded using the reference block.

[0157] 15. The method of clause 14, wherein the bandpass filter is a cosine-windowed sine filter.

[0158] 16. The method according to clause 15, wherein the cosine-windowed sine filter has a kernel function in fc is the cutoff frequency of the cosine-windowed sine filter, L is the kernel length, and r is the downsampling rate.

[0159] 17. The method of clause 16, wherein fc is equal to 0.9 and L is equal to 13.

[0160] 18. A method according to any of clauses 15-17, wherein the cosine-windowed sine filter is an 8-tap filter.

[0161] 19. A method according to any of clauses 15-17, wherein the cosine-windowed sine filter is a 4-tap filter.

[0162] 20. A method according to any of clauses 15-17, wherein the cosine-windowed sine filter is a 32-phase filter.

[0163] 21. A method according to clause 20, wherein the 32-phase filter is used in chroma motion compensated interpolation.

[0164] 22. A method according to any of clauses 14 to 21, wherein applying a bandpass filter to the reference image comprises:

[0165] Gets a luma or chroma sample at a fractional sample position.

[0166] 23. An apparatus comprising:

[0167] one or more memories storing computer instructions; and

[0168] one or more processors configured to execute the computer instructions to cause the device to:

[0169] In response to a target image and a reference image having different resolutions, a bandpass filter is applied to the reference image to perform motion compensated interpolation and generate a reference block. And

[0170] The block of the target image is encoded or decoded using the reference block.

[0171] 24. A non-transitory computer-readable medium storing a set of instructions executable by at least one processor of a computer system to cause the computer system to perform a method for processing video content, the method comprising:

[0172] 25. A non-transitory computer-readable medium storing a set of instructions executable by at least one processor of a computer system to cause the computer system to perform a method for processing video content, the method comprising:

[0173] comparing the resolution of the target image and the first reference image;

[0174] In response to the target image and the first reference image having different resolutions, resampling the first reference image to generate a second reference image; and

[0175] The target image is encoded or decoded using the second reference image.

[0176] In the drawings and the specification, exemplary embodiments have been disclosed. However, many variations and modifications may be made to these embodiments. Therefore, although specific terms are used, they are used in a generic and descriptive sense only and not for the purpose of limitation.

Claims

1. A computer-implemented video processing method, applied to an encoder, comprising: Compare the resolution of the target image and the reference image; In response to the target image and the reference image having different resolutions, encoding or decoding the target image by applying a bandpass filter to the reference image; wherein the bandpass filter is used to resample the reference image and perform motion compensation interpolation based on the resampled reference image; wherein the bandpass filter has a plurality of coefficients derived from a cosine-windowed sine filter; The cosine windowed sine filter has a kernel function in fc is the cutoff frequency of the cosine-windowed sine filter, L is the kernel length, and r is the downsampling rate.

2. The method of claim 1 , wherein applying a bandpass filter to the reference image comprises: An 8-tap bandpass filter is applied to the luma samples of the reference image.

3. The method of claim 1 , wherein applying a bandpass filter to the reference image comprises: A 4-tap bandpass filter is applied to the chroma samples of the reference image. The method of claim 1 , wherein a sum of the plurality of coefficients is equal to 64. The method of claim 1 , wherein a sum of the plurality of coefficients is equal to 128. The method of claim 1 , wherein the reference image has a higher resolution than the target image. The method of claim 1 , wherein fc is equal to 0.9 and L is equal to 13.

8. The method of claim 1, wherein the cosine-windowed sine filter is a 32-phase filter.

9. The method of claim 1 , wherein applying a bandpass filter to the reference image comprises: Gets a luma or chroma sample at a fractional sample position.

10. A computer-implemented video processing device, applied to a decoder, comprising: one or more memories storing computer instructions; and one or more processors configured to execute the computer instructions to cause the device to: Compare the resolution of the target image with the reference image; In response to the target image and the reference image having different resolutions, encoding or decoding the target image by applying a bandpass filter to the reference image; wherein the bandpass filter is used to resample the reference image and perform motion compensation interpolation based on the resampled reference image; wherein the bandpass filter has a plurality of coefficients derived from a cosine-windowed sine filter; The cosine windowed sine filter has a kernel function in fc is the cutoff frequency of the cosine-windowed sine filter, L is the kernel length, and r is the downsampling rate.

11. The device of claim 10, wherein the one or more processors are further configured to execute the computer instructions so that the device: An 8-tap bandpass filter is applied to the luma samples of the reference image.

12. The device of claim 10, wherein the one or more processors are configured to execute the computer instructions to cause the device to: A 4-tap bandpass filter is applied to the chroma samples of the reference image.

13. The apparatus of claim 10, wherein a sum of the plurality of coefficients is equal to 64. The apparatus of claim 10 , wherein a sum of the plurality of coefficients is equal to 128.

15. The apparatus of claim 10, wherein the reference image has a higher resolution than the target image.

16. A non-transitory computer-readable medium having stored thereon a bitstream generated by executing a set of instructions, the set of instructions being executable by at least one processor of a computer system to cause the computer system to perform a method comprising the steps of: Compare the resolution of the target image and the reference image; In response to the target image and the reference image having different resolutions, encoding or decoding the target image by applying a bandpass filter to the reference image; in, The bandpass filter is used to resample the reference image and perform motion compensated interpolation based on the resampled reference image; wherein the bandpass filter has a plurality of coefficients derived from a cosine-windowed sine filter; The cosine windowed sine filter has a kernel function in fc is the cutoff frequency of the cosine-windowed sine filter, L is the kernel length, and r is the downsampling rate.

Images