Filter performing motion compensation interpolation by resampling

Abstract

Systems and methods for processing video content using motion compensation interpolation are provided. The method includes: applying a band-pass filter to a reference picture for a target picture and the reference picture having different resolutions, and performing motion compensation interpolation by reference down-sampling to generate a reference block; and encoding or decoding a block of a target picture using the reference block.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 62 / 904,608, filed on September 23, 2019, and U.S. Provisional Application No. 62 / 906,930, filed on September 27, 2019, both of which are incorporated herein by reference. Background Art

[0002] The present application relates to video processing, and more particularly to filters for performing motion compensated interpolation by resampling.

[0003] A video is a set of static pictures (or "frames") that capture visual information. In order to reduce storage memory and transmission bandwidth, the video can be compressed before storage or transmission, and decompressed before display. The compression process is usually called encoding, and the decompression process is usually called decoding. There are currently a variety of video coding formats that use standardized video coding technologies. The most common ones are video coding formats based on prediction, transform, quantization, entropy coding, and in-loop filtering. Video coding standards that specify specific video coding formats, such as the High Efficiency Video Coding (HEVC / H.265) standard, the Versatile Video Coding (VVC / H.266) standard, the AVS standard, etc., are developed by standardization organizations. As more and more advanced video coding technologies are adopted in video standards, the coding efficiency of new video coding standards is also getting higher and higher. Application Contents

[0004] Embodiments of the present application provide a computer-implemented method for performing motion-compensated interpolation through resampling. The method may include: applying a bandpass filter to a target picture and a reference picture having different resolutions, performing motion-compensated interpolation through reference downsampling to generate a reference block; and processing a block of the target picture using the reference block.

[0005] The present application also provides a system for performing motion-compensated interpolation. The system may include: a memory storing a set of instructions; and at least one processor configured to execute the set of instructions to cause the device to: for a target image and a reference image having different resolutions, apply a bandpass filter to the reference image, perform motion-compensated interpolation by reference downsampling to generate a reference block; and process a block of the target image using the reference block.

[0006] The present application also provides a non-transitory computer-readable medium storing a set of instructions executable by at least one processor of a computer system to cause the computer system to perform a method for processing video content. The method may include: applying a bandpass filter to a target image and a reference image having different resolutions, performing motion-compensated interpolation on the reference image through reference downsampling to generate a reference block; and processing a block of the target image using the reference block. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments and aspects of the present application are illustrated in the following detailed description and accompanying drawings.The various features shown in the drawings are not drawn to scale.

[0008] Figure 1 The structure of an exemplary video sequence consistent with an embodiment of the present application is shown.

[0009] Figure 2A A schematic diagram showing an exemplary encoding process of a hybrid video coding system consistent with an embodiment of the present application is shown.

[0010] Figure 2B A schematic diagram showing another exemplary encoding process of a hybrid video coding system consistent with an embodiment of the present application is shown.

[0011] Figure 3A A schematic diagram showing an exemplary decoding process of a hybrid video coding system consistent with an embodiment of the present application is shown.

[0012] Figure 3B A schematic diagram showing another exemplary decoding process of a hybrid video coding system consistent with an embodiment of the present application is shown.

[0013] Figure 4 is a block diagram of an exemplary apparatus for encoding or decoding video consistent with embodiments of the present application.

[0014] Figure 5 A schematic diagram of a reference picture and a current picture consistent with an embodiment of the present application is shown.

[0015] Figure 6 (Table 6: Exemplary interpolation filter coefficients based on 6-tap DCT for 4×4 luminance component in VVC) shows an exemplary interpolation filter coefficient table based on 6-tap DCT for 4×4 luminance component consistent with an embodiment of the present application.

[0016] Figure 7 (Table 7: Exemplary 8-tap interpolation filter coefficient table for luma component in VVC) shows an exemplary 8-tap interpolation filter coefficient table for luma component consistent with an embodiment of the present application.

[0017] Figure 8 (Table 8: Exemplary 4-tap 32-phase interpolation filter coefficient table for chroma components in VVC) shows an exemplary 4-tap 32-phase interpolation filter coefficient table for chroma components consistent with an embodiment of the present application.

[0018] Figure 9 The frequency response of an exemplary ideal low-pass filter consistent with embodiments of the present application is shown.

[0019] Figure 10 (Table 10: Example 12-tap cosine window function difference interpolation filter coefficient table for 2:1 downsampling) shows an example 12-tap cosine window function difference interpolation filter coefficient table for 2:1 downsampling consistent with an embodiment of the present application.

[0020] Figure 11 (Table 11: Exemplary 12-tap cosine window function difference interpolation filter coefficients for 1.5:1 downsampling) shows an exemplary 12-tap cosine window function difference interpolation filter coefficient table for 1.5:1 downsampling consistent with an embodiment of the present application.

[0021] Figure 12 (Table 12: Exemplary luma sample interpolation filtering process for reference downsampling) shows an exemplary luma sample interpolation filtering process for reference downsampling consistent with an embodiment of the present application.

[0022] Figure 13 (Table 13: Modification example of the interpolation filtering process for chroma samples for reference downsampling case) shows an exemplary chroma sample interpolation filtering process for reference downsampling consistent with an embodiment of the present application.

[0023] Figure 14 (Table 14: Modification example of the interpolation filtering process for luma samples for reference downsampling case) shows an exemplary luma sample interpolation filtering process for reference downsampling consistent with an embodiment of the present application.

[0024] Figure 15 (Table 15: Modification example of the interpolation filtering process for chroma samples in reference downsampling case) shows an exemplary chroma sample interpolation filtering process for reference downsampling consistent with an embodiment of the present application.

[0025] Figure 16 (Table 16: Modification example of the interpolation filtering process for V luma samples for reference downsampling case) shows an exemplary luma sample interpolation filtering process for reference downsampling consistent with an embodiment of the present application.

[0026] Figure 17(Table 17: Modification example of the interpolation filtering process for chroma samples for reference downsampling case) shows an exemplary chroma sample interpolation filtering process for reference downsampling consistent with an embodiment of the present application.

[0027] Figure 18 (Table 18: Modified example of calculation of chroma fractional sampling positions for reference downsampling case) shows an exemplary chroma sample interpolation filtering process for reference downsampling consistent with an embodiment of the present application.

[0028] Figure 19 (Table 19: Exemplary 8-tap filter for performing MC interpolation with a reference downsampling rate of 2:1) shows an exemplary 8-tap filter for performing MC interpolation consistent with an embodiment of the present application with a reference downsampling rate of 2:1.

[0029] Figure 20 (Table 20: Exemplary 8-tap filter for performing MC interpolation with a reference downsampling rate of 1.5:1) shows an exemplary 8-tap filter for performing MC interpolation consistent with an embodiment of the present application with a reference downsampling rate of 1.5:1.

[0030] Figure 21 (Table 21: An exemplary 8-tap filter for performing MC interpolation, with a reference downsampling rate of 2:1 and a phase of 32) shows an exemplary 8-tap filter for performing MC interpolation, with a reference downsampling rate of 2:1 and a phase of 32, consistent with an embodiment of the present application.

[0031] Figure 22 (Table 22: An exemplary 8-tap filter for performing MC difference, with a reference downsampling rate of 1.5:1 and a phase of 32) shows an exemplary 8-tap filter for performing MC difference consistent with an embodiment of the present application, with a reference downsampling rate of 1.5:1 and a phase of 32.

[0032] Figure 23 (Table 23: An exemplary 6-tap filter coefficient table for 4×4 luma block MC interpolation, with a reference downsampling ratio of 2:1 and a phase of 16 phases) shows an exemplary 6-tap filter coefficient table for 4×4 luma block MC interpolation consistent with an embodiment of the present application, with a reference downsampling ratio of 2:1 and a phase of 16 phases.

[0033] Figure 24 (Table 24: An exemplary 6-tap filter coefficient table for luma 4×4 block MC interpolation with a reference downsampling rate of 1.5:1 and 16 phases) shows an exemplary 6-tap filter coefficient table for luma 4×4 block MC interpolation with a reference downsampling rate of 1.5:1 and 16 phases consistent with an embodiment of the present application.

[0034] Figure 25 (Table 25: 8-tap filter for MC interpolation with reference downsampling at a ratio of 2:1) shows a table of an exemplary 8-tap filter consistent with an embodiment of the present application, which is used for MC interpolation with reference downsampling at a ratio of 2:1.

[0035] Figure 26 (Table 26: Exemplary 8-tap filter for MC interpolation with a reference downsampling ratio of 1.5:1) shows an exemplary 8-tap filter for MC interpolation consistent with an embodiment of the present application with a reference downsampling ratio of 1.5:1.

[0036] Figure 27 (Table 27: An exemplary 6-tap filter for MC interpolation with 16 phases and reference downsampling at a ratio of 2:1) shows an exemplary 6-tap filter for MC interpolation consistent with an embodiment of the present application, with 16 phases and reference downsampling at a ratio of 2:1.

[0037] Figure 28 (Table 28: An exemplary 6-tap filter for MC interpolation with 16 phases and reference downsampling at a ratio of 1.5:1) shows an exemplary 6-tap filter for MC interpolation consistent with an embodiment of the present application, with 16 phases and reference downsampling at a ratio of 1.5:1.

[0038] Figure 29 (Table 29: An exemplary 4-tap filter for MC interpolation with 32 phases and reference downsampling at a ratio of 2:1) shows an exemplary 4-tap filter for MC interpolation consistent with an embodiment of the present application, with 32 phases and reference downsampling at a ratio of 2:1.

[0039] Figure 30 (Table 30: An exemplary 4-tap filter for MC interpolation with 32 phases and reference downsampling at a ratio of 1.5:1) shows an exemplary 4-tap filter for MC interpolation consistent with an embodiment of the present application, with 32 phases and reference downsampling at a ratio of 1.5:1.

[0040] Figure 31 (Table 31: Exemplary 8-tap filter for MC interpolation with a reference downsampling ratio of 2:1) shows an exemplary 8-tap filter for MC interpolation consistent with an embodiment of the present application with a reference downsampling ratio of 2:1.

[0041] Figure 32(Table 32: Exemplary 8-tap filter for MC interpolation with a reference downsampling ratio of 1.5:1) shows an exemplary 8-tap filter for MC interpolation consistent with an embodiment of the present application with a reference downsampling ratio of 1.5:1.

[0042] Figure 33 (Table 33: An exemplary 6-tap filter for MC interpolation with 16 phases and reference downsampling at a ratio of 2:1) shows an exemplary 6-tap filter for MC interpolation consistent with an embodiment of the present application, with 16 phases and reference downsampling at a ratio of 2:1.

[0043] Figure 34 (Table 34: An exemplary 6-tap filter for MC interpolation with 16 phases and reference downsampling at a ratio of 1.5:1) shows an exemplary 6-tap filter for MC interpolation consistent with an embodiment of the present application, with 16 phases and reference downsampling at a ratio of 1.5:1.

[0044] Figure 35 (Table 35: An exemplary 4-tap filter for MC interpolation, the filter having 32 phases and reference downsampling at a ratio of 2:1) shows an exemplary 4-tap filter for MC interpolation consistent with an embodiment of the present application, the filter having 32 phases and reference downsampling at a ratio of 2:1.

[0045] Figure 36 (Table 36: Exemplary 4-tap filter for MC interpolation with a reference downsampling rate of 1.5:1 and 32 phases) shows an exemplary 4-tap filter for MC interpolation consistent with an embodiment of the present application, with a reference downsampling rate of 1.5:1 and 32 phases.

[0046] Figure 37 (Table 37: Example of filter coefficient signaling) shows an example of filter coefficient signaling consistent with an embodiment of the present application.

[0047] Figure 38 (Table 38: Exemplary syntax structure for marking resampling rates and corresponding filter groups) shows an exemplary syntax structure for marking resampling rates and corresponding filter groups consistent with an embodiment of the present application.

[0048] Figure 39 is a flowchart of an exemplary method for processing video content consistent with an embodiment of the present application.

[0049] Figure 40 4 is a flowchart of an exemplary method for rounding filter coefficients of a cosine window function difference filter consistent with an embodiment of the present application. DETAILED DESCRIPTION

[0050] Reference can now be made in detail to example embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which the same numbers in different figures represent the same or similar elements, unless otherwise specified. The embodiments set forth in the following description of example embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with the aspects related to the present application described in the appended claims. Certain aspects of the present application are described in more detail below. In the event of a conflict with terms and / or definitions incorporated by reference, the terms and definitions provided herein shall prevail.

[0051] Video coding systems are commonly used to compress digital video signals, for example to reduce storage space consumption or transmission bandwidth consumption associated with such signals. With the increasing popularity of high-definition (HD) video (e.g., with a resolution of 1920×1080 pixels) in various video compression applications, such as online video streaming, video conferencing, or video surveillance, there is a growing demand for video coding tools that can improve the efficiency of video data compression.

[0052] For example, video surveillance applications are becoming increasingly widespread in many scenarios (e.g., security, transportation, and environmental monitoring), and the number and resolution of surveillance devices continue to grow rapidly. Many video surveillance applications prefer to provide users with high-definition video to capture more information, with more pixels per frame to capture such information. However, high-definition video bitstreams can have high bitrates, which requires high-bandwidth transmission and large storage space. For example, a surveillance video stream with an average resolution of 1920×1080 can require up to 4Mbps of bandwidth for real-time transmission. Furthermore, video surveillance typically monitors continuously 24 / 7, which greatly challenges storage systems if video data is to be stored. Therefore, the high bandwidth and large storage requirements of high-definition video have become the main limitations of its large-scale deployment in video surveillance.

[0053] A video is a set of still images (or "frames") arranged in a temporal sequence to store visual information. A video capture device (e.g., a camera) can be used to capture and store these images in temporal order, and a video playback device (e.g., a television, computer, smartphone, tablet, video player, or any end-user terminal with a display) can be used to display these images in temporal order. Furthermore, in some applications, the video capture device can transmit the captured video to a video playback device (e.g., a computer with a display) in real time, for example, for monitoring, conferencing, or live broadcasting.

[0054] To reduce the storage space and transmission bandwidth required for such applications, video can be compressed before storage and transmission, and decompressed before display. Compression and decompression can be implemented by software executed by a processor (e.g., a general-purpose computer processor) or dedicated hardware. The module used for compression is generally referred to as an "encoder," and the module used for decompression is generally referred to as a "decoder." Encoders and decoders can be collectively referred to as "codecs." Encoders and decoders can be implemented as any of a variety of suitable hardware, software, or combinations thereof. For example, hardware implementations of encoders and decoders can include circuits such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, or any combination thereof. Software implementations of encoders and decoders can include program code, computer-executable instructions, firmware, or any suitable computer-implemented algorithm or process fixed in a computer-readable medium. Video compression and decompression can be implemented using various algorithms or standards, such as MPEG-1, MPEG-2, MPEG-4, the H.26x series, etc. In some applications, a codec can decompress video from a first coding standard and recompress the decompressed video using a second coding standard. In this case, the codec can be referred to as a "transcoder."

[0055] The video encoding process identifies and retains useful information that can be used to reconstruct the image, while ignoring unimportant information during the reconstruction process. If the ignored, unimportant information cannot be fully reconstructed, the encoding process is called "lossy." Otherwise, it is called "lossless." Most encoding processes are lossy, a trade-off made to reduce required storage space and transmission bandwidth.

[0056] Useful information about the picture being encoded (called the "current picture") includes changes relative to a reference picture (e.g., a previously encoded and reconstructed picture). Such changes can include changes in pixel position, brightness, or color, with position changes being of greatest interest. A change in the position of a group of pixels representing an object can reflect the object's motion between the reference picture and the current picture.

[0057] A picture that is encoded without reference to another picture (i.e., it is its own reference picture) is called an "I-picture." A picture that is encoded using a previous picture as a reference picture is called a "P-picture." A picture that is encoded using both a previous picture and a future picture as reference pictures (i.e., the reference is "bidirectional") is called a "B-picture."

[0058] As mentioned earlier, video surveillance using high-definition video faces the challenges of high bandwidth and storage requirements. To address these challenges, the bit rate of the encoded video can be reduced. Among I-, P-, and B-pictures, I-pictures have the highest bit rate. Since the background of most surveillance videos is nearly static, one way to reduce the overall bit rate of the encoded video is to use fewer I-pictures for video encoding.

[0059] However, the improvement from using fewer I pictures may be negligible, as I pictures typically do not dominate the encoded video. For example, in a typical video bitstream, the ratio of I, B, and P pictures may be 1:20:9, where I pictures may account for less than 10% of the total bitrate. In other words, in such an example, even if all I pictures were removed, the bitrate reduction would not exceed 10%.

[0060] Figure 1 The structure of an example video sequence 100 according to some embodiments of the present application is illustrated. The video sequence 100 can be real-time video or video that has been captured and archived. The video 100 can be real-life video, computer-generated video (e.g., computer game video), or a combination thereof (e.g., real-life video with augmented reality effects). The video sequence 100 can be input from a video capture device (e.g., a camera), a video archive containing previously captured video (e.g., a video file stored on a storage device), or a video input interface (e.g., a video broadcast transceiver) to receive video from a video content provider.

[0061] like Figure 1 As shown, video sequence 100 may include a series of pictures arranged temporally along a timeline, including pictures 102, 104, 106, and 108. Pictures 102-106 are consecutive, and there are more pictures between pictures 106 and 108. Figure 1 In the example, picture 102 is an I-picture, and its reference picture is picture 102 itself. Picture 104 is a P-picture, as indicated by the arrow, and its reference picture is picture 102. Picture 106 is a B-picture, as indicated by the arrow, and its reference pictures are pictures 104 and 108. In some embodiments, the reference pictures of a picture (e.g., picture 104) may not be directly before or after the picture. For example, the reference picture of picture 104 may be a picture before picture 102. It should be noted that the reference pictures of pictures 102-106 are only examples, and this application does not limit the embodiments of reference pictures to Figure 1 The example shown in .

[0062] Due to the computational complexity of such tasks, video codecs typically do not encode or decode an entire picture at once. Instead, they can divide the picture into basic segments and encode or decode the picture segment by segment. In this application, these basic segments are called basic processing units ("BPUs"). For example, Figure 1 Structure 110 in shows an example structure of a picture (e.g., any one of pictures 102-108) of video sequence 100. In structure 110, the picture is divided into 4×4 basic processing units, whose boundaries are shown as dashed lines. In some embodiments, the basic processing unit may be referred to as a "macroblock" in some video coding standards (e.g., the MPEG family, H.261, H.263, or H.264 / AVC), or as a "coding tree unit" ("CTU") in some other video coding standards (e.g., H.265 / HEVC or H.266 / VVC). The basic processing units in a picture can have different sizes, such as 128×128, 64×64, 32×32, 16×16, 4×8, 16×32, or pixels of arbitrary shape and size. The size and shape of the basic processing unit for a picture can be selected based on a balance between coding efficiency and the level of detail to be retained in the basic processing unit.

[0063] A basic processing unit may be a logical unit that may include a set of different types of video data stored in a computer memory (e.g., in a video frame buffer). For example, a basic processing unit for a color picture may include a luma component (Y) representing achromatic luma information, one or more chroma components (e.g., Cb and Cr) representing color information, and associated syntax elements, where the luma and chroma components may have basic processing units of the same size. In some video coding standards (e.g., H.265 / HEVC or H.266 / VVC), the luma and chroma components may be referred to as "coding tree blocks" ("CTBs"). Any operation performed on a basic processing unit may be repeated for each of its luma and chroma components.

[0064] Video encoding has multiple stages of operation, examples of which are given in Figures 2A-2B and Figures 3A-3BAs shown in . For each stage, the size of the basic processing unit may still be too large to be processed, so it can be further divided into segments referred to as "basic processing sub-units" in this application. In some embodiments, the basic processing sub-unit may be referred to as a "block" in some video coding standards (for example, the MPEG series, H.261, H.263 or H.264 / AVC), or as a "coding unit" ("CU") in certain other video coding standards (for example, H.265 / HEVC or H.266 / VVC). The basic processing sub-unit may have the same or smaller size as the basic processing unit. Similar to the basic processing unit, the basic processing sub-unit is also a logical unit, which may include storage in computer memory (for example, in a video frame buffer). Any operation performed on the basic processing sub-unit can be repeated for each of its luminance and chrominance components. It should be noted that this division can be performed to a further level according to processing needs. It should also be noted that different stages can use different schemes to divide the basic processing units.

[0065] For example, in the mode decision phase (an example of which is in Figure 2B ), the encoder can decide which prediction mode to use for a basic processing unit (e.g., intra-picture prediction or inter-picture prediction), and the basic processing unit may be too large to make such a decision. The encoder can split the basic processing unit into multiple basic processing sub-units (e.g., CUs in H.265 / HEVC or H.266 / VVC) and determine the prediction type for each individual basic processing sub-unit.

[0066] For another example, in the prediction phase (an example of which is Figures 2A-2B ), the encoder can perform prediction operations at the level of basic processing sub-units (e.g., CUs). However, in some cases, the basic processing sub-units may still be too large to process. The encoder can further split the basic processing sub-units into smaller segments (e.g., called "prediction blocks" or "PBs" in H.265 / HEVC or H.266 / VVC), at which level prediction operations can be performed.

[0067] For another example, in the transformation phase (an example of which is Figures 2A-2B), the encoder can perform transform operations on the residual basic processing sub-units (e.g., CUs). However, in some cases, these basic processing sub-units may still be too large to process. The encoder can further split the basic processing sub-units into smaller segments (e.g., called "transform blocks" or "TBs" in H.265 / HEVC or H.266 / VVC), and transform operations can be performed at the level of the segments. It should be noted that the division scheme of the same basic processing sub-unit can be different in the prediction stage and the transform stage. For example, in H.265 / HEVC or H.266 / VVC, the prediction blocks and transform blocks of the same CU can have different sizes and numbers.

[0068] exist Figure 1 In the structure 110, the basic processing unit 112 is further divided into 3×3 basic processing sub-units, whose boundaries are shown by dotted lines. In different solutions, different basic processing units of the same image can be divided into different basic processing sub-units.

[0069] In some embodiments, in order to provide parallel processing and fault resilience for video encoding and decoding, a picture can be divided into multiple processing regions. This allows the encoding or decoding process for a certain region of the picture to be independent of information from any other region of the picture. In other words, each region of the picture can be processed independently. In this way, the codec can process different regions of the picture in parallel, thereby improving coding efficiency. In addition, when the data of one region is corrupted during processing or lost during network transmission, the codec can correctly encode or decode other regions of the same picture without relying on the damaged or lost data, thereby providing fault resilience. In some video coding standards, a picture can be divided into different types of regions. For example, H.265 / HEVC and H.266 / VVC provide two types of regions: "slices" and "tiles." It should also be noted that different pictures in the video sequence 100 can have different partitioning schemes for dividing the picture into regions.

[0070] For example, in Figure 1 In FIG, the structure 110 is divided into three regions 114, 116 and 118, whose boundaries are shown as solid lines inside the structure 110. Region 114 includes four basic processing units. Each of regions 116 and 118 includes six basic processing units. It should be noted that Figure 1 The basic processing units, basic processing sub-units, and regions of the structure 110 are merely examples, and the present application does not limit their implementation.

[0071] Figure 2A Schematic diagram of an example encoding process 200A consistent with an embodiment of the present invention is shown. For example, the encoding process 200A can be performed by an encoder. Figure 2A As shown, the encoder can encode the video sequence 202 into a video code stream 228 according to the encoding process 200A. Figure 1 The video sequence 100 in FIG. 200 may include a set of pictures (referred to as “original pictures”) arranged in time sequence. Figure 1 Based on the structure 110 in FIG. 1 , the encoder may divide each original picture of the video sequence 202 into basic processing units, basic processing sub-units, or regions for processing. In some embodiments, the encoder may perform encoding process 200A at the basic processing unit level for each original picture of the video sequence 202. For example, the encoder may perform encoding process 200A in an iterative manner, wherein the encoder may encode a basic processing unit in one iteration of process 200A. In some embodiments, the encoder may perform process 200A in parallel for regions (e.g., regions 114-118) of each original picture of the video sequence 202.

[0072] exist Figure 2A 200A, an encoder may input a basic processing unit (BPU) of an original picture of a video sequence 202 (referred to as an "original BPU") into a prediction stage 204 to generate prediction data 206 and a prediction BPU 208. The encoder may subtract the prediction BPU 208 from the original BPU to generate a residual BPU 210. The encoder may input the residual BPU 210 into a transform stage 212 and a quantization stage 214 to generate quantized transform coefficients 216. The encoder may input the prediction data 206 and the quantized transform coefficients 216 into a binary encoding stage 226 to generate a video code stream 228. Components 202, 204, 206, 208, 210, 212, 214, 216, 226, and 228 may be referred to as a "forward path." During process 200A, after the quantization stage 214, the encoder may input the quantized transform coefficients 216 into an inverse quantization stage 218 and an inverse transform stage 220 to generate a reconstructed residual BPU 222. The encoder can add the reconstructed residual BPU 222 to the prediction BPU 208 to generate a prediction reference 224, which is used in the prediction stage 204 for the next iteration of process 200A. Components 218, 220, 222, and 224 of process 200A can be referred to as a "reconstruction path." The reconstruction path can be used to ensure that both the encoder and decoder use the same reference data for prediction.

[0073] The encoder can iteratively perform process 200A to encode each original BPU of the original picture (in the forward path) and generate a prediction reference 224 for encoding the next original BPU of the original picture (in the reconstruction path). After encoding all the original BPUs of the original picture, the encoder can continue to encode the next picture in the video sequence 202.

[0074] Referring to process 200A, an encoder may receive a video sequence generated by a video capture device (e.g., a camera) 202. The term "receive," as used herein, may refer to any action of receiving, inputting, acquiring, retrieving, obtaining, reading, accessing, or in any way inputting data.

[0075] In the prediction phase 204, in the current iteration, the encoder may receive the original BPU and the prediction reference 224 and perform a prediction operation to generate the prediction data 206 and the predicted BPU 208. The prediction reference 224 may be generated from the reconstruction path of the previous iteration of the process 200A. The purpose of the prediction phase 204 is to reduce information redundancy by extracting the prediction data 206 that can be used to reconstruct the original BPU into the predicted BPU 208 from the prediction data 206 and the prediction reference 224.

[0076] Ideally, the predicted BPU 208 would be identical to the original BPU. However, due to non-ideal prediction and reconstruction operations, the predicted BPU 208 typically differs slightly from the original BPU. To account for this difference, after generating the predicted BPU 208, the encoder can subtract it from the original BPU to generate a residual BPU 210. For example, the encoder can subtract the value of the corresponding pixel of the original BPU from the value (e.g., grayscale value or RGB value) of the pixel corresponding to the predicted BPU 208. Each pixel of the residual BPU 210 can have a residual value that is the result of this subtraction between the corresponding pixel of the original BPU and the predicted BPU 208. Compared to the original BPU, the predicted data 206 and the residual BPU 210 can have fewer bits, but they can be used to reconstruct the original BPU without significantly reducing quality, thereby compressing the original BPU.

[0077] To further compress the residual BPU 210, in the transform stage 212, the encoder can reduce its spatial redundancy by decomposing the residual BPU 210 into a set of two-dimensional "basis patterns", each of which is associated with a "transform coefficient". The basis patterns can have the same size (e.g., the size of the residual BPU 210). Each basis pattern can represent a frequency-varying (e.g., frequency-varying) component of the residual BPU 210. No basis pattern can be reproduced from any combination (e.g., linear combination) of any other basis patterns. In other words, the decomposition can decompose the variation of the residual BPU 210 into the frequency domain. This decomposition is analogous to a discrete Fourier transform of a function, where the basis patterns are analogous to the basis functions (e.g., trigonometric functions) of the discrete Fourier transform, and the transform coefficients are analogous to the coefficients associated with the basis functions.

[0078] Different transform algorithms can use different base patterns. In the transform stage 212, various transform algorithms can be used, such as discrete cosine transform, discrete sine transform, etc. The transform in the transform stage 212 is reversible. That is, the encoder can restore the residual BPU 210 by performing the inverse operation of the transform (referred to as an "inverse transform"). For example, to restore the pixels of the residual BPU 210, the inverse transform may be multiplying the values ​​of the corresponding pixels of the base pattern by the corresponding correlation coefficients and adding the products to produce a weighted sum. For video coding standards, both the encoder and decoder can use the same transform algorithm (and therefore the same base pattern). Therefore, the encoder can only record the transform coefficients, and the decoder can reconstruct the residual BPU 210 from the transform coefficients without receiving the base pattern from the encoder. Compared to the residual BPU 210, the transform coefficients may have fewer bits, but they can be used to reconstruct the residual BPU 210 without significantly reducing quality. As a result, the residual BPU 210 is further compressed.

[0079] The encoder can further compress the transform coefficients during the quantization stage 214. During the transform process, different basis patterns can represent different frequencies of change (e.g., the frequency of brightness changes). Because the human eye is generally better at detecting low-frequency changes, the encoder can ignore information about high-frequency changes without significantly degrading decoding quality. For example, during the quantization stage 214, the encoder can generate quantized transform coefficients 216 by dividing each transform coefficient by an integer value (called a "quantization parameter") and rounding the quotient to the nearest integer. This operation can convert some transform coefficients of high-frequency basis patterns to zero, while transform coefficients of low-frequency basis patterns can be converted to smaller integers. The encoder can ignore zero-valued quantized transform coefficients 216, thereby further compressing the transform coefficients. The quantization process is also reversible, where the quantized transform coefficients 216 can be reconstructed into transform coefficients in the inverse operation of quantization (called "inverse quantization").

[0080] Because the encoder ignores the remainder of this division during rounding operations, the quantization stage 214 can be lossy. Generally, the quantization stage 214 can cause the greatest information loss in process 200A. The greater the information loss, the fewer bits may be required to quantize the transform coefficients 216. To achieve different levels of information loss, the encoder can use different values ​​for the quantization parameter or any other parameter in the quantization process.

[0081] In the binary encoding stage 226, the encoder may encode the prediction data 206 and the quantized transform coefficients 216 using a binary encoding technique (e.g., entropy coding, variable length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless or lossy compression algorithm). In some embodiments, in addition to the prediction data 206 and the quantized transform coefficients 216, the encoder may encode other information in the binary encoding stage 226, such as the prediction mode used in the prediction stage 204, the parameters of the prediction operation, the transform type in the transform stage 212, the parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., bitrate control parameters), etc. The encoder may use the output data of the binary encoding stage 226 to generate a video stream 228. In some embodiments, the video stream 228 may be further packaged for network transmission.

[0082] Referring to the reconstruction path of process 200A, at the inverse quantization stage 218, the encoder may perform inverse quantization on the quantized transform coefficients 216 to generate reconstructed transform coefficients. At the inverse transform stage 220, the encoder may generate a reconstructed residual BPU 222 based on the reconstructed transform coefficients. The encoder may add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a prediction reference 224 to be used in the next iteration of process 200A.

[0083] It should be noted that other variations of process 200A may also be used to encode video sequence 202. In some embodiments, the encoder may perform the stages of process 200A in a different order. In some embodiments, one or more stages of process 200A may be combined into a single stage. In some embodiments, a single stage of process 200A may be divided into multiple stages. For example, transform stage 212 and quantization stage 214 may be combined into a single stage. In some embodiments, process 200A may include additional stages. In some embodiments, process 200A may be omitted. Figure 2A one or more stages in a process.

[0084] Figure 2B A schematic diagram of another example encoding process 200B consistent with embodiments of the present application is shown. Process 200B can be modified from process 200A. For example, process 200B can be used by encoders compliant with hybrid video coding standards (e.g., the H.26x series). Compared to process 200A, the forward path of process 200B additionally includes a mode decision stage 230 and divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044. The reconstruction path of process 200B additionally includes a loop filter stage 232 and a buffer 234.

[0085] Generally, prediction techniques can be divided into two types: spatial prediction and temporal prediction. Spatial prediction (e.g., intra-picture prediction or "intra-frame prediction") can use pixels from one or more coded adjacent BPUs in the same picture to predict the current BPU. That is, the prediction reference 224 in spatial prediction can include adjacent BPUs. Spatial prediction can reduce the spatial redundancy inherent in the picture. Temporal prediction (e.g., inter-picture prediction or "inter-frame prediction") can use regions from one or more coded pictures to predict the current BPU. That is, the prediction reference 224 in temporal prediction can include a coded picture. Temporal prediction can reduce the temporal redundancy inherent in the picture.

[0086] Referring to process 200B, in the forward path, the encoder performs prediction operations in a spatial prediction stage 2042 and a temporal prediction stage 2044. For example, in the spatial prediction stage 2042, the encoder may perform intra-frame prediction. For the original BPU of the picture being encoded, the prediction reference 224 may include one or more neighboring BPUs that have been encoded (in the forward path) and reconstructed (in the reconstruction path) in the same picture. The encoder may generate a predicted BPU 208 by extrapolating the neighboring BPUs. Extrapolation techniques may include, for example, linear extrapolation or interpolation, polynomial extrapolation or interpolation, and the like. In some embodiments, the encoder may perform extrapolation at the pixel level, for example, by extrapolating the value of the corresponding pixel for each pixel of the predicted BPU 208. The neighboring BPUs used for extrapolation may be positioned relative to the original BPU in various directions, such as vertically (e.g., on top of the original BPU), horizontally (e.g., to the left of the original BPU), diagonally (e.g., below, below, above, or above the original BPU), or in any direction defined in the video coding standard being used. For intra prediction, the prediction data 206 may include, for example, the location (eg, coordinates) of the used neighboring BPUs, the size of the used neighboring BPUs, extrapolation parameters, the direction of the used neighboring BPUs relative to the original BPU, and the like.

[0087] For another example, during the temporal prediction stage 2044, the encoder may perform inter-frame prediction. For the original BPU of the current picture, the prediction reference 224 may include one or more pictures (referred to as "reference pictures") that have been encoded (in the forward path) and reconstructed (in the reconstruction path). In some embodiments, the reference pictures may be encoded and reconstructed using the BPU. For example, the encoder may add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a reconstructed BPU. When all reconstructed BPUs for the same picture are generated, the encoder may generate the reconstructed picture as the reference picture. The encoder may perform a "motion estimation" operation to search for a matching region within a range of reference pictures (referred to as a "search window"). The position of the search window in the reference picture may be determined based on the position of the original BPU in the current picture. For example, the search window may be centered in the reference picture at a location with the same coordinates as the original BPU in the current picture and may extend outward a predetermined distance. When the encoder identifies (e.g., using a pixel recursive algorithm, a block matching algorithm, etc.) a region similar to the original BPU in the search window, the encoder may determine such a region as a matching region. The matching region may have a different size (e.g., smaller, equal, larger, or different shape) than the original BPU. Figure 1 ), so it can be considered that the matching area "moves" to the position of the original BPU over time. The encoder can record the direction and distance of this movement as a "motion vector". Figure 1 When the encoder is performing a multi-image search of the reference picture 106), the encoder can search for matching areas and determine its associated motion vector for each reference picture. In some embodiments, the encoder can assign weights to the pixel values ​​of the matching areas of each matching reference picture.

[0088] Motion estimation can be used to identify various types of motion, such as translation, rotation, scaling, etc. For inter-frame prediction, the prediction data 206 may include, for example, the location (e.g., coordinates) of the matching region, the motion vector associated with the matching region, the number of reference pictures, the weights associated with the reference pictures, etc.

[0089] To generate the predicted BPU 208, the encoder may perform a "motion compensation" operation. Motion compensation may be used to reconstruct the predicted BPU 208 based on the prediction data 206 (e.g., motion vectors) and the prediction reference 224. For example, the encoder may move a matching area of ​​the reference picture according to the motion vector, where the encoder may predict the original BPU of the current picture. When multiple reference pictures are used (e.g., Figure 1For example, for a picture 106 in the reference picture, the encoder can move the matching region of the reference picture based on the corresponding motion vector and the average pixel value of the matching region. In some embodiments, if the encoder has assigned weights to the pixel values ​​of the matching region of each matching reference picture, the encoder can add the weighted sum of the pixel values ​​of the moved matching region.

[0090] In some embodiments, inter-frame prediction can be unidirectional or bidirectional. Unidirectional inter-frame prediction can use one or more reference pictures in the same temporal direction as the current picture. For example, Figure 1 The picture 104 in is a unidirectional inter-frame prediction picture, where the reference picture (i.e., picture 102) precedes the picture 104. Bidirectional inter-frame prediction can use one or more reference pictures in both temporal directions relative to the current picture. For example, Figure 1 Picture 106 in is a bidirectional inter-predicted picture, where the reference pictures (ie, pictures 104 and 108 ) are in both temporal directions relative to picture 104 .

[0091] Still referring to the forward path of process 200B, after the spatial prediction stage 2042 and the temporal prediction stage 2044, in the mode decision stage 230, the encoder can select a prediction mode (e.g., one of intra-frame prediction and inter-frame prediction) for the current iteration of process 200B. For example, the encoder can perform a rate-distortion optimization technique, in which the encoder can select a prediction mode based on the bit rate of a candidate prediction mode and the distortion of a reference picture reconstructed under the candidate prediction mode to minimize the value of a cost function. Based on the selected prediction mode, the encoder can generate a corresponding predicted BPU 208 and predicted data 206.

[0092] In the reconstruction path of process 200B, if intra-prediction mode is selected in the forward path, after generating prediction reference 224 (e.g., the current BPU in the current picture that has been encoded and reconstructed), the encoder can directly input prediction reference 224 into spatial prediction stage 2042 for later use (e.g., for extrapolating the next BPU of the current picture). If inter-prediction mode is selected in the forward path, after generating prediction reference 224 (e.g., the current picture in which all BPUs have been encoded and reconstructed), the encoder can input prediction reference 224 into loop filter stage 232, at which point the encoder can apply a loop filter to prediction reference 224 to reduce or eliminate distortion (e.g., blocking artifacts) introduced by inter-prediction. The encoder can apply various loop filter techniques in loop filter stage 232, such as deblocking, sample adaptive offset, adaptive loop filter, etc. The loop-filtered reference picture can be stored in buffer 234 (or "decoded picture buffer") for later use (e.g., as an inter-prediction reference picture for a future picture in the video sequence 202). The encoder may store one or more reference pictures in a buffer 234 for use in a temporal prediction stage 2044. In some embodiments, the encoder may encode parameters of the loop filter (e.g., loop filter strength) in a binary encoding stage 226, along with the quantized transform coefficients 216, prediction data 206, and other information.

[0093] Figure 3A A schematic diagram of an example decoding process 300A consistent with an embodiment of the present application is shown. Process 300A may be a decompression process corresponding to compression process 200A in FIG. In some embodiments, process 300A may be similar to the reconstruction path of process 200A. The decoder may decode video code stream 228 into video stream 304 according to process 300A. Video stream 304 may be very similar to video sequence 202. However, due to information loss during compression and decompression (e.g., Figures 2A-2B quantization stage 214 in), typically, the video stream 304 is different from the video sequence 202. Figures 2A-2B 200A and 200B, the decoder may perform process 300A at the basic processing unit (BPU) level for each picture encoded in the video stream 228. For example, the decoder may perform process 300A in an iterative manner, where the decoder may decode a basic processing unit in one iteration of decoding process 300A. In some embodiments, the decoder may perform process 300A in parallel for each region (e.g., regions 114-118) of each picture encoded in the video stream 228.

[0094] exist Figure 3AIn the process 300A, the decoder may input a portion of the video code stream 228 associated with a basic processing unit (referred to as a "coded BPU") of a coded picture into a binary decoding stage 302. In the binary decoding stage 302, the decoder may decode the portion into prediction data 206 and quantized transform coefficients 216. The decoder may input the quantized transform coefficients 216 into an inverse quantization stage 218 and an inverse transform stage 220 to generate a reconstructed residual BPU 222. The decoder may input the prediction data 206 into the prediction stage 204 to generate a predicted BPU 208. The decoder may add the reconstructed residual BPU 222 to the predicted BPU 208 to generate a predicted reference 224. In some embodiments, the predicted reference 224 may be stored in a buffer (e.g., a decoded picture buffer in computer memory). The decoder may input the predicted reference 224 into the prediction stage 204 for performing a prediction operation in the next iteration of the process 300A.

[0095] The decoder may iteratively perform process 300A to decode each coded BPU of a coded picture and generate a prediction reference 224 for decoding the next coded BPU of the coded picture. After decoding all coded BPUs of the coded picture, the decoder may output the picture to a video stream 304 for display and continue decoding the next coded picture in the video code stream 228.

[0096] In the binary decoding stage 302, the decoder may perform the inverse of the binary coding technique used by the encoder (e.g., entropy coding, variable length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless compression algorithm). In some embodiments, in addition to the prediction data 206 and the quantized transform coefficients 216, the decoder may also decode other information in the binary decoding stage 302, such as the prediction mode, parameters of the prediction operation, the transform type, the quantization parameter process (e.g., quantization parameter), encoder control parameters (e.g., bit rate control parameter), etc. In some embodiments, if the video stream 228 is transmitted over the network in packet form, the decoder may depacketize the video stream 228 before inputting it into the binary decoding stage 302.

[0097] Figure 3B A schematic diagram of another example decoding process 300B consistent with embodiments of the present application is shown. Process 300B can be modified from process 300A. For example, process 300B can be used by decoders compliant with hybrid video coding standards (e.g., the H.26x series). Compared to process 300A, process 300B additionally divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044, and additionally includes a loop filter stage 232 and a buffer 234.

[0098] In process 300B, for a coded basic processing unit (referred to as a "current BPU") of a coded picture being decoded (referred to as a "current picture"), the prediction data 206 decoded by the decoder from the binary decoding stage 302 may include various types of data, depending on what prediction mode the encoder used to encode the current BPU. For example, if the encoder used intra-frame prediction to encode the current BPU, the prediction data 206 may include a prediction mode indicator (e.g., an identification value) indicating intra-frame prediction, parameters of the intra-frame prediction operation, etc. The parameters of the intra-frame prediction operation may include, for example, the location (e.g., coordinates) of one or more neighboring BPUs used as a reference, the size of the neighboring BPUs, extrapolation parameters, the direction of the neighboring BPU relative to the original BPU, etc. For example, if the encoder uses inter-frame prediction to encode the current BPU, the prediction data 206 may include a prediction mode indicator indicating inter-frame prediction (e.g., an identification value), parameters of the inter-frame prediction operation, etc. The parameters of the inter-frame prediction operation may include, for example, the number of reference pictures associated with the current BPU, the weights associated with the reference pictures respectively, the positions (e.g., coordinates) of one or more matching regions in the corresponding reference pictures, one or more motion vectors associated with the matching regions respectively, etc.

[0099] Based on the prediction mode indicator, the decoder can decide whether to perform spatial prediction (e.g., intra prediction) in the spatial prediction stage 2042 or temporal prediction (e.g., inter prediction) in the temporal prediction stage 2044. The details of performing such spatial prediction or temporal prediction are described in Figure 2B After performing such spatial prediction or temporal prediction, the decoder may generate a predicted BPU 208. The decoder may add the predicted BPU 208 and the reconstructed residual BPU 222 to generate a prediction reference 224, as shown in FIG. Figure 3A Described in .

[0100] In process 300B, the decoder may input the prediction reference 224 to the spatial prediction stage 2042 or the temporal prediction stage 2044 to perform a prediction operation in the next iteration of process 300B. For example, if the current BPU is decoded using intra-frame prediction in the spatial prediction stage 2042, then after generating the prediction reference 224 (e.g., the decoded current BPU), the decoder may input the prediction reference 224 directly to the spatial prediction stage 2042 for later use (e.g., for extrapolating the next BPU of the current picture). If the current BPU is decoded using inter-frame prediction in the temporal prediction stage 2044, then after generating the prediction reference 224 (e.g., the reference picture in which all BPUs have been decoded), the encoder may input the prediction reference 224 to the loop filter stage 232 to reduce or eliminate distortion (e.g., blocking artifacts). The decoder may Figure 2B The loop filter is applied to the prediction reference 224 in the manner described in

[15] . The loop-filtered reference picture may be stored in a buffer 234 (e.g., a decoded picture buffer in a computer memory) for later use (e.g., as an inter-frame prediction reference picture for a future coded picture in the video codestream 228). The decoder may store one or more reference pictures in the buffer 234 for use in the temporal prediction stage 2044. In some embodiments, when the prediction mode indicator of the prediction data 206 indicates that inter-frame prediction was used to encode the current BPU, the prediction data may further include parameters of the loop filter (e.g., loop filter strength).

[0101] Figure 4 4 is a block diagram of an example apparatus 400 for encoding or decoding a video consistent with an embodiment of the present application. Figure 4 As shown, the device 400 may include a processor 402. When the processor 402 executes the instructions described herein, the device 400 may become a special-purpose machine for video encoding or decoding. The processor 402 may be any type of circuit capable of manipulating or processing information. For example, the processor 402 may include any number of central processing units (or "CPUs"), graphics processing units (or "GPUs"), neural processing units ("NPUs"), microcontroller units ("MCUs"), optical processors, programmable logic controllers, microcontrollers, microprocessors, digital signal processors, intellectual property (IP) cores, programmable logic arrays (PLAs), programmable array logic (PALs), general array logic (GALs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), systems on chip (SoCs), application-specific integrated circuits (ASICs), and the like. In some embodiments, the processor 402 may also be a group of processors grouped into a single logic controller. For example, as Figure 4 As shown, processor 402 may include multiple processors, including processor 402a, processor 402b, and processor 402n.

[0102] The device 400 may also include a memory 404 configured to store data (eg, a set of instructions, computer code, intermediate data, etc.). Figure 4As shown, the stored data may include program instructions (e.g., program instructions for implementing stages in process 200A, 200B, 300A, or 300B) and data for processing (e.g., video sequence 202, video code stream 228, or video stream 304). Processor 402 may access program instructions and data for processing (e.g., via bus 410) and execute program instructions to operate or manipulate the data for processing. Memory 404 may include a high-speed random access memory device or a non-volatile memory device. In some embodiments, memory 404 may include any number of random access memories (RAM), read-only memories (ROM), optical disks, magnetic disks, hard drives, solid-state drives, flash drives, secure digital (SD) cards, memory sticks, compact flash (CF) cards, etc. Memory 404 may also be a group of memories grouped into a single logical component ( Figure 4 not shown).

[0103] The bus 410 may be a communication device that transmits data between components inside the apparatus 400 , such as an internal bus (eg, a CPU-memory bus), an external bus (eg, a Universal Serial Bus port, a Peripheral Component Interconnect Express port), and the like.

[0104] For ease of explanation and to avoid ambiguity, the processor 402 and other data processing circuitry are collectively referred to as "data processing circuitry" in this application. The data processing circuitry may be implemented entirely in hardware, or in a combination of software, hardware, or firmware. Furthermore, the data processing circuitry may be a single standalone module, or may be fully or partially integrated into any other component of the apparatus 400.

[0105] The device 400 may also include a network interface 406 to provide wired or wireless communications with a network (e.g., the Internet, an intranet, a local area network, a mobile communication network, etc.). In some embodiments, the network interface 406 may include any number of network interface controllers (NICs), radio frequency (RF) modules, repeaters, transceivers, modems, routers, gateways, any combination of wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication ("NFC") adapters, cellular network chips, etc.

[0106] In some embodiments, the apparatus 400 may optionally further include a peripheral interface 408 to provide a connection to one or more peripheral devices. Figure 4 As shown, peripheral devices may include, but are not limited to, a cursor control device (e.g., a mouse, touchpad, or touch screen), a keyboard, a display (e.g., a cathode ray tube display, a liquid crystal display, or a light emitting diode display), a video input device (e.g., a camera or a communication input interface coupled to a video archive), etc.

[0107] It should be noted that a video codec (e.g., a codec that performs processes 200A, 200B, 300A, or 300B) can be implemented as any combination of software or hardware modules in apparatus 400. For example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more software modules of apparatus 400, such as program instructions that can be loaded into memory 404. For another example, some or all stages of processes 200A, 200B, 300A, or 300B can be implemented as one or more hardware modules of apparatus 400, such as dedicated data processing circuits (e.g., FPGAs, ASICs, NPUs, etc.).

[0108] One of the key requirements of the VVC standard is to provide video conferencing applications with the ability to tolerate network and device diversity and to quickly adapt to changing network conditions, including quickly reducing the encoding bitrate when network conditions deteriorate and quickly increasing the video quality when network conditions improve. The expected video quality may range from very low to very high. The standard should also support fast representation switching for adaptive streaming services that provide multiple representations of the same content, each with different properties (such as spatial resolution or sampling bit depth). When switching from one representation to another (for example, from one resolution to another), the standard should be able to use an effective prediction structure without compromising the ability to switch quickly and seamlessly.

[0109] The purpose of Adaptive Resolution Change (ARC) is to allow a stream to change spatial resolution between coded pictures in the same video sequence, such as if a new IDR frame is needed in a scalable video codec, and if multiple layers are needed. Instead, at the switching point, the resolution of the picture changes and prediction is made from reference pictures of the same resolution (if available) and reference pictures of a different resolution. If the reference pictures have different resolutions, the reference pictures are resampled, such as Figure 5 For example, Figure 5 As shown, the resolution of reference picture 506 is the same as the resolution of current picture 502, while the resolution of reference pictures 504 and 508 is different from the resolution of current picture 502. After reference pictures 504 and 508 are resampled to the resolution of current picture 502, motion compensated prediction can be performed on these reference pictures. Therefore, adaptive resolution change (ARC) is sometimes also called reference picture resampling (RPR), and these two terms are used interchangeably in this application.

[0110] When the resolution of the reference frame is different from the resolution of the current frame, one approach to generating a motion compensated prediction signal is picture-based resampling, where the reference picture is first resampled to the same resolution as the current picture, and an existing motion compensation process with motion vectors can be applied. The motion vectors can be scaled (if sent in units before the resampling is applied) or not (if sent in units after the resampling is applied). For picture-based resampling, and especially for downsampling of the reference picture (i.e., the resolution of the reference picture is larger than the resolution of the current picture), information may be lost in the reference resampling step before motion compensated interpolation, because downsampling is typically achieved by low-pass filtering and decimation).

[0111] Another approach is block-based resampling, which performs resampling at the block level. This is done by checking the reference pictures used by the current block and, if one or both of the reference pictures have a different resolution than the current picture, resampling is performed in conjunction with a sub-pixel motion compensated interpolation process.

[0112] In block-based resampling, resampling and motion-compensated interpolation are combined into a single filtering operation, which can reduce this information loss. For example, consider the case where the motion vector of the current block has half-pixel precision in one dimension (e.g., the horizontal dimension) and the reference picture is twice as wide as the current picture. In this case, compared to picture-level resampling, which reduces the width of the reference picture by half to match the width of the current picture and then performs half-pixel motion interpolation, the block-based resampling method can directly use odd-numbered positions in the reference picture as reference blocks with half-pixel precision. At the 15th JVET meeting, VVC adopted a block-based ARC resampling method, in which motion-compensated (MC) interpolation and reference resampling are combined and performed in a single filter step. In VVC draft 6, the existing filter used for MC interpolation without reference resampling is reused for MC interpolation with reference resampling. The same filter is used for reference upsampling and reference downsampling. Details on filter selection are described below.

[0113] For the luminance component, if the half-pixel AMVR mode is selected and the interpolation position is half-pixel, a 6-tap filter [3,9,20,20,9,3] is used. If the motion compensation block size is 4×4, the following is used: Figure 6 The 6-tap filter shown in Table 6 is used. Otherwise, use Figure 7 The 8-tap filter shown in Table 7.

[0114] For chroma components, use Figure 8 The 4-tap filter shown in Table 8.

[0115] In VVC, the same filter is used for MC interpolation without reference resampling and MC interpolation with reference resampling. Although the VVC motion compensated interpolation filter (MCIF) is designed based on DCT upsampling, it may not be appropriate to use it as a one-step filter combined with reference downsampling and MC interpolation. For example, for phase 0 filtering (e.g., scaling motion vectors to integers), the VVC 8-tap MCIF coefficients are [0,0,0,64,0,0,0,0], which means that the prediction samples are copied directly from the reference samples. Although MC interpolation may not be a problem in the case of no reference downsampling or reference upsampling, in the case of reference downsampling, this may cause aliasing artifacts due to the lack of a low-pass filter before decimation.

[0116] The present application provides a method for MC interpolation of reference downsampling using a cosine window function difference filter.

[0117] The window function difference filter is a bandpass filter that separates one frequency band from other frequency bands. The window function difference filter is a low-pass filter whose frequency response allows all frequencies below the cutoff frequency to pass when the amplitude is not 1, and stops all frequencies above the cutoff frequency when the amplitude is 0, such as Figure 9 shown.

[0118] The filter kernel, also known as the filter's impulse response, is obtained by taking the inverse Fourier transform of the frequency response of an ideal low-pass filter. The impulse response of a low-pass filter is a general form of an interpolation function (sinc function) based on the following formula (1). Where fc is the cutoff frequency, whose value is in [0,1], and r is the downsampling ratio, i.e., 1.5 for 1.5:1 downsampling and 2 for 2:1 downsampling. The interpolation function is defined based on the following formula (2).

[0119] The interpolation function is infinite. To make the filter kernel have a finite length, a window function is used to truncate the filter kernel to point L. To obtain a smooth conical curve, a cosine window function based on the following formula (3) is used.

[0120] The kernel of the cosine window function difference filter is the product of the ideal response function h(n) and the cosine window function w(n), which is obtained by the following formula (4).

[0121] The window function difference kernel has two parameters to choose from: the cutoff frequency fc and the kernel length L. By adjusting the values ​​of L and fc, the desired filter response can be obtained. For example, for the downsampling filter used in the Scalable HEVC Test Model (SHM), fc = 0.9 and L = 13.

[0122] The filter coefficients obtained in formula (4) are real numbers. Applying the filter is equivalent to calculating the weighted average of the reference samples, with the weights being the filter coefficients. For efficient computation in a digital computer or hardware, the coefficients are normalized, multiplied by a scalar, and rounded to integers so that the sum of the coefficients is equal to 2^N, where N is an integer. The filtered samples are divided by 2^N (equivalent to a right shift of N bits). For example, in VVC draft 6, the sum of the interpolation filter coefficients is 64.

[0123] In some embodiments, for luma and chroma components, a downsampling filter can be used in SHM for VVC motion compensation interpolation with reference downsampling, and an existing MCIF can be used for motion compensation interpolation with reference upsampling. When the kernel length L = 13, the first coefficient is very small and rounded to zero, and the filter length can be reduced to 12 without affecting the filter performance.

[0124] For example, Figure 10 Table 10 and Figure 11 Table 11 shows the filter coefficients for 2:1 downsampling and 1.5:1 downsampling, respectively.

[0125] Besides the values ​​of the coefficients, there are some other differences between the design of the SHM filter and the existing MCIF.

[0126] As a first difference, the SHM filter needs to filter at both integer and fractional sample positions, while the MCIF filter only needs to filter at fractional sample positions. Figure 12 Table 12 describes an example of modification of the interpolation filtering process for VVC draft 6 luma samples for the reference downsampling case.

[0127] under Figure 13 An example of modification of the interpolation filtering process for VVC draft 6 chroma samples for the reference downsampling case is described in Table 13 of .

[0128] As a second difference, for the SHM filter, the sum of the filter coefficients is 128, while for the existing MCIF, the sum of the filter coefficients is 64. In VVC draft 6, in order to reduce losses due to rounding errors, the intermediate prediction signal is kept at a higher precision (represented at a higher bit depth) than the output signal. The precision of the intermediate signal is called internal precision. In some embodiments, in order to keep the internal precision the same as in VVC draft 6, the output of the SHM filter needs to be right-shifted by an additional 1 bit compared to using the existing MCIF. Figure 14 An example of modification of the interpolation filtering process for luma samples of VVC draft 6 for the reference downsampling case is shown in Table 14 of FIG.

[0129] under Figure 15 An example of modification of the interpolation filtering process for chroma samples of VVC draft 6 for the reference downsampling case is shown in Table 15 of .

[0130] According to some embodiments, the internal precision is increased by 1 bit and a plus 1 bit right shift may be used to convert the internal precision to the output precision. Figure 16 An example of modification of the interpolation filtering process for luma samples of VVC draft 6 for the reference downsampling case is shown in Table 16 of .

[0131] under Figure 17 An example of modification of the interpolation filtering process for chroma samples of VVC draft 6 for the reference downsampling case is shown in Table 17 of .

[0132] As a third difference, the SHM filter has 12 taps. Therefore, to generate the interpolated sample, 11 adjacent samples are required (5 to the left, 6 to the right, or 5 above or 6 below). Compared to MCIF, more adjacent samples are extracted. In VVC draft 6, the chroma mv accuracy is 1 / 32. However, the SHM filter has only 16 phases. Therefore, the chroma mv can be rounded to 1 / 16 and used as a reference downsampling. This can be achieved by shifting the last 5 bits of the chroma mv right by 1 bit. Figure 18 A modified example of the calculation of the VVC draft 6 chroma fractional sample positions for the reference downsampling case is shown in Table 18 of .

[0133] According to some embodiments, to align with the existing MCIF design in the VVC draft, it is proposed to use an 8-tap cosine window difference filter. The filter coefficients can be derived by setting L = 9 in the cosine window difference function in equation (4). The sum of the filter coefficients can be set to 64 to further align with the existing MCIF filter. In the disclosed embodiments, the filter for the complementary phase can be symmetric (e.g., the filter coefficients of the complementary phase are inversely proportional) or asymmetric. Figure 19 Table 19 and Figure 20 Table 20 shows example filter coefficients for 2:1 and 1.5:1 ratios, respectively.

[0134] According to some embodiments, in order to accommodate the 1 / 32 sampling accuracy of the chroma components, a 32-phase cosine window function difference filter bank may be used in the reference downsampled chroma motion compensation interpolation. Figure 21 The following table 21 and Figure 22 Table 22 of FIG2 gives examples of filter coefficients for ratios of 2:1 and 1.5:1, respectively.

[0135] According to some embodiments, for a 4×4 luma block, a 6-tap cosine window function difference filter may be used for MC interpolation of reference downsampling. Figure 23 Table 23 and Figure 24 Table 24 of FIG24 gives examples of filter coefficients with a ratio of 2:1 and 1.5:1, respectively.

[0136] In VVC, different interpolation filters are used in different motion compensation scenarios. For example, in normal motion compensation, an 8-tap DCT-based interpolation filter is used. In motion compensation for 4×4 sub-blocks, a 6-tap DCT-based interpolation filter is used. In motion compensation with an MVD precision of 1 / 2 and an MVD phase of 1 / 2, different 6-tap filters are used. In chroma motion compensation, a 4-tap DCT-based interpolation filter is used.

[0137] In some embodiments, when the reference picture has a higher resolution than the current coded picture, the DCT-based interpolation filter can be replaced by a cosine window function difference filter with the same filter length. In these embodiments, the 8-tap DCT-based interpolation filter can be replaced by an 8-tap cosine window function difference filter for conventional motion compensation; the 6-tap DCT-based interpolation filter can be replaced by a 6-tap cosine window function difference filter for 4×4 sub-block motion compensation. The 4-tap DCT-based interpolation filter can be replaced by a 4-tap cosine window function difference filter for chroma motion compensation. In addition, when the MVD accuracy is 1 / 2 and the phase is 1 / 2, the 6-tap DCT-based interpolation filter can be used for motion compensation. The filter selection may depend on the ratio between the reference picture resolution and the current picture resolution. Exemplary 8-tap, 6-tap, and 4-tap cosine window function difference filters for 2:1 and 1.5:1 downsampling rates are shown in Figure 2. Figure 19-24 As shown in Table 19-24. Figure 25-30 The filters shown in Tables 25-30 are symmetrical in complementary phase.

[0138] The filter coefficients obtained in formula (4) are real values. After normalization and scaling, the filter coefficients can be rounded to integers. The sum of the filter coefficients, also known as the filter gain, represents the filter accuracy. In digital processing, the filter gain is usually set to 2^N. Due to rounding operations, the sum of the filter coefficients may not be equal to the filter gain (2^N). The filter coefficients are additionally adjusted so that the sum of the filter coefficients is equal to the gain (2^N). In some embodiments, the above adjustment includes determining an appropriate rounding direction, which may include rounding up or rounding down. Therefore, the adjusted filter coefficients can be added to the filter gain to minimize / maximize the cost function. In some embodiments, the cost function can be the sum of absolute differences (SAD) or the sum of squared errors (SSE) between the rounded filter coefficients and the true filter coefficients before rounding. In some embodiments, the cost function can be the SAD or SSE between the frequency response of the rounded filter coefficients and the frequency response of a reference value. The reference value can be the true filter coefficient before rounding, or the frequency response of an ideal filter, or the frequency response of a filter that is longer than the designed filter. For example, when designing an 8-tap filter, the reference may be a 12-tap filter. Based on the above rounding method, another set of cosine window function difference filters with lengths of 8 taps, 6 taps, and 4 taps are as follows Figures 31-36 As shown in Tables 31-36.

[0139] Since the source image downsampling and reconstructed image upsampling processes in adaptive resolution change applications may not be standardized in the standard, this means that users can use any resampling filter. Since the filter selected by the user may not match the reference resampling filter, the performance may be degraded due to filter mismatch. In some disclosed embodiments, the coefficients of the reference resampling filter are marked in the bitstream. And the encoder / decoder can use the user-defined filter for reference resampling. The signaling of the filter coefficients is as follows: Figure 37 and 38 As shown in Tables 37-38.

[0140] The syntax structures illustrated in Tables 37 and 38 may be presented in a high-level syntax, such as a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a slice header, and the like.

[0141] The use of user-defined RPR filters and the presence of the above-described exemplary syntax structures may be controlled by flags in the high-level syntax, including SPS, PPS, APS, slice header, and the like.

[0142] Consistent with the disclosed embodiments, hardware implementations may be more efficient when one or more labeled filters use fixed filter lengths and filter norms. Thus, in some embodiments, the filter norm and / or filter taps may not be labeled but rather implicit (e.g., implicitly as default values ​​defined in the standard). Filter specifications, filter taps, filter accuracy, or maximum bit depth may be part of an encoder's conformance requirements.

[0143] In some embodiments, the encoder may send a Supplemental Enhancement Information (SEI) message to suggest which non-standard upsampling filter the receiving device should use after decoding the picture. Such SEI messages are optional. However, through the SEI message, the decoder that uses the suggested upsampling filter can achieve better video quality. Table 31 ( Figure 31 The filter coefficients() syntax structure in ) can be used to specify the upsampling filter proposed in such SEI messages.

[0144] Figure 39 3900 is a flowchart of an exemplary method for processing video content consistent with an embodiment of the present application. In some embodiments, the method 3900 can be performed by a codec (e.g., using Figures 2A-2B The encoding process 200A or 200B of the encoder, or using Figures 3A-3B For example, a codec may be implemented as one or more software or hardware components of a device (e.g., device 400) for encoding or transcoding a video sequence. In some embodiments, the video sequence may be an uncompressed video sequence (e.g., video sequence 202) or a decoded compressed video sequence (e.g., video stream 304). In some embodiments, the video sequence may be a monitoring video sequence that may be processed by a monitoring device (e.g., processor 402) associated with a processor of the device. Figure 4 The video sequence may be captured by a video input device (e.g., a video input device). The video sequence may include multiple pictures. The device may perform method 3900 at the picture level. For example, in method 3900, the device may process one picture at a time. For example, in method 3900, the device may process multiple pictures at a time. Method 3900 may include the following steps.

[0145] In step 3902, for a target picture and a reference picture having different resolutions, a bandpass filter may be applied to the reference picture to perform motion compensated interpolation using reference downsampling to generate a reference block.

[0146] The bandpass filter can be a cosine window function difference filter generated based on an ideal low-pass filter and a window function. For example, the kernel function f(n) of the cosine window function difference filter can be the product of the ideal low-pass filter h(n) and the window function w(n), which can be obtained based on formula (4).

[0147] therefore, Where fc is the cutoff frequency of the cosine window function difference filter f(n), L is the kernel length, and r is the downsampling rate of the reference downsampling.

[0148] The filter coefficients of the cosine window function difference filter may be rounded before application. Figure 40 3900 is a flowchart of an exemplary method 4000 for rounding filter coefficients of a cosine window function difference filter consistent with an embodiment of the present application. It is understood that method 4000 can be implemented independently or as part of method 3900. Method 4000 may include the following steps.

[0149] In step 4002, the true filter coefficients of the bandpass filter may be obtained.

[0150] In step 4004, multiple rounding directions of the real filter coefficients may be determined separately.

[0151] In step 4006, a plurality of combinations of rounded filter coefficients may be generated by rounding the real filter coefficients according to a plurality of rounding directions, respectively.

[0152] In step 4008, a combination of rounded filter coefficients that minimizes or maximizes the cost function can be selected from multiple combinations. In some embodiments, the cost function can be associated with the rounded filter coefficients and a reference value. The reference value can be the true filter coefficients of the bandpass filter before rounding, the frequency response of an ideal filter, or the frequency response of a filter longer than the bandpass filter. For example, when the reference value can be the true filter coefficients of the bandpass filter before rounding, the cost function can be the sum of absolute differences (SAD) or the sum of squared errors (SSE) between the rounded filter coefficients before rounding and the true filter coefficients. As another example, when the reference value is the frequency response of an ideal filter, the cost function can be the SAD or SSE between the frequency response of the rounded filter coefficients and the frequency response of the ideal filter. As described above, the reference value can also be the frequency response of a filter longer than the bandpass filter. For example, if the bandpass filter is an 8-tap filter, the reference can be a 12-tap filter, which is longer than an 8-tap bandpass filter.

[0153] The selected combination of rounding filter coefficients for the bandpass filter may be signaled in at least one of a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), and a slice header, along with a syntax structure associated with the filter coefficients.

[0154] It should be understood that, as shown in Table 19-36 ( Figure 19-36 ), the cosine window function difference filter is an 8-tap filter, a 6-tap filter, or a 4-tap filter. In some embodiments, based on the ratio between the resolution of the reference picture and the resolution of the target picture, the bandpass filter can be determined as one of an 8-tap filter, a 6-tap filter, and a 4-tap filter.

[0155] When applying a bandpass filter to a reference picture, luma samples or chroma samples may be obtained at fractional sample positions. Filter coefficients for the obtained luma samples or chroma samples may be determined by referring to a lookup table (e.g., Tables 19-36) using fractional sample positions.

[0156] Back to Figure 39 In step 3904, the block of the target picture may be processed using the reference block. For example, the block of the target picture may be encoded or decoded using the reference block.

[0157] In some embodiments, a non-transitory computer-readable storage medium including instructions is also provided, and these instructions can be executed by a device for performing the above method (e.g., an encoder and a decoder disclosed in the present application). Common forms of non-transitory media include, for example, a floppy disk, a floppy disk, a hard disk, a solid-state drive, a tape or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with a hole pattern, RAM, PROM and EPROM, FLASH-EPROM or any other flash memory, NVRAM, a cache, a register, any other memory chip or cartridge memory, and a network version of the same. The device may include one or more processors (CPU), an input / output interface, a network interface, and / or a memory.

[0158] The above embodiments can be further described using the following terms: 1. A computer-implemented method for performing motion-compensated interpolation, comprising: For a target picture and a reference picture having different resolutions, applying a bandpass filter to the reference picture, performing motion compensated interpolation by reference downsampling to generate a reference block; and A block of the target picture is processed using the reference block. 2. The method according to clause 1, wherein the bandpass filter is a cosine window function difference filter generated based on an ideal low-pass filter and a window function. 3. The method according to clause 2, wherein the cosine window function difference filter is based on a kernel function in Where fc is the cutoff frequency of the cosine window function difference filter, L is the kernel length, and r is the downsampling rate of the reference downsampling. 4. The method according to clause 1, further comprising: Obtaining true filter coefficients of the bandpass filter; respectively determining a plurality of rounding directions of the real filter coefficients; generating a plurality of combinations of rounded filter coefficients by rounding the real filter coefficients according to the plurality of rounding directions; and Among the plurality of combinations, a combination of rounding filter coefficients that minimizes or maximizes the cost function is selected. 5. The method of clause 4, wherein the cost function is associated with rounding filter coefficients and a reference value. 6. The method according to clause 5, wherein: The reference value is a true filter coefficient of the bandpass filter before rounding, a frequency response of an ideal filter, or a frequency response of a filter longer than the bandpass filter length. 7. The method according to clause 2, wherein the cosine window function difference filter is an 8-tap filter, a 6-tap filter or a 4-tap filter. 8. The method of clause 1, wherein applying a bandpass filter to the reference picture comprises: Gets a luma or chroma sample at a fractional sample position. 9. The method according to clause 1, further comprising: The bandpass filter is determined to be one of an 8-tap filter, a 6-tap filter, and a 4-tap filter based on a ratio between a resolution of a reference picture and a resolution of a target picture. 10. The method according to clause 4, further comprising: A selected combination of rounding filter coefficients of a bandpass filter is marked in at least one of a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), and a slice header. 11. A system for performing motion compensated interpolation, comprising: a memory storing a set of instructions; and at least one processor configured to execute a set of instructions to cause the system to: For a target picture and a reference picture having different resolutions, applying a bandpass filter to the reference picture, performing motion compensated interpolation by reference downsampling to generate a reference block; and A block of the target picture is processed using the reference block. 12. The system of clause 11, wherein the bandpass filter is a cosine window function difference filter generated based on an ideal low-pass filter and a window function. 13. The system according to clause 12, wherein the cosine window function difference filter is based on a kernel function in Where fc is the cutoff frequency of the cosine window function difference filter, L is the kernel length, and r is the downsampling rate of the reference downsampling. 14. The system of clause 11, wherein the at least one processor is configured to execute a set of instructions to cause the system to further: Obtaining true filter coefficients of the bandpass filter; respectively determining a plurality of rounding directions of the real filter coefficients; generating a plurality of combinations of rounded filter coefficients by rounding the real filter coefficients according to the plurality of rounding directions; and Among the plurality of combinations, a combination of rounding filter coefficients that minimizes or maximizes the cost function is selected. 15. The system of clause 14, wherein the cost function is associated with rounding filter coefficients and a reference value. 16. A system according to clause 15, wherein The reference value is the actual filter coefficient of the bandpass filter before rounding, the frequency response of an ideal filter, or the frequency response of a filter longer than the bandpass filter length. 17. The system of clause 12, wherein the cosine window function difference filter is an 8-tap filter, a 6-tap filter, or a 4-tap filter. 18. The system of clause 11, wherein applying a bandpass filter to the reference picture comprises: Gets a luma or chroma sample at a fractional sample position. 19. The system of clause 11, wherein at least one processor is configured to execute a set of instructions to cause the system to further: The bandpass filter is determined to be one of an 8-tap filter, a 6-tap filter, and a 4-tap filter based on a ratio between a resolution of a reference picture and a resolution of a target picture. 20. A non-transitory computer-readable medium storing a set of instructions executable by at least one processor of a computer system to cause the computer system to perform a method for processing video content, the method comprising: In response to a target picture and a reference picture having different resolutions, applying a bandpass filter to the reference picture, performing motion compensated interpolation by reference downsampling to generate a reference block; and A block of the target picture is processed using the reference block.

[0159] It should be noted that the relational terms such as "first" and "second" in this document are used only to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or order between these entities or operations. In addition, the words "including", "having", "comprising" and "including" and other similar forms have the same meaning and are open-ended, as the one or more items following any of these words are not intended to be an exhaustive list of such items or to be limited to the listed items.

[0160] As used herein, unless expressly stated otherwise, the term "or" encompasses all possible combinations unless not feasible. For example, if it is stated that a component can include A or B, then unless expressly stated otherwise or not feasible, the component can include A, or B, or A and B. As a second example, if it is stated that a component can include A, B, or C, then unless expressly stated otherwise or not feasible, the component can include A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0161] It will be appreciated that the above embodiments may be implemented by hardware or software (program code) or a combination of hardware and software. If implemented by software, it may be stored in the above-mentioned computer-readable medium. When executed by a processor, the software may execute the disclosed method. The computing units and other functional units described in this application may be implemented by hardware, by software, or by a combination of hardware and software. Those of ordinary skill in the art may also appreciate that the above-mentioned multiple modules / units may be combined into one module / unit, and each of the above-mentioned modules / units may be further divided into multiple submodules / subunits.

[0162] In the foregoing description, embodiments have been described with reference to many specific details, which may vary depending on the implementation. Certain adjustments and modifications may be made to the described embodiments. Other embodiments will be apparent to those skilled in the art, taking into account the specifications and practices of the present application disclosed herein. The foregoing description and embodiments are intended to be examples only, with the true scope and spirit of the present application being indicated by the claims. The order of steps shown in the figures is also intended to be for illustrative purposes only and is not intended to be limited to any particular order of steps. Therefore, it will be understood by those skilled in the art that these steps may be performed in different orders while implementing the same method.

[0163] In the drawings and the specification, exemplary embodiments have been disclosed. However, many variations and modifications may be made to these embodiments. Therefore, although specific terms are used, they are used in a general and descriptive sense only and not for limitation purposes.

Claims

1. A computer-implemented method for processing video content, comprising: encoding the image block by applying one or more filters to a reference image; wherein applying the one or more filters generates chroma samples at fractional sample positions, and wherein the one or more filters comprises a 4-tap filter having a plurality of coefficients for each fractional sample position, wherein the coefficients associated with a fractional sample position are {17, 30, 17, 0}.

2. The method according to claim 1, characterized in that The plurality of coefficients are signaled in at least one of a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), and a slice header.

3. The method according to claim 1, characterized in that Applying the one or more filters includes selecting the one or more filters based on a sampling rate of the reference image.

4. A computer-implemented method for processing video content, comprising: decoding the image block by applying one or more filters to a reference image; wherein applying the one or more filters generates chroma samples at fractional sample positions, and wherein, The one or more filters include a 4-tap filter having a plurality of coefficients for each fractional sample position, wherein the coefficients associated with a fractional sample position are {17, 30, 17, 0}.

5. The method according to claim 4, characterized in that The plurality of coefficients are signaled in at least one of a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), and a slice header.

6. The method according to claim 4, characterized in that Applying the one or more filters includes selecting the one or more filters based on a sampling rate of the reference image.

7. A non-transitory computer-readable storage medium storing computer instructions and a bitstream, wherein the computer instructions are executed by a processor to implement the following method to generate the bitstream, the method comprising: encoding the image block by applying one or more filters to a reference image; wherein applying the one or more filters generates chroma samples at fractional sample positions, and wherein the one or more filters comprises a 4-tap filter having a plurality of coefficients for each fractional sample position, wherein the coefficients associated with a fractional sample position are {17, 30, 17, 0}.

8. The non-transitory computer-readable storage medium according to claim 7, wherein: The method further comprises: The plurality of coefficients are signaled in at least one of a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), and a slice header.

9. The non-transitory computer-readable storage medium according to claim 7, wherein: The method further comprises: Applying the one or more filters includes selecting the one or more filters based on a sampling rate of the reference image.

Images