Planar mode improvement for intra prediction
By determining and applying reference sample filters for different video block sizes in video processing, the problem of intra prediction is solved, and more efficient video data compression and decoding performance is achieved.
Patent Information
- Application Number
- CN202380029563.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-11-22
- Filing Date
- 2023-03-22
- Publication Date
- 2024-11-08
AI Technical Summary
The existing video encoding and decoding technology is difficult to effectively utilize the size changes of video blocks during intra prediction, resulting in low prediction efficiency and complex design of reference sample filters, which affects compression performance.
The processor is used to determine the reference sample filter based on the video block size and apply it to the received reference sample for filtering. It is used to perform intra prediction and supports multiple reference sample filters of different video block sizes to improve prediction accuracy. and efficiency.
The accuracy and efficiency of intra-frame prediction of video blocks are improved, the changes in different video block sizes are adapted to the compression performance of video data is optimized, and the complexity of encoding and decoding is reduced.
Smart Images

Figure CN118923116A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on and claims the benefit of priority of U.S. Provisional Application No. 63 / 322,478, filed on March 22, 2022. This application is a continuation-in-part application of PCT Application No. PCT / US22 / 50696, filed on November 22, 2022, which claims the benefit of priority of U.S. Provisional Application No. 63 / 281,822, filed on November 22, 2021. The contents of all of the above applications are incorporated herein by reference in their entirety. Technical Field
[0003] The present application relates to video coding and compression, and more particularly to a video processing apparatus and method for intra-frame prediction with reference sample filtering. Background Art
[0004] Digital video is supported by various electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc. Electronic devices send and receive or otherwise communicate digital video data across a communication network, and / or store digital video data on a storage device. Due to the limited bandwidth capacity of the communication network and the limited storage resources of the storage device, video codecs can be used to compress video data according to one or more video codec standards before communicating or storing video data. For example, video codec standards include Versatile Video Coding (VVC), Joint Exploration test Model (JEM), High-Efficiency Video Coding (HEVC / H.265), Advanced Video Coding (AVC / H.264), Moving Picture Expert Group (MPEG) codecs, etc. Video codecs typically use prediction methods that exploit the redundancy inherent in video data (eg, inter-frame prediction, intra-frame prediction, etc.) Video codecs aim to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation of video quality. Summary of the invention
[0005] Embodiments of the present disclosure provide a video processing apparatus and method for performing intra-frame prediction on a video block. The video processing method may include: receiving, by a processor, a reference sample from a video frame of a video including a video block. The video processing method may also include: determining, by the processor, a reference sample filter based on the size of each video block. The reference sample filter is among a plurality of reference sample filters derived for different video block sizes. The video processing method may also include: applying, by the processor, the determined reference sample filter to the received reference sample. The video processing method may further include: performing intra-frame prediction on the video block using the filtered reference sample by the processor.
[0006] An embodiment of the present disclosure also provides a video processing device for performing intra-frame prediction on a video block. The video processing device may include a memory and one or more processors coupled to the memory. The one or more processors may be configured to receive reference samples from a video frame of a video including a video block. The one or more processors may also be configured to determine a reference sample filter based on the size of each video block. The reference sample filter is among a plurality of reference sample filters derived for different video block sizes. The one or more processors may also be configured to apply the determined reference sample filter to the received reference sample. The one or more processors may also be configured to perform intra-frame prediction on the video block using the filtered reference sample.
[0007] An embodiment of the present disclosure also provides a non-transitory computer-readable storage medium having instructions stored therein, which, when executed by one or more processors, causes the one or more processors to perform a video processing method for performing intra-frame prediction on a video block. The video processing method may include: receiving reference samples from a video frame of a video including a video block. The video processing method may also include: determining a reference sample filter based on the size of each video block. The reference sample filter is among a plurality of reference sample filters derived for different video block sizes. The video processing method may also include: applying the determined reference sample filter to the received reference sample. The video processing method may further include: performing intra-frame prediction on the video block using the filtered reference sample.
[0008] It is to be understood that both the foregoing general description and the following detailed description are exemplary only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0010] Figure 1is a block diagram illustrating an exemplary system for encoding and decoding video blocks according to some embodiments of the present disclosure.
[0011] Figure 2 is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.
[0012] Figure 3 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.
[0013] FIG. 4A to FIG. 4E is a graphical representation showing how a frame may be recursively divided into multiple video blocks of different sizes and shapes according to some embodiments of the present disclosure.
[0014] Figure 5 A diagram illustrating intra mode according to some embodiments of the present disclosure.
[0015] Fig. 6A and Figure 6B Exemplary reference samples for wide-angle intra prediction according to some embodiments of the present disclosure are shown.
[0016] Figure 7 An exemplary discontinuity in the case of wide-angle intra prediction direction is shown according to some embodiments of the present disclosure.
[0017] Figure 8 Interpolation of samples in a current video block for intra prediction in planar mode according to some embodiments of the present disclosure is shown.
[0018] Fig. 9 is a block diagram illustrating an exemplary sample prediction process according to some embodiments of the present disclosure.
[0019] Fig.10 A diagram illustrating intra prediction along an interpolation direction and a vertical interpolation direction using samples of a video block according to some embodiments of the present disclosure.
[0020] Fig.11 A diagram illustrating intra prediction for interpolating samples in a video block using block-level interpolation directions according to some embodiments of the present disclosure.
[0021] Fig.12 Diagram illustrating intra prediction using fractional reference samples to interpolate samples in a video block according to some embodiments of the present disclosure.
[0022] Fig.13 Diagram illustrating intra prediction for interpolating samples in a video block using second order gradients of reconstructed samples in neighboring video blocks according to some embodiments of the present disclosure.
[0023] Fig.14 Diagram illustrating intra prediction for interpolating samples in a video block using second order gradients of fractional reference samples generated based on reconstructed samples in neighboring video blocks according to some embodiments of the present disclosure.
[0024] Fig.15 Diagram illustrating intra prediction for interpolating samples in a video block using different sample-specific interpolation directions according to some embodiments of the present disclosure.
[0025] Fig.16 is a flowchart of an exemplary method for performing intra-prediction on a video block according to some embodiments of the present disclosure.
[0026] Fig.17 is a diagram of exemplary adaptive reference sample filtering according to some embodiments of the present disclosure.
[0027] Fig.18 is a diagram of an exemplary derivation of an adaptive reference sample filter using reconstruction samples according to some embodiments of the present disclosure.
[0028] Fig.19 is a flowchart of an exemplary method for performing planar prediction using an adaptive reference sample filter according to some embodiments of the present disclosure.
[0029] Fig. 20 is a block diagram illustrating an exemplary block size-based reference sample filtering process according to some embodiments of the present disclosure.
[0030] Fig.21 is a flowchart illustrating an exemplary method of performing reference sample filtering using a block-size based reference sample filter according to some embodiments of the present disclosure.
[0031] Fig. 22 is a flowchart illustrating an exemplary method for offline training of a reference sample filter based on a fixed block size according to some embodiments of the present disclosure.
[0032] Fig.23 is a flowchart illustrating an exemplary method for determining a reference sample filter based on an adaptive block size according to some embodiments of the present disclosure.
[0033] Fig.24A and Fig. 24B A diagram showing reference samples for position-dependent intra prediction combining (PDPC) in top-right diagonal mode and bottom-left diagonal mode, respectively, according to some embodiments of the present disclosure.
[0034] Fig.25 is a block diagram illustrating a computing environment coupled to a user interface according to some implementations of the present disclosure. DETAILED DESCRIPTION
[0035] Reference will now be made in detail to specific embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, many non-limiting specific details are set forth to help understand the subject matter presented herein. However, it will be apparent to those of ordinary skill in the art that various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, it will be apparent to those of ordinary skill in the art that the subject matter presented herein may be implemented on many types of electronic devices with digital video capabilities. It should be noted that the terms "first", "second", etc. used in the specification, claims, and drawings of the present disclosure are used to distinguish objects, rather than to describe any particular order or sequence. It should be understood that the data used in this manner may be interchangeable under appropriate conditions, so that the embodiments of the present disclosure described herein may be implemented in an order other than those shown in the drawings or described in the present disclosure.
[0036] Figure 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. Figure 1 As shown in , system 10 includes a source device 12 that generates and encodes video data to be later decoded by a destination device 14. Source device 12 and destination device 14 may include any of a wide variety of electronic devices, including desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and destination device 14 are equipped with wireless communication capabilities.
[0037] In some embodiments, the target device 14 may receive the encoded video data to be decoded via the link 16. The link 16 may include any type of communication medium or device capable of forwarding the encoded video data from the source device 12 to the target device 14. In one example, the link 16 may include a communication medium that enables the source device 12 to send the encoded video data directly to the target device 14 in real time. The encoded video data may be modulated according to a communication standard (such as a wireless communication protocol) and sent to the target device 14. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form a portion of a packet-based network (e.g., a local area network, a wide area network, or a global network such as the Internet). The communication medium may include a router, a switch, a base station, or any other device that may be conducive to facilitating communication from the source device 12 to the target device 14.
[0038] In some other embodiments, the encoded video data may be sent from the output interface 22 to the storage device 32. Subsequently, the encoded video data in the storage device 32 may be accessed by the target device 14 via the input interface 28. The storage device 32 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a digital versatile disk (DVD), a compact disc read-only memory (CD-ROM), a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device 32 may correspond to a file server or another intermediate storage device that can store the encoded video data generated by the source device 12. The target device 14 can access the stored video data from the storage device 32 via streaming or downloading. The file server can be any type of computer capable of storing encoded video data and sending the encoded video data to the target device 14. Exemplary file servers include a web server (e.g., for a website), a File Transfer Protocol (FTP) server, a Network Attached Storage (NAS) device, or a local disk drive. The target device 14 may access the encoded video data through any standard data connection suitable for accessing the encoded video data stored on the file server, including a wireless channel (e.g., a Wireless Fidelity (Wi-Fi) connection), a wired connection (e.g., a Digital Subscriber Line (DSL), a cable modem, etc.), or any combination thereof. The transmission of the encoded video data from the storage device 32 may be a streaming transmission, a download transmission, or a combination of both streaming and download transmissions.
[0039] like Figure 1 As shown in , source device 12 includes video source 18, video encoder 20 and output interface 22. Video source 18 may include sources such as or a combination of such sources: a video capture device (e.g., a camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video. As an example, if video source 18 is a camera of a security monitoring system, source device 12 and target device 14 may include a camera phone or a video phone. However, the embodiments described in the present disclosure may be generally applicable to video encoding and decoding, and may be applied to wireless and / or wired applications.
[0040] The captured, pre-captured, or computer-generated video may be encoded by the video encoder 20. The encoded video data may be sent directly to the target device 14 via the output interface 22 of the source device 12. The encoded video data may also (or alternatively) be stored on a storage device 32 for later access by the target device 14 or other devices for decoding and / or playback. The output interface 22 may also include a modem and / or a transmitter.
[0041] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data communicated via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included in the encoded video data sent over a communication medium, stored on a storage medium, or stored on a file server.
[0042] In some embodiments, the target device 14 may include a display device 34, which may be an integrated display device and an external display device configured to communicate with the target device 14. The display device 34 displays the decoded video data to the user and may include any of a variety of display devices, such as a Liquid Crystal Display (LCD), a plasma display, an Organic Light Emitting Diode (OLED) display, or another type of display device.
[0043] The video encoder 20 and the video decoder 30 may operate according to a proprietary standard or an industry standard (e.g., VVC, HEVC, MPEG-4, Part 10, AVC) or an extension of such a standard. It should be understood that the present disclosure is not limited to a specific video encoding / decoding standard and may be applicable to other video encoding / decoding standards. It is generally believed that the video encoder 20 of the source device 12 may be configured to encode video data according to any of these current standards or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 may be configured to decode video data according to any of these current standards or future standards.
[0044] The video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder and / or decoder circuits, respectively, such as one or more microprocessors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), discrete logic, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device may store instructions for the software in a suitable non-transitory computer-readable medium, and use one or more processors to execute the instructions in the hardware to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, and either of the encoders or decoders may be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.
[0045] Figure 2 is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in the present application. The video encoder 20 may perform intra-frame prediction encoding and inter-frame prediction encoding of video blocks within a video frame. Intra-frame prediction encoding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame prediction encoding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that the term "frame" may be used as a synonym for the term "image" or "picture" in the field of video coding and decoding.
[0046] like Figure 2As shown in FIG. 1 , the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (Decoded Picture Buffer, DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 also includes a motion estimation unit 42, a motion compensation unit 44, a partition unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copy (Block Copy, BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63 (such as a deblocking filter) may be located between the adder 62 and the DPB 64 to filter the block boundaries to remove block artifacts from the reconstructed video data. In addition to the deblocking filter, another loop filter (such as an SAO filter, a cross component sample adaptive offset (CCSAO) filter and / or an adaptive in-Loop filter (ALF)) may be used to filter the output of the adder 62. It should be noted that, for the CCSAO technology, the present application is not limited to the embodiments described herein, but the present application may be applied to such a case: an offset is selected for any other one of the luminance component, the Cb chrominance component and the Cr chrominance component according to any one of the luminance component, the Cb chrominance component and the Cr chrominance component to modify the any other component based on the selected offset. Further, it should be noted that the first component mentioned herein may be any one of the luminance component, the Cb chrominance component and the Cr chrominance component, the second component mentioned herein may be any other one of the luminance component, the Cb chrominance component and the Cr chrominance component, and the third component mentioned herein may be the remaining one of the luminance component, the Cb chrominance component and the Cr chrominance component. In some examples, the loop filter may be omitted and the decoded video blocks may be provided directly by adder 62 to DPB 64. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed among one or more of the illustrated fixed or programmable hardware units.
[0047] Video data memory 40 may store video data to be encoded by components of video encoder 20. The video data in video data memory 40 may be obtained, for example, from Figure 1The video data memory 40 and the DPB 64 are obtained from the video source 18 shown. The DPB 64 is a buffer that stores reference video data (reference frames or pictures) for use by the video encoder 20 (e.g., in intra-frame or inter-frame prediction coding mode) when encoding video data. The video data memory 40 and the DPB 64 may be formed by any of a variety of memory devices. In various examples, the video data memory 40 may be on-chip with other components of the video encoder 20, or off-chip relative to those components.
[0048] like Figure 2 As shown in , after receiving the video data, the partitioning unit 45 within the prediction processing unit 41 partitions the video data into video blocks. This partitioning operation may also include partitioning the video frame into strips, tiles (e.g., a set of video blocks) or other larger coding units (Coding Unit, CU) according to a predefined splitting structure associated with the video data (e.g., a quadtree (Quad-Tree, QT) structure). A video frame is or can be considered to be a two-dimensional array or matrix of samples with sample values. The samples in the array may also be referred to as pixels or pels. The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the video frame. The video frame may be divided into multiple video blocks by, for example, using QT partitioning. A video block is also or can be considered to be a two-dimensional array or matrix of samples with sample values, but the size of the video block is smaller than the video frame. The number of samples in the horizontal and vertical directions (or axes) of the video block defines the size of the video block. The video block may be further divided into one or more block partitions or sub-blocks (which may again form blocks) by, for example, iteratively using QT partitioning, binary-tree (BT) partitioning or ternary-tree (TT) partitioning or any combination thereof. It should be noted that the term "block" or "video block" as used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. With reference to, for example, HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU) or a transform unit (TU) and / or may be or correspond to a corresponding block, such as a coding tree block (CTB), a coding block (CB), a prediction block (PB) or a transform block (TB). Alternatively or additionally, a block or video block may be or correspond to a sub-block of a CTB, CB, PB, TB, etc.
[0049] Prediction processing unit 41 may select one of a plurality of feasible prediction coding modes for the current video block based on the error results (e.g., code rate and distortion level), such as one of one or more inter-frame prediction coding modes in a plurality of intra-frame prediction coding modes. Prediction processing unit 41 may provide the resulting intra-frame or inter-frame prediction coding block (e.g., prediction block) to adder 50 to generate a residual block, and to adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. Prediction processing unit 41 also provides syntax elements (such as motion vectors, intra-frame mode indicators, partition information, and other such syntax information) to entropy coding unit 56.
[0050] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block to be encoded relative to one or more neighboring blocks in the same frame as the current block to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block relative to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate coding mode for each block of video data.
[0051] In some embodiments, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames, the motion vector indicating the displacement of a video block within the current video frame relative to a prediction block within a reference frame. Motion estimation performed by motion estimation unit 42 may be a process of generating a motion vector that may estimate motion for a video block. For example, a motion vector may indicate the displacement of a video block within a current video frame or picture relative to a prediction block within a reference frame. The predetermined pattern may designate a video frame in a sequence as a P frame or a B frame. Intra BC unit 48 may determine a vector (e.g., a block vector) for intra BC coding in a manner similar to the motion vectors determined by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine the block vector.
[0052] In terms of pixel differences, the prediction block of the video block may be or may correspond to a block or reference block of a reference frame that closely matches the video block to be encoded, and the pixel difference may be determined by the sum of absolute differences (SAD), the sum of square differences (SSD), or other difference metrics. In some embodiments, the video encoder 20 may calculate values for sub-integer pixel positions of the reference frame stored in the DPB 64. For example, the video encoder 20 may interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference frame. Thus, the motion estimation unit 42 may perform a motion search relative to the full pixel positions and the fractional pixel positions and output a motion vector with fractional pixel precision.
[0053] Motion estimation unit 42 calculates a motion vector for a video block in an inter-prediction coded frame by comparing the position of the video block to the position of a prediction block of a reference frame selected from a first reference frame list (list 0) or a second reference frame list (list 1), each of which identifies one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to motion compensation unit 44 and then to entropy encoding unit 56.
[0054] Motion compensation performed by motion compensation unit 44 may involve obtaining or generating a prediction block based on the motion vector determined by motion estimation unit 42. Upon receiving a motion vector for a current video block, motion compensation unit 44 may locate the prediction block pointed to by the motion vector in one of the reference frame lists, fetch the prediction block from DPB 64, and forward the prediction block to adder 50. Adder 50 then forms a residual block of pixel difference values by subtracting the pixel values of the prediction block provided by motion compensation unit 44 from the pixel values of the current video block being encoded. The pixel difference values forming the residual block may include luma component differences or chroma component differences or both. Motion compensation unit 44 may also generate syntax elements associated with the video block of a video frame for use by video decoder 30 when decoding the video block of the video frame. The syntax elements may include, for example, syntax elements defining a motion vector for identifying a prediction block, any flag indicating a prediction mode, or any other syntax information described herein. Note that motion estimation unit 42 and motion compensation unit 44 may be integrated together, but for conceptual purposes are described in detail below. Figure 2 are shown separately in.
[0055] In some embodiments, the intra BC unit 48 may generate vectors and obtain prediction blocks in a manner similar to that described above in conjunction with the motion estimation unit 42 and the motion compensation unit 44, but these prediction blocks are in the same frame as the current block being encoded, and these vectors are referred to as block vectors rather than motion vectors. Specifically, the intra BC unit 48 may determine the intra prediction mode to be used to encode the current block. In some examples, the intra BC unit 48 may, for example, use various intra prediction modes during different encoding passes to encode the current block, and test their performance through rate-distortion analysis. Next, the intra BC unit 48 may select a suitable intra prediction mode to use among the various tested intra prediction modes and generate an intra mode indicator accordingly. For example, the intra BC unit 48 may calculate rate-distortion values for the various tested intra prediction modes using rate-distortion analysis, and select the intra prediction mode with the best rate-distortion characteristic among the tested modes as the suitable intra prediction mode to be used. The rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and the original uncoded block that was encoded to generate the coded block, as well as the bit rate (i.e., the number of bits) used to generate the coded block. Intra BC unit 48 may calculate ratios from the distortions and rates for the various coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.
[0056] In other examples, intra BC unit 48 may use, in whole or in part, motion estimation unit 42 and motion compensation unit 44 to perform such functions for intra BC prediction in accordance with embodiments described herein. In either case, for intra block copying, the prediction block may be a block that is considered to closely match the block to be encoded in terms of pixel differences, which may be determined by SAD, SSD, or other difference metrics, and identification of the prediction block may include calculating values for sub-integer pixel positions.
[0057] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from different frames according to inter-frame prediction, video encoder 20 can form a residual block by subtracting the pixel values of the prediction block from the pixel values of the current video block being encoded to form pixel difference values. The pixel difference values forming the residual block may include both luma component differences and chroma component differences.
[0058] As an alternative to the inter-frame prediction performed by the motion estimation unit 42 and the motion compensation unit 44 or the intra-frame block copy prediction performed by the intra BC unit 48 as described above, the intra-frame prediction processing unit 46 may perform intra-frame prediction on the current video block. Specifically, the intra-frame prediction processing unit 46 may determine an intra-frame prediction mode for encoding the current block. For example, the intra-frame prediction processing unit 46 may use various intra-frame prediction modes to encode the current block, for example, during different encoding passes, and the intra-frame prediction processing unit 46 (or in some examples, the mode selection unit) may select a suitable intra-frame prediction mode from the tested intra-frame prediction modes to use. The intra-frame prediction processing unit 46 may provide information indicating the intra-frame prediction mode selected for the block to the entropy encoding unit 56. The entropy encoding unit 56 may encode the information indicating the selected intra-frame prediction mode into the bitstream.
[0059] After the prediction processing unit 41 determines the prediction block for the current video block via inter-frame prediction or intra-frame prediction, the adder 50 forms a residual block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to the transform processing unit 52. The transform processing unit 52 transforms the residual video data into transform coefficients using a transform such as a discrete cosine transform (DCT) or a conceptually similar transform.
[0060] Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan on the matrix including the quantized transform coefficients. Optionally, entropy encoding unit 56 may perform the scan.
[0061] After quantization, entropy coding unit 56 may encode the quantized transform coefficients into a video bitstream using an entropy coding technique, such as Context Adaptive Variable Length Coding (CAVLC), Context Adaptive Binary Arithmetic Coding (CABAC), Syntax-based context-adaptive Binary Arithmetic Coding (SBAC), Probability Interval Partitioning Entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be sent to a video bitstream such as a decoder. Figure 1 The video decoder 30 shown, or archived as Figure 1 The video frame may be stored in storage device 32 as shown for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also encode motion vectors and other syntax elements for the current video frame being encoded using entropy encoding techniques.
[0062] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain for use in generating a reference block for predicting other video blocks. A reconstructed residual block may be generated thereby. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values for motion estimation.
[0063] Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used as a prediction block by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 to inter-predict another video block in a subsequent video frame.
[0064] Figure 3 30 is a block diagram showing an exemplary video decoder 30 according to some embodiments of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform the above combined Figure 2The encoding process is substantially the inverse of the decoding process described with respect to video encoder 20. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, and intra-prediction unit 84 may generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.
[0065] In some examples, units of the video decoder 30 may be tasked to perform embodiments of the present application. In addition, in some examples, embodiments of the present disclosure may be dispersed in one or more of the multiple units of the video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present application alone or in combination with other units of the video decoder 30 (such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80). In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81 (such as the motion compensation unit 82).
[0066] The video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of the video decoder 30. The video data stored in the video data memory 79 may be obtained, for example, from the storage device 32, from a local video source (such as a camera), via a wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). The video data memory 79 may include a coded picture buffer (CPB) that stores encoded video data from the encoded video bitstream. The DPB 92 of the video decoder 30 stores reference video data for use by the video decoder 30 when decoding the video data (e.g., in an intra-frame or inter-frame prediction decoding mode). The video data memory 79 and the DPB 92 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For illustrative purposes, the video data memory 79 and the DPB 92 are shown in FIG. Figure 3 9 as two different components of video decoder 30. However, it will be apparent to those skilled in the art that video data memory 79 and DPB 92 may be provided by the same memory device or separate memory devices. In some examples, video data memory 79 may be on-chip with other components of video decoder 30, or off-chip relative to those components.
[0067] During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video frame and associated syntax elements. Video decoder 30 may receive syntax elements at the video frame level and / or the video block level. Entropy decoding unit 80 of video decoder 30 may decode the bitstream using entropy decoding techniques to obtain quantization coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. Entropy decoding unit 80 then forwards the motion vectors or intra-frame prediction mode indicators and other syntax elements to prediction processing unit 81.
[0068] When a video frame is encoded as an intra-prediction coded (I) frame or for intra-coded prediction blocks in other types of frames, intra-prediction unit 84 of prediction processing unit 81 may generate prediction data for a video block of the current video frame based on a signaled intra-prediction mode and reference data from a previously decoded block of the current frame.
[0069] When the video frame is encoded as an inter-frame prediction coded (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vectors and other syntax elements received from the entropy decoding unit 80. Each of the prediction blocks may be generated from a reference frame within one of the reference frame lists. The video decoder 30 may construct the reference frame lists, e.g., List 0 and List 1, using a default construction technique based on the reference frames stored in the DPB 92.
[0070] In some examples, when a video block is encoded or decoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture processed by video encoder 20 as the current video block.
[0071] The motion compensation unit 82 and / or the intra BC unit 85 determine prediction information for a video block of the current video frame by parsing the motion vector and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded. For example, the motion compensation unit 82 uses some of the received syntax elements to determine a prediction mode (e.g., intra prediction or inter prediction) for encoding and decoding a video block of a video frame, an inter prediction frame type (e.g., B or P), construction information for one or more of a reference frame list for a frame, a motion vector for each inter prediction encoded video block of a frame, an inter prediction state for each inter prediction encoded video block of a frame, and other information for decoding a video block in the current video frame.
[0072] Similarly, the intra BC unit 85 may use some of the received syntax elements, such as a flag to determine whether the current video block is predicted using intra BC mode, construction information of which video blocks of the frame are within the reconstruction region and should be stored in the DPB 92, a block vector for each intra BC predicted video block of the frame, an intra BC prediction state for each intra BC predicted video block of the frame, and other information for decoding the video blocks in the current video frame.
[0073] Motion compensation unit 82 may also perform interpolation using interpolation filters as used during encoding of the video blocks by video encoder 20 to calculate interpolated values for sub-integer pixels of reference blocks. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 from the received syntax elements and use these interpolation filters to generate the prediction blocks.
[0074] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 80, using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.
[0075] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. The decoded video block may also be referred to as a reconstructed block of the current video block. A loop filter 91 (such as a deblocking filter, an SAO filter, a CCSAO filter, and / or an ALF) may be located between the adder 90 and the DPB 92 to further process the decoded video block. In some examples, the loop filter 91 may be omitted, and the decoded video block may be provided directly to the DPB 92 by the adder 90. The decoded video block in a given frame is then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 may also store the decoded video for later presentation on a display device (such as Figure 1 on a display device 34).
[0076] In a typical video encoding and decoding process (e.g., including a video encoding process and a video decoding process), a video sequence generally includes an ordered set of frames or pictures. Each frame may include three sample arrays, denoted as SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In other cases, a frame may be monochrome and therefore include only a two-dimensional array of luma samples.
[0077] like Figure 4A As shown in , the video encoder 20 (or more specifically, the partitioning unit 45) generates an encoded representation of a frame by first partitioning the frame into a set of CTUs. A video frame may include an integer number of CTUs arranged consecutively from left to right and from top to bottom in a raster scan order. Each CTU is the largest logical coding unit, and the width and height of the CTU are signaled by the video encoder 20 in a sequence parameter set so that all CTUs in a video sequence have the same size, one of 128×128, 64×64, 32×32, and 16×16. It should be noted, however, that the CTUs in the present disclosure are not necessarily limited to a specific size. As Figure 4B As shown in , each CTU may include one CTB for luma samples, two corresponding coding tree blocks for chroma samples, and syntax elements for encoding and decoding samples of the coding tree blocks. The syntax elements describe the properties of different types of units of coded pixel blocks and how the video sequence can be reconstructed at the video decoder 30, including inter-frame prediction or intra-frame prediction, intra-frame prediction mode, motion vectors, and other parameters. In a monochrome picture or a picture with three separate color planes, a CTU may include a single coding tree block and syntax elements for encoding and decoding samples of the coding tree block. The coding tree block may be an N×N sample block.
[0078] To achieve better performance, video encoder 20 may recursively perform tree partitioning, such as binary tree partitioning, ternary tree partitioning, quadtree partitioning, or a combination thereof, on the coding tree block of the CTU and divide the CTU into smaller CUs. Figure 4C As depicted in , a 64×64 CTU 400 is first partitioned into four smaller CUs, each CU having a block size of 32×32. Among the four smaller CUs, CU 410 and CU 420 are partitioned into four CUs of a block size of 16×16, respectively. Two 16×16 CUs 430 and CU 440 are further partitioned into four CUs of a block size of 8×8, respectively. Figure 4D Depicted is a diagram showing Figure 4C The quadtree data structure of the final result of the partitioning process of CTU 400 depicted in FIG. 4 is a quadtree data structure, each leaf node of the quadtree corresponds to a CU of each size ranging from 32×32 to 8×8. Figure 4BEach CU may include a CB of luma samples and two corresponding coding blocks of chroma samples of the same size frame, and syntax elements for encoding and decoding the samples of the coding blocks. In a monochrome picture or a picture with three separate color planes, a CU may include a single coding block and syntax structures for encoding and decoding the samples of the coding block. It should be noted that Figure 4C and Figure 4D The quadtree partitioning depicted in FIG is for illustrative purposes only, and one CTU may be split into CUs based on quadtree / ternary tree / binary tree partitioning to adapt to varying local characteristics. In a multi-type tree structure, one CTU is divided by a quadtree structure, and each quadtree leaf CU may be further divided by a binary tree structure and a ternary tree structure. Figure 4E As shown, there are multiple possible partition types for a coding block with a width of W and a height of H, namely, quadruple partition, vertical binary partition, horizontal binary partition, vertical ternary partition, vertical extended ternary partition, horizontal ternary partition and horizontal extended ternary partition.
[0079] In some embodiments, the video encoder 20 may further divide the coding block of the CU into one or more M×NPBs. The PB may include a rectangular (square or non-square) sample block to which the same prediction (inter or intra) is applied. The PU of the CU may include a PB of luma samples, two corresponding PBs of chroma samples, and syntax elements for predicting the PB. In a monochrome picture or a picture with three separate color planes, the PU may include a single PB and a syntax structure for predicting the PB. The video encoder 20 may generate a predicted luma block, a predicted Cb block, and a predicted Cr block for the luma PB, Cb PB, and Cr PB of each PU of the CU.
[0080] Video encoder 20 may use intra prediction or inter prediction to generate a prediction block for a PU. If video encoder 20 uses intra prediction to generate a prediction block for a PU, video encoder 20 may generate the prediction block for the PU based on decoded samples of a frame associated with the PU. If video encoder 20 uses inter prediction to generate a prediction block for a PU, video encoder 20 may generate the prediction block for the PU based on decoded samples of one or more frames other than the frame associated with the PU.
[0081] After the video encoder 20 generates the predicted luma block, the predicted Cb block, and the predicted Cr block for one or more PUs of a CU, the video encoder 20 may generate a luma residual block for the CU by subtracting the predicted luma block of the CU from the original luma coding block of the CU, so that each sample in the luma residual block of the CU indicates the difference between a luma sample in one of the predicted luma blocks of the CU and a corresponding sample in the original luma coding block of the CU. Similarly, the video encoder 20 may generate a Cb residual block and a Cr residual block for the CU, respectively, so that each sample in the Cb residual block of the CU indicates the difference between a Cb sample in one of the predicted Cb blocks of the CU and a corresponding sample in the original Cb coding block of the CU, and each sample in the Cr residual block of the CU may indicate the difference between a Cr sample in one of the predicted Cr blocks of the CU and a corresponding sample in the original Cr coding block of the CU.
[0082] In addition, if Figure 4C As shown in , the video encoder 20 may use quadtree partitioning to decompose the luma residual block, Cb residual block, and Cr residual block of a CU into one or more luma transform blocks, Cb transform blocks, and Cr transform blocks, respectively. A transform block may include a rectangular (square or non-square) sample block to which the same transform is applied. A TU of a CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements for transforming the transform block samples. Therefore, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In a monochrome picture or a picture with three separate color planes, a TU may include a single transform block and a syntax structure for transforming the samples of the transform block.
[0083] The video encoder 20 may apply one or more transforms to the luma transform block of the TU to generate a luma coefficient block for the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficient may be a scalar. The video encoder 20 may apply one or more transforms to the Cb transform block of the TU to generate a Cb coefficient block for the TU. The video encoder 20 may apply one or more transforms to the Cr transform block of the TU to generate a Cr coefficient block for the TU.
[0084] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), video encoder 20 may quantize the coefficient block. Quantization generally refers to a process in which transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. After video encoder 20 quantizes the coefficient block, video encoder 20 may apply entropy coding techniques to encode syntax elements indicating the quantized transform coefficients. For example, video encoder 20 may perform CABAC on the syntax elements indicating the quantized transform coefficients. Finally, video encoder 20 may output a bitstream including a sequence of bits that form a representation of an encoded frame and associated data, and the bitstream is stored in storage device 32 or sent to target device 14.
[0085] After receiving the bitstream generated by the video encoder 20, the video decoder 30 may parse the bitstream to obtain syntax elements from the bitstream. The video decoder 30 may reconstruct a frame of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing the video data is generally the inverse of the encoding process performed by the video encoder 20. For example, the video decoder 30 may perform an inverse transform on a coefficient block associated with a TU of the current CU to reconstruct a residual block associated with the TU of the current CU. The video decoder 30 also reconstructs the coding block of the current CU by adding samples of the prediction block for the PU of the current CU to corresponding samples of the transform block of the TU of the current CU. After reconstructing the coding block for each CU of the frame, the video decoder 30 may reconstruct the frame.
[0086] As mentioned above, video codecs mainly use two modes, namely, intra-frame prediction (or intra-frame prediction) and inter-frame prediction (or inter-frame prediction) to achieve video compression. It should be noted that intra-frame block copy (IBC) can be regarded as intra-frame prediction or a third mode. Between the two modes, inter-frame prediction contributes more to codec efficiency than intra-frame prediction because motion vectors are used to predict the current video block from the reference video block.
[0087] However, with the ever-improving video data capture technology and more refined video block sizes for retaining details in the video data, the amount of data required to represent the motion vector of the current frame has also increased significantly. One way to overcome this challenge is to benefit from the fact that not only a group of neighboring CUs in both the spatial domain and the temporal domain have similar video data for prediction purposes, but the motion vectors between these neighboring CUs are also similar. Therefore, the motion information of spatially neighboring CUs and / or temporally co-located CUs can be used as an approximation of the motion information (e.g., motion vector) of the current CU by exploring the spatial and temporal correlation of the spatially neighboring CUs and / or the temporally co-located CUs, which is also referred to as the "motion vector predictor (MVP)" of the current CU.
[0088] Instead of using the actual motion vector of the current CU (e.g., Figure 2 The actual motion vector determined by the motion estimation unit 42 as described above is encoded into the video bitstream, and the motion vector predictor of the current CU is subtracted from the actual motion vector of the current CU to generate the motion vector difference (MVD) of the current CU. By doing so, the motion vector determined by the motion estimation unit 42 for each CU of the frame does not need to be encoded into the video bitstream, and the amount of data used to represent motion information in the video bitstream can be significantly reduced.
[0089] As in the process of selecting a prediction block in a reference frame during inter-frame prediction of a coding block, both the video encoder 20 and the video decoder 30 may adopt a set of rules for constructing a motion vector candidate list (also referred to as a "merge list") of the current CU using those potential candidate motion vectors associated with the spatial neighboring CUs and / or temporally co-located CUs of the current CU, and then selecting a member from the motion vector candidate list as the motion vector predictor of the current CU. By doing so, the motion vector candidate list itself does not need to be sent from the video encoder 20 to the video decoder 30, and the index of the selected motion vector predictor in the motion vector candidate list is sufficient for the video encoder 20 and the video decoder 30 to encode and decode the current CU using the same motion vector predictor in the motion vector candidate list. Therefore, only the index of the selected motion vector predictor needs to be sent from the video encoder 20 to the video decoder 30.
[0090] A brief description of CTU partitioning in the Enhanced Compression Model (ECM) is provided herein. In HEVC, a quadtree structure (represented as a coding tree) can be used to partition a CTU into CUs to accommodate various local features. A decision can be made at the leaf-CU level whether to encode a picture area using inter-picture (temporal) prediction or intra-picture (spatial) prediction. Each leaf-CU can be further partitioned into 1, 2, or 4 PUs according to the PU partition type. Within each PU, the same prediction process can be applied, and relevant information can be sent to the video decoder 30 based on the PU. After obtaining the residual block by applying a prediction process based on the PU partition type, the leaf-CU can be divided into transform units (TUs) according to another quadtree structure similar to the coding tree of the CU. Features of the HEVC structure include that it has multiple partition unit concepts, such as CU, PU, and TU.
[0091] In VVC, a quadtree with nested multi-type trees using a binary or ternary partitioning structure replaces the concept of multiple partition unit types (e.g., CU, PU, TU) in HEVC. For example, in VCC, the distinction between the concepts of CU, PU, and TU is removed, except for the need for CUs whose size is too large for the maximum transform length, thereby supporting higher flexibility in the CU partition shape. In the coding tree structure, a CU can have a square or rectangular shape. The CTU is first divided by a quadtree (also called a quadtree) structure. Then, the quadtree leaf nodes can be further divided by a multi-type tree structure. Figure 4E As shown, there are multiple types of splitting in the multi-type tree structure, including vertical binary splitting (SPLIT_BT_VER), horizontal binary splitting (SPLIT_BT_HOR), vertical ternary splitting (SPLIT_TT_VER) and horizontal ternary splitting (SPLIT_TT_HOR). The multi-type tree leaf node is called CU. Unless the CU is too large for the maximum transform length, this splitting with a multi-type tree structure is used for prediction and transform processing without any further partitioning. This means that in most cases, CU, PU and TU have the same block size in a quadtree with a nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is less than the width or height of the color component of the CU.
[0092] Figure 5 A diagram showing intra modes according to some examples. In general, the basic intra prediction scheme applied in VVC is almost the same as that of HEVC, except that some prediction tools are further extended, added and / or improved, such as extended intra prediction with wide-angle intra mode, multiple reference line (MRL) intra prediction, position-dependent intra prediction combination (PDPC), ISP prediction, cross-component linear model (CCLM) prediction, and matrix-weighted intra prediction (MIP).
[0093] A brief description of extended intra prediction with wide-angle intra modes is provided here. Similar to HEVC, VVC predicts the samples of the current CU using a set of reference samples adjacent to the current CU (e.g., samples above and to the left of the current CU). However, in order to capture finer edge directions present in natural videos (especially for high-resolution (e.g., 4K) video content), the number of angular intra modes is extended from 33 in HEVC to 93 in VVC. Figure 5 As shown, among the 93 angle intra modes, modes 2 to 66 are conventional angle intra modes, modes -1 to -14 and modes 67 to 80 are wide angle intra modes. In addition to the angle intra modes, HEVC's planar modes ( Figure 5 Mode 0) and DC mode ( Figure 5Mode 1) in is also applied to VVC.
[0094] Since the quadtree / binarytree / ternarytree partitioning structure is applied in VVC, in addition to square video blocks, there are also rectangular video blocks for intra-frame prediction in VVC. Since the width and height of a given video block are not equal, multiple groups of angle intra-frame modes can be selected from 93 angle intra-frame modes for different block shapes. For example, for square and rectangular video blocks, in addition to the plane and DC modes, each block shape also supports 65 of the 93 angle intra-frame modes. When the rectangular block shape of the video block meets specific conditions, the video decoder 30 can use the mapping relationship shown in Table 1 below to adaptively determine the index of the wide-angle intra-frame mode of the video block according to the index of the traditional angle intra-frame mode received from the video encoder 20. That is, for non-square blocks, the video encoder 20 uses the index of the traditional angle intra mode to signal the wide angle intra mode, and the index of the traditional angle intra mode is mapped to the index of the wide angle intra mode by the video decoder 30 after being parsed, thereby ensuring that the total number (i.e., 67) of intra modes (i.e., planar mode, DC mode, and 65 angle intra modes among 93 angle intra modes) remains unchanged, and the intra mode encoding method remains unchanged. Therefore, while providing a consistent design across different block sizes, good efficiency in signaling intra modes is achieved.
[0095] Table 1 below shows the mapping relationship between the index of the conventional angle intra mode and the index of the wide angle intra mode for intra prediction of different block shapes in VCC, where W represents the width of the video block and H represents the height of the video block.
[0096] Table 1
[0097]
[0098]
[0099] like Fig. 6A and Figure 6B As shown, the top reference samples (shown in gray) of length 2W+1 and the left reference samples (shown in gray) of length 2H+1 are used for wide-angle intra prediction to support angular intra mode.
[0100] like Figure 7 As shown, the current video block (e.g., current CU) is a flat rectangle and the interpolation angle α is less than 45 degrees. In the case of wide-angle intra prediction, two vertically adjacent prediction samples (e.g., 702 and 704) can be predicted based on two non-adjacent reference samples (e.g., 706 and 708), respectively. A low-pass reference sample filter and side smoothing can be applied to wide-angle prediction to reduce Figure 7 Middle gap Δp αIncreased negative impact. When the wide-angle mode represents a non-fractional offset, there are 8 modes in the wide-angle mode that meet this condition, which are [-14, -12, -10, -6, 72, 76, 78, 80]. When predicting a video block through these modes, the samples in the reference buffer are directly copied without applying any interpolation. With such modifications, the number of samples that need to be smoothed is reduced. In addition, the modification makes the design of the non-fractional mode in the traditional prediction mode and the wide-angle mode consistent.
[0101] In VVC, 4:2:2 and 4:4:4 chroma formats are supported as well as 4:2:0. The chroma derivation mode (DM) derivation table for the 4:2:2 chroma format was originally ported from HEVC, expanding the number of entries from 35 to 67 to coincide with the expansion of intra prediction modes. Since the HEVC specification does not support interpolation angles below -135 degrees and above 45 degrees, the luma intra prediction modes ranging from 2 to 5 are mapped to 2. Therefore, the chroma DM derivation table for the 4:2:2 chroma format is updated by replacing some values of the entries of the mapping table to more accurately convert the interpolation angles for chroma blocks.
[0102] As mentioned earlier, intra prediction samples are generated from a set of adjacent reference samples, which may introduce discontinuities along the block boundaries between the current video block and its adjacent video blocks. The PDPC tool can be used in VVC to address such issues by adopting a weighted combination of intra prediction samples and boundary reference samples. In VVC, PDPC can be enabled for the following intra modes without signaling: planar mode, DC mode, angular intra mode with an index less than or equal to the index of the horizontal intra mode (i.e., mode 18), and angular intra mode with an index greater than or equal to the index of the vertical intra mode (i.e., mode 50) and less than or equal to 80. PDPC is not applied if the Block Differential Pulse Coded Modulation (BDPCM) mode is applied to the current block, or the index of the selected row / column of the reference samples used for MRL intra prediction is greater than 0.
[0103] Figure 8 FIG. 4 shows the interpolation of samples in the current video block for intra prediction in planar mode according to some embodiments of the present disclosure. Figure 8As shown, sample 802 is one of the samples in the current video block to be predicted / interpolated. The reference samples of the current video block used to interpolate the samples (e.g., 802) of the current video block located above and to the left of the current video block are shown in gray squares. The upper right reference sample 804 located at coordinates (W, -1) and having a value of p[W][-1] is used as the right reference for all horizontal filtering. Similarly, the lower left reference sample 806 located at coordinates (-1, H) and having a value of p[-1][H] is used as the bottom reference for all vertical operations, where W and H are the width and height of the current video block, respectively. Reference sample 808 has the same horizontal coordinate as sample 802, and reference sample 810 has the same vertical coordinate as sample 802. If sample 802 is located at coordinates (x, y), then reference samples 808 and 810 are located at (x, -1) and (-l, y), respectively. The horizontal prediction p for sample 802 is p[W][-1]. h [x][y] and vertical prediction p v [x][y] is calculated according to equations (1-1) and (1-2) as follows:
[0104] p h [x][y]=(W-1-x)*p[-1][y]+(x+1)*p[W][-1] (1-1),
[0105] p v [x][y]=(H-1-y)*p[x][-1]+(y+1)*p[-1][H] (1-2).
[0106] The prediction value for sample 802 is calculated by averaging the horizontal and vertical predictions on a sample basis according to equation (2):
[0107] p[x][y]={(p h [x][y]<<log 2 W)+(p v [x][y]<<log 2 H)+(1<<(log 2 W+log 2 H))}>>(1+log 2 W+log 2 H) (2),
[0108] Among them, >> represents a bit-by-bit right shift operation, and << represents a bit-by-bit left shift operation.
[0109] Fig. 9is a block diagram illustrating an exemplary sample prediction process 900 according to some embodiments of the present disclosure. In some embodiments, the sample prediction process 900 may be performed by the prediction processing unit 41 of the video encoder 20 or the prediction processing unit 81 of the video decoder 30. In some embodiments, the sample prediction process 900 may be performed by a processor on the encoder side or the decoder side (e.g., Fig.25 For illustrative purposes only, the processor is provided Fig. 9 In some embodiments, the sample prediction process 900 may include reference sample determination 902, block-level interpolation direction determination 904 or sample-specific interpolation direction determination 906, and sample interpolation 908. Figures 10 to 15 Describe together Fig. 9 .
[0110] Initially, the processor may receive a plurality of reconstructed samples in a neighboring block region (e.g., a previously decoded block) in a video frame of a video. In VVC and ECM standards, the neighboring video block is located on the left neighboring region of the current video block, the top neighboring region of the current video block, or both, such as Figure 8 In other current or future standards, the reconstructed sample points may be located in any adjacent area of the current video block, such as Fig.10 Shown are gray squares.
[0111] In some embodiments, to implement reference sample determination 902, the processor may determine reference samples in the adjacent video block according to two oblique lines passing through samples of the current video block. One of the two oblique lines is along the interpolation direction, and the other oblique line is perpendicular to the first oblique line. For example, Fig.10 As shown, on sample 1002, oblique line 1012 is perpendicular to oblique line 1014. When the interpolation direction of sample 1002 is along oblique line 1012, four reconstructed sample points (e.g., sample 1004, sample 1006, sample 1008, and sample 1010) are determined by the processor as reference sample points for interpolating sample 1002 in the current video block according to oblique lines 1012 and 1014. The processor may generate a prediction of sample 1002 based on sample 1004, sample 1006, sample 1008, and sample 1010 according to bilinear interpolation.
[0112] In some embodiments, the processor interpolates each sample point in the current video block along a single interpolation direction. That is, the interpolation direction is a block-level interpolation direction. For example, Fig.11 As shown, the interpolation angle is θ 1 and θ 2 The block-level interpolation direction is used to interpolate all sample points (eg, sample points 1102 and 1104) in the current video block.
[0113] In some embodiments, block-level interpolation direction determination 904 may be implemented by a processor in an encoder (e.g., video encoder 20). The processor determines the block-level interpolation direction of the current video block based on a set of predetermined candidate interpolation directions. For example, the set of predetermined candidate interpolation directions may include but are not limited to: (i.e., {0, 30°, 45°, 60°, 135°}). The processor calculates the rate-distortion cost of the current video block when the video block is interpolated according to each candidate interpolation direction. In some embodiments, for each candidate interpolation direction, the processor first determines a reference sample along the candidate interpolation direction for each sample, and interpolates the sample based on the determined reference sample. The distance between the sample and each reference sample is used as a weight in the interpolation. Then, the processor calculates the rate-distortion cost of the sample, wherein the rate-distortion cost of the sample is determined based on the amount of distortion between the interpolated sample and the original uncoded sample. The processor further adds the rate-distortion costs of all samples in the current video block as the rate-distortion cost of the current video block. The processor determines the candidate interpolation direction associated with the rate-distortion cost having the smallest value as the block-level interpolation direction. The processor may generate a syntax element (e.g., planar_pred_dir) and signal the determined block-level interpolation direction to a decoder (e.g., decoder 30) in the bitstream. A processor in a decoder (eg, video decoder 30) may receive an encoded current video block and an index (eg, syntax element) that signals a block-level interpolation direction. The processor in the decoder may decode the encoded current video block based on the received syntax element.
[0114] like Fig.11 As shown, because the interpolation angle θ 1 =45° and θ 2 =135°, so all reference samples used to interpolate samples in the current video block (e.g., samples 1102 and 1104) are located in the interpolation direction. In some embodiments, the reference samples are not located in the interpolation direction. For example, Fig.12 Diagram illustrating intra prediction using fractional reference samples to interpolate samples in a video block according to some embodiments of the present disclosure. Fig.12 The interpolation angle θ in is not equal to 45° or 135°, and some reconstructed samples (shown by gray squares) in the adjacent block area may not be located in the interpolation direction (shown by dashed arrow lines). The processor generates fractional reference samples (e.g., samples 1204, 1206, 1208, and 1210) by interpolating two adjacent reconstructed samples closest to the interpolation direction (e.g., dashed lines 1212 and 1214). Fig.12As shown, fractional reference sample 1204 (shown as a dotted square) is generated by interpolating reconstructed sample 1204-1 and 1204-2. Similarly, fractional reference sample 1206 (shown as a dotted square) is generated by interpolating reconstructed sample 1206-1 and 1206-02. Fractional reference sample 1208 (shown as a dotted square) is generated by interpolating reconstructed sample 1208-1 and 1208-02. Fractional reference sample 1210 (shown as a dotted square) is generated by interpolating reconstructed sample 1210-1 and 1210-02.
[0115] In some embodiments, the processor may calculate the coordinates of the fractional reference sample points based on the width and height of the current video block and the interpolation direction of the current video block according to equations (3-1) to (6-2):
[0116]
[0117]
[0118] Where W and H represent the width and height of the current video block, θ represents the interpolation angle of the current video block and (x, y) represents the sample point to be interpolated (for example, Fig.12 The coordinates of the sample point 1202 in (x 1 ,y 1 ) represents the fractional reference sample points in the adjacent block area located to the right of the current video block (e.g., Fig.12 The coordinates of the sample point 1204 in (x 2 ,y 2 ) represents the fractional reference sample points in the adjacent block area located to the left of the current video block (e.g., Fig.12 The coordinates of the sample point 1206 in (x 3 ,y 3 ) represents the fractional reference samples in the neighboring block area above the current video block (e.g., Fig.12 The coordinates of the sample point 1208 in (x 4 ,y 4 ) represents the fractional reference samples in the adjacent block area below the current video block (e.g., Fig.12 The processor may then determine the weights for interpolating each fractional reference sample based on the calculated coordinates of the fractional reference sample and the two adjacent reconstructed samples. The processor may also interpolate the samples in the current video block (e.g., Fig.12 The sample points 1202 in are interpolated.
[0119] The processor may use finite precision interpolation to generate fractional reference samples and interpolate the samples into the current video block. In some embodiments, typical interpolation precisions applied by the processor include integers, 1 / 2, 1 / 4, 1 / 8, 1 / 16, etc. For example, when the processor uses any one of equations (3-1) to (6-2) to calculate the coordinates of the fractional reference samples, the calculated coordinates of the fractional reference samples are rounded to the nearest allowed fractional position.
[0120] Consistent with the present disclosure, in the VVC or ECM standard, because the reconstructed samples at the bottom right of the current video block are filled from the top right and bottom left samples of the current video block, the reference samples at the bottom right of the current video block are not used to interpolate the current video block. For example, if the video is encoded by the VVC or ECM standard, the processor does not perform interpolation based on samples 1204 and 1210. Fig.12 Instead, the processor interpolates sample 1202 based on samples 1206 and 1208. If the reference sample at the bottom right of the current video block can be generated by other methods for interpolating the current video block, or in other standards, four reference samples are used to interpolate the current video block.
[0121] In some embodiments, to implement block-level interpretation direction determination 904, an interpolation direction of the current video block is inferred from neighboring reconstructed samples in an encoder (e.g., video encoder 20) or a decoder (e.g., decoder 30). Fig.13 Diagram illustrating intra prediction for interpolating samples in a video block using second order gradients of reconstructed samples in neighboring video blocks according to some embodiments of the present disclosure. Fig.13 The three adjacent rows / columns (e.g., the zeroth line 1301, the first line 1303, and the second line 1305) of the reconstructed sample points in the current video block are used to infer the interpolation direction of the current video block. In some embodiments, the second-order gradients of the adjacent reconstructed sample points are used to determine the block-level interpolation direction. For each sample point in the current video block (e.g., Fig.13 The second-order gradient g along the inferred interpolation direction (e.g., line 1320) and its perpendicular direction (e.g., line 1322) is calculated according to equations (7-1) and (7-2) as follows: 1 and g 2 :
[0122] g 1 =(B -2,x -B -1,x )-(B -3,x -B -2,x )=2B -2,x -B -1,x -B -3,x (7-1),
[0123] g 2 =(C y,-2 -C y,-1 )-(C y,-3 -C y,-2 )=2C y,-2 -C y,-1 -C y,-3 (7-2),
[0124] Among them, C y,-3 , C y,-2 and C y,-1 represents the right adjacent sample point along the interpolation direction (eg, line 1320), B -3,x , B -2,x and B -1,x represents the left neighboring sample along the vertical interpolation direction (e.g., line 1322). In some embodiments, the L1 norm or the L2 norm is used to measure the energy of the change caused by the change between adjacent reconstructed samples. The cost of the current video block can be calculated by summing the costs of all samples in the current video block, where the cost of each sample in the current video block is calculated based on the second-order gradient according to equations (7-1) and (7-2) according to the L1 norm or the L2 norm. The processor can calculate the cost of the current video block according to equation (8) or equation (9), respectively.
[0125]
[0126] In some embodiments, as shown in equations (7-1) and (7-2), a first-order gradient is used to generate a second-order gradient. -2,x -B -1,x )、(B -3,x -B -2,x )、(C y,-2 -C y,-1 ) and (C y,-3 -C y,-2 )) can be normalized based on the distance between adjacent reconstruction samples used to calculate each first-order gradient. Then, the second-order gradients g1 and g2 are calculated based on the normalized first-order gradients according to equations (10-1) and (10-2):
[0127]
[0128] Among them, d 1 Represents the reconstruction sample point B -2,x and B -1,x The distance between them, d2 represents the reconstruction sample point B -3,x and B -2,x The distance between 3 Represents the reconstruction sample point C y,-2 and Cy,-1 The distance between 4 Represents the reconstruction sample point C y,-3 and C y,-2 The distance between.
[0129] In some embodiments, the reference sample points in the zeroth line, the first line, or the second line may not be located in the interpolation direction. Consistent with the present disclosure, the processor may generate fractional reference sample points (e.g., Fig.14 For example, the processor uses the same Fig.12 The same interpolation method as the samples 1204, 1206, 1208 and 1210 in FIG. 1 is used to interpolate the samples 1404, 1406, 1408, 1410, 1412 and 1414. Then, the cost of the sample in the current video block (e.g., sample 1402) is calculated according to equation (8) or equation (9) using the generated fractional reference samples.
[0130] In some embodiments, to implement the sample-specific interpolation direction determination 906, the processor may generate different sample-specific interpolation directions for interpolating each of the multiple sample points in the current video block. Fig.15 Diagram illustrating intra prediction for interpolating samples in a video block using different sample-specific interpolation directions according to some embodiments of the present disclosure. Fig.15 Lines 1510 and 1512 infer a hypothesized interpolation direction and a perpendicular hypothesized interpolation direction for interpolating sample point 1502, and lines 1514 and 1516 infer a hypothesized interpolation direction and a perpendicular hypothesized interpolation direction for interpolating sample point 1504.
[0131] For each sample point in the current video block, the processor determines a sample-specific interpolation direction for each individual sample point based on a cost when the sample point is interpolated according to each of the plurality of hypothesized interpolation directions. The cost of a sample point at coordinate (x, y) (e.g., sample point 1302) may be calculated according to the L1 norm or L2 of the second-order gradient using equation (11) or equation (12) as follows:
[0132] cost1 = |2B -2,x -B -1,x -B -3,x |+|2C y,-2 -C y,-1 -C y,-3 |
[0133] =|(B -2,x-B -1,x )-(B -3,x -B -2,x )|+|(C y,-2 -C y,-1 )-(C y,-3 -C y,-2 )| (11)
[0134] cost2 = |2B -2,x -B -1,x -B -3,x | 2 +|2C y,-2 -C y,-1 -C y,-3 | 2
[0135] =|(B -2,x -B -1,x )-(B -3,x -B -2,x )| 2 +|(C y,-2 -C y,-1 )-(C y,-3 -C y,-2 )| 2 (12)
[0136] Among them, C y,-3 , C y,-2 and C y,-1 represents the right adjacent sample point along the interpolation direction (eg, line 1320), B -3,x , B -2,x and B -1,x represents the left neighboring sample point along the vertical interpolation direction (e.g., line 1322). In addition, the processor calculates the cost of each hypothesized interpolation direction according to the sample point according to equation (11) or equation (12). The processor determines the hypothesized interpolation direction associated with the cost having the minimum value as the sample point specific interpolation direction.
[0137] In some embodiments consistent with the present disclosure, the first-order gradients (e.g., (B -2,x -B -1,x )、(B -3,x -B -2,x )、(C y,-2 -C y,-1 ) and (C y,-3 -C y,-2 )) can be normalized based on the distance between adjacent reconstruction samples used to calculate each first-order gradient. The cost cost1 of the L1 norm and the cost cost2 of the L2 norm can be calculated based on the normalized first-order gradient according to equations (13) and (14) as follows:
[0138]
[0139] Among them, d 1 Represents the reconstruction sample point B -2,x and B -1,x The distance between 2 Represents the reconstruction sample point B -3,x and B -2,x The distance between 3 Represents the reconstruction sample point C y,-2 and C y,-1 The distance between 4 Represents the reconstruction sample point C y,-3 and C y,-2 The distance between.
[0140] In accordance with the present disclosure, when one or more reconstructed sample points in the adjacent block region are not in the assumed interpolation direction of the sample point, such as Fig.14 As shown, the processor may first generate fractional reference samples by interpolating two adjacent reconstructed samples closest to the assumed interpolation direction. Then, the processor may use the generated fractional reference samples to interpolate samples along the assumed interpolation direction.
[0141] Reference Fig. 9 , in order to implement the sample interpolation 908, the processor may interpolate the current video block based on the determined block-level interpolation direction or the sample-specific interpolation block. If the processor is located at the encoder side and the block-level interpolation direction is determined from a set of predetermined candidate interpolation directions, the processor may generate an index of the predetermined block-level interpolation direction and then signal the index to the decoder in the bitstream. If the processor is located at the decoder side and receives the index indicating the block-level interpolation direction and the encoded video block, the processor may interpolate the encoded video block based on the received block-level interpolation direction.
[0142] Fig.16 1600 is a flowchart of an exemplary method for performing intra-frame prediction on a video block according to some embodiments of the present disclosure. The method 1600 may be implemented by a processor associated with the video encoder 20 or the video decoder 30. Fig. 9 The sample prediction process described may be used to implement method 1600. As described below, method 1600 may include steps 1602 to 1610, wherein steps 1604 to 1606 may be optional. In order to perform the disclosure provided herein, some steps may be optional. In addition, some steps may be performed simultaneously or in combination with Fig.16 The order shown in the figure is different from the order in which it is executed.
[0143] In step 1602, the processor may receive a video frame of a video. The video frame may include a set of reconstructed samples or blocks, and the set of reconstructed samples or blocks are video blocks to be predicted (e.g., Figures 10 to 15 The processor may also receive a set of predetermined candidate interpolation directions encoded and decoded in the current video block or video frame.
[0144] In some implementations of method 1600 , steps 1604 and 1606 may be optionally performed to determine the interpolation direction used in step 1608 .
[0145] In step 1604, the processor may determine the block-level interpolation direction of the current video block based on the reconstructed samples in the received adjacent block area. In some embodiments, an explicit method may be used to determine the block-level interpolation direction on the encoder side. The processor may determine the reference sample based on a set of received candidate interpolation directions. When the current video block is interpolated according to each of the multiple candidate interpolation directions, the processor may calculate the rate-distortion cost of the current video block. The candidate interpolation direction associated with the rate-distortion cost with the minimum value is determined by the processor as the block-level interpolation direction. In some embodiments, the processor implicitly determines the block-level interpolation direction based on the second-order gradient of the adjacent reconstructed samples. The implicit method may be performed on the encoder side or the decoder side. For example, the processor first calculates the cost of each sample in the current video block based on the second-order gradient of the adjacent reconstructed samples in each assumed interpolation direction. The sum of the costs of each sample is used as the cost of the current video block to determine the block-level interpolation direction in the assumed interpolation direction. That is, the processor determines the assumed interpolation direction associated with the cost of the current video block with the minimum value as the block-level interpolation direction.
[0146] As an alternative to step 1604, in step 1606, the processor may determine a different sample-specific interpolation direction for each sample in the current video block. That is, each sample in the current video block is interpolated using a separate sample-specific interpolation direction. To determine the sample-specific interpolation direction for each sample, the processor may calculate the cost for each sample in the current video block based on the second-order gradient of the neighboring reconstructed sample in each hypothetical interpolation direction. The processor also determines the hypothetical interpolation direction associated with the cost of the sample with the minimum value as the sample-specific interpolation direction for the sample.
[0147] If the processor is at the encoder side and the block-level interpolation direction is selected from a set of predetermined candidate interpolation directions, then after step 1604 or 1606, the processor may generate an index of the block-level interpolation direction and signal the index to the decoder in the bitstream.
[0148] In step 1608, the processor determines reference samples in neighboring video blocks of the video block along the interpolation direction of each sample of the current video block. For example, the processor determines a reconstructed sample located on a line passing through the sample along the interpolation direction or a line passing through the sample along a perpendicular interpolation direction as a reference sample for interpolating the corresponding sample in the current video block. If the reconstructed sample is not located in the interpolation direction or the perpendicular interpolation direction, the processor may generate fractional reference samples based on two adjacent reconstructed samples closest to the line along the interpolation direction or the line along the perpendicular interpolation direction. Then, the fractional reference samples are used to interpolate the corresponding sample in the current video block.
[0149] In step 1610 , the processor may interpolate each sample of the current video block based on the reference sample.
[0150] In some embodiments, if the processor is located on the decoder side, the processor may receive an index indicating a block-level interpolation direction and an encoded video block, and interpolate the encoded video block based on the block-level interpolation direction signaled in the received bitstream. In some other embodiments, the decoder processor may determine the interpolation direction by itself, for example, using the implicit method described in the section of step 1604 (for determining the block-level interpolation direction) and step 1606 (for determining the sample-specific interpolation direction). In some embodiments, the reference samples in the planar mode are first smoothed before being used to interpolate the samples in the current video block. For example, in the VVC and ECM standards, the reference samples are filtered by a fixed filter with a coefficient of
[121] / 4 before being used to interpolate other samples. In another example, an adaptive loop filter (ALF) technique is used to capture different properties of video encoding and decoding by deriving filter coefficients for each CTU. In some embodiments, an adaptive reference sample filter can be used to filter reference samples for intra-frame prediction in planar mode. For example, Fig.17 is a diagram of exemplary adaptive reference sample filtering according to some embodiments of the present disclosure. Fig.17 The three-tap filter [r 0 , r 1 , r 2 ] is used to smooth the reference samples (e.g., sample 1504) in the adjacent video block above the current video block, and another three-tap filter [c0, c1, c2] is used to smooth the reference samples (e.g., sample 1506) in the adjacent video block to the left of the current video block. Fig.17 Samples 1704 and 1706 in FIG. 1 are reference samples used to interpolate sample 1702 in a planar mode. Consistent with the present disclosure, if sample 1702 is located at coordinates (x, y), then samples 1704 and 1706 are located at coordinates (x, -1) and (-1, y), respectively. The filtered value of sample 1704 may be based on the filter [r0 , r 1 , r 2 ] is calculated based on the unfiltered values of sample 1704 and its two neighboring samples in the same row (e.g., samples 1704-1 and 1704-2). Similarly, the filtered value of sample 1706 can be calculated based on the filter [c0, c1, c2] based on the unfiltered values of sample 1704 and its two neighboring samples in the same row (e.g., samples 1706-1 and 1706-2). The filtered values of samples 1704 and 1706 can be calculated as follows according to equations (15-1) and (15-2):
[0151] p f [x][-1] = r 0 *p[x-1][-1]+r 1 *p[x][-1]+r 2 *p[x+1][-1] (15-1),
[0152] p f [-1][y]=c 0 *p[-1][y+1]+c 1 *p[-1][y]+c 2 *p[-1][y-1] (15-2),
[0153] Among them, p f [x][-1] indicates that the sample 1704 is based on the filter [r 0 , r 1 , r 2 ] represents the filtered value of sample 1704-1, p[x-1][-1] represents the unfiltered value of sample 1704, p[x+1][-1] represents the unfiltered value of sample 1704-2, and p[x-1][-1] represents the unfiltered value of sample 1704-1. f [-1][y] indicates that the sample point 1706 is based on [c 0 , c 1 , c 2 ] represents the filtered value of sample point 1706-1, p[-1][y] represents the unfiltered value of sample point 1706, and p[-1][y-1] represents the unfiltered value of sample point 1706-2.
[0154] Consistent with the present disclosure, the horizontal interpolation p for sample point 1702 in planar mode is h [x][y] and vertical interpolation p v [x][y] can be calculated according to equation (16-1) and equation (16-2):
[0155] p h[x][y]=(W-1-x)*p f [-1][y]+(x+1)*p f [W][-1] (16-1),
[0156] p v [x][y]=(H-1-y)*p f [x][-1]+(y+1)*p f [-1][H] (16-2).
[0157] The predicted value of sample point 1702 can be obtained based on the horizontal interpolation p according to equation (17): h [x][y] and vertical interpolation p v The average value of [x][y] is obtained as follows:
[0158] q[x][y]=0.5*p h [x][y]+0.5*p v [x][y] (17).
[0159] In some embodiments, the cost function of a sample in the current video block (eg, sample 1702) is defined as:
[0160]
[0161] Where s[x][y] represents the original value of the sample, and q[x][y] is the predicted value of the sample interpolated from the reference sample. In some embodiments, equation (18) can be solved using the Wiener-Hopf equation as in the ALF method. In some alternative embodiments, the processor can use the least squares method to optimize the filter coefficients (e.g., [r 0 , r 1 , r 2 ] and / or [c 0 , c 1 ,c 2 ]) to minimize the cost value in the cost function (e.g., equation (18)). When the processor is located in an encoder (e.g., video encoder 20), the processor signals the optimized filter coefficients to a decoder (e.g., decoder 30) in a bitstream.
[0162] In some embodiments, a processor in an encoder may receive a set of predetermined candidate filters. The processor calculates a cost value based on each received candidate filter using equation (18). The processor then determines the filter associated with the cost having the smallest value as the optimal filter. The processor may quantize the optimal filter into an integer type. In addition, the processor generates an index of the optimal filter in the bitstream and sends the index to the decoder using a signal. It should be understood that the type of filter used to filter the reference samples is not limited to a 3-tap filter, and any other type of filter (e.g., a 2-tap filter, a 4-tap filter) may be incorporated into the present disclosure. In some embodiments, a single filter is used to filter all reference samples. In some alternative embodiments, different filters (e.g., Fig.17 The filter in [r 0 ,r 1 ,r 2 ] and filter [c 0 ,c 1 ,c 2 ]) is used to filter different reference samples.
[0163] In some embodiments, the reconstructed samples may be used by a processor on the encoder side or the decoder side to derive filter coefficients. Fig.18 is a diagram of an exemplary derivation of an adaptive reference sample filter using reconstruction samples according to some embodiments of the present disclosure. Fig.18 As shown, four lines of reconstructed samples in adjacent blocks (eg, the zeroth line 1801, the first line 1803, the second line 1805, and the third line 1807) are used to determine an adaptive reference sample filter (eg, filter [r 0 ,r 1 ,r 2 ] and filter [c 0 ,c 1 ,c 2 ]). The third line 1807 is used as a reference sample to predict the samples in the zeroth line 1801, the first line 1803 and the second line 1805. Filter [r 0 ,r 1 ,r 2 ] can filter the samples in the row of the third line 1807, the filter [c 0 ,c 1 ,c 2 ] The samples in the columns of the third line 1807 may be filtered. Each sample in the zeroth line 1801, the first line 1803, and the second line 1805 may be interpolated by filtering the samples in the third line 1907. For example, Fig.18 As shown, samples 1804-1 and 1804-2 are first filtered using a filter [r0 ,r 1 ,r 2 ] and filter [c 0 ,c 1 ,c 2 ] are filtered respectively. Then, the sample 1804 can be interpolated based on the filtered samples 1804-1 and 1804-2 by equations such as equations (16-1), (16-2) and (17). A cost function such as equation (18) for all predicted samples (e.g., samples in the zeroth line 1801, the first line 1803, the second line 1805 and the third line 1807) can be optimized, and the filter coefficients (e.g., [r 0 ,r 1 ,r 2 ]、[c 0 ,c 1 ,c 2 ]) can be solved by the Wiener-Hopf equation. The optimized filter can be used to filter the reference samples (e.g., in the zeroth line 1801). The filtered reference samples are used to interpolate the samples in the current video block in a planar mode.
[0164] It should be understood that the type of filter used to filter the reconstructed samples is not limited to a 3-tap filter, and any other type of filter (e.g., a 2-tap filter, a 4-tap filter) may be incorporated into the present disclosure. In some embodiments, a single filter may be used to filter all reconstructed samples. In some alternative embodiments, different filters (e.g., Fig.18 The filter in [r 0 ,r 1 ,r 2 ] and filter [c 0 ,c 1 ,c 2 ]) is used to filter different reconstructed samples.
[0165] Fig.19 1 is a flowchart of an exemplary method for performing plane prediction using an adaptive reference sample filter according to some embodiments of the present disclosure. Method 1900 may be implemented by a processor associated with video encoder 20 or video decoder 30. Method 1900 may include steps 1902 to 1908 as described below. In order to perform the disclosure provided herein, some steps may be optional. In addition, some steps may be performed simultaneously or in different orders.
[0166] In step 1902, the processor may receive a video frame of a video. The video frame may include video blocks to be predicted (e.g., Figure 17 to Figure 18A set of reconstructed sample points or blocks of neighboring block areas of a current video block in a video frame. The current video block may be a CTU or a CU / PU.
[0167] In step 1904, for each sample in the current video block, the processor may determine a reference sample for performing planar prediction. Fig.17 As shown, samples 1704 and 1706 are determined by the processor as reference samples for interpolating sample 1702 in planar mode.
[0168] In step 1906, the processor applies at least one adaptive reference sample filter to the reference samples determined in step 1904. For example, Fig.17 As shown, the filter [r 0 ,r 1 ,r 2 ] and filter [c 0 ,c 1 ,c 2 ] are applied to sample points 1704 and 1706 respectively.
[0169] In step 1908, the processor performs planar prediction on each sample in the current video block using the reference samples filtered in step 1906. For example, the processor performs planar prediction on each sample in the current video block based on the filtered samples 1704 and 1706 in planar mode. Fig.17 The sample point 1702 in is interpolated.
[0170] In some embodiments, adaptive sample filtering (e.g., method 1900) may be performed in conjunction with directional plane interpolation (e.g., method 1600). For example, method 1900 may be performed between steps 1608 and 1610 of method 1600. However, it is contemplated that adaptive sample filtering and directional plane interpolation may also be separated from each other and applied separately. Either technique provides an improvement in the performance of intra prediction.
[0171] Fig. 20 2 is a block diagram illustrating an exemplary block size-based reference sample filtering process 2000 (hereinafter referred to as “process 2000”) according to some embodiments of the present disclosure. In some embodiments, process 2000 may be performed by the prediction processing unit 41 of the video encoder 20 or the prediction processing unit 81 of the video decoder 30. In some embodiments, process 2000 may be performed by a processor on the encoder side or the decoder side (e.g., Fig.25 For illustrative purposes only, the processor is provided Fig. 20 In some embodiments, process 2000 may include reference sample reception 2002 , reference sample filter determination 2004 , and reference sample filtering 2006 .
[0172] Initially, the processor may receive a video block to be predicted in a video frame of a video in reference sample point reception 2002 (eg, Figure 17 to Figure 18 A plurality of reference samples in a neighboring block region (eg, a previously decoded block) of a current video block in the video.
[0173] As mentioned above, a video frame in a video can be divided into multiple CTUs, and then each CTU can be divided into smaller video blocks (e.g., CUs) to accommodate changing local characteristics. These video blocks of a video frame can have different sizes, such as Figure 4C and Figure 4E In some embodiments, in order to implement reference sample filter determination 2004 on a current video block to be predicted based on multiple reference samples in a neighboring block region, the processor may adaptively determine a reference sample filter based on the size of the current video block, and use the filter to filter the reference samples before the reference samples are used to predict the block.
[0174] In some embodiments, the determined reference sample filter is among a plurality of reference sample filters pre-programmed in the processor. For example, the reference sample filter may be trained offline, and then the filter coefficients of the trained filter may be pre-programmed into the processor in the encoder and / or the processor in the decoder. Because the reference sample filters are trained offline and pre-programmed into the encoder and decoder separately, signaling of these filters between the encoder and the decoder is not required.
[0175] Offline training of the reference sample filter may be performed on the same processor or a separate processor that is later used for intra-frame prediction. The image or video frame used to train the reference sample filter may be segmented into blocks based on the statistics or local characteristics of the image or video frame. Blocks of the same size are collected for training the corresponding reference sample filter of the block size. In some embodiments, reference samples of video blocks of the same size are used as inputs for training data, and original samples of image blocks of the same size are used as targets for training data for deriving filter coefficients of the reference sample filter of the block size. In some embodiments, the filter coefficients of the reference sample filter may be derived using a least squares method (e.g., equation (18)). In some embodiments, training samples may be collected for different quantization parameters (QPs). Thus, different reference sample filters may be trained for different QPs. These reference sample filters may be associated with the same block size, but correspond to different QPs. Each reference sample filter is derived using training data that is compressed by an encoder using different QPs. In some alternative embodiments, one reference sample filter may be trained using training data that corresponds to all different QPs.
[0176] In some alternative embodiments, the determined reference sample filter for filtering the reference samples of the video block is among a plurality of reference sample filters adaptively derived by the encoder on a per video frame basis. For example, a processor in the encoder may adaptively derive a plurality of reference sample filters on a per video frame basis. Each derived reference sample filter corresponds to a different video block size in the video frame. The processor may then quantize the derived reference sample filters in the bitstream and signal them to the decoder. For example, filter coefficients of the derived reference sample filters may be signaled.
[0177] Reference sample filters may be adaptively derived by a processor in an encoder based on two-pass encoding. For example, in a first pass, a default reference sample filter (e.g., [1 2 1] / 4) may be used to filter reference samples of each video block in a video frame. The processor then uses the filtered reference samples to perform intra prediction (e.g., angular intra prediction or plane intra prediction) of the video block to obtain intra prediction samples for each video block. This first pass compression creates a statistical summary of the video data. For example, filter covariances may be collected. A processor in an encoder may derive reference sample filters, each reference sample filter being associated with a different video block size guided by a first pass compression of the video block, the first pass compression of the video block being obtained using the above-mentioned default reference sample filter. For example, the collected filter covariances may be used to derive reference sample filters. In some embodiments, filter coefficients of the reference sample filters may be derived using a least squares method (e.g., equation (18)).
[0178] In some embodiments, when encoding and decoding video in VVC and ECM, PDPC can be applied after prediction in angular intra mode. For example, in the first pass, the processor can first perform angular intra prediction on the video block using the filtered reference samples, and then perform PDPC on the predicted samples of the video block to obtain the final intra prediction samples of the video block used to derive the reference sample filter.
[0179] In a second pass, the processor then compresses each video block by filtering the reference samples using the derived reference sample filter according to the size of the video block.
[0180] In some embodiments, the processor may apply the determined reference sample filter to the reference samples in the reference sample filtering process 2006. The processor may then use the filtered reference samples to perform intra prediction (e.g., planar prediction) on the current video block. When the processor is in an encoder, the reference sample filter may be retrieved from a memory, or the reference sample filter may be adaptively derived. When the processor is in an encoder and the reference sample filter is adaptively derived by the processor, the reference sample filtering process 2006 may be the second pass described above. When the processor is in a decoder, the reference sample filter may be retrieved from a memory, or the reference sample filter may be obtained from signaling in a bitstream received from an encoder.
[0181] Fig.21 2 is a flowchart illustrating an exemplary method 2100 for performing reference sample filtering using a block size-based reference sample filter according to some embodiments of the present disclosure. The method 2100 may be implemented by a processor associated with the video encoder 20 or the video decoder 30. The method 2100 may include steps 2102 to at least 2108 as described below. Some steps may be optional for performing the disclosure provided herein. In addition, some steps may be performed simultaneously or in a different order.
[0182] At step 2102, the processor may receive a video frame of a video. The video frame may include video blocks to be predicted (e.g., Figure 17 to Figure 18 A set of reference samples or blocks of neighboring block areas of a current video block in a video frame. The current video block may be a CTU or a CU / PU.
[0183] At step 2104, the processor may determine the size of each video block, and for each video block size, the processor may determine a reference sample filter to filter the received reference samples of the video block of that size. In some embodiments, the processor may determine the reference sample filter from a plurality of reference sample filters pre-programmed in the processor. The pre-programmed reference sample filters may be trained offline by the same processor or a different processor.
[0184] For example, Fig. 222200 is a flowchart illustrating an exemplary method 2200 for offline training of a reference sample filter based on a fixed block size according to some embodiments of the present disclosure. In step 2202, a processor may receive input training data and target training data. For example, the input training data may be reference samples obtained by compressing original samples of video blocks of different sizes using an encoder. The target training data may be the corresponding original samples of the video block. In step 2204, the processor may derive multiple reference sample filters based on the input training data and the target training data using a least squares method (e.g., equation (18)). In some embodiments, when the encoder is used to collect and compress training data using different QPs, reference sample filters associated with corresponding block sizes may be derived for each different QP or for all different QPs together. In step 2206, filter coefficients of the derived multiple reference sample filters are pre-programmed in a processor of the encoder and / or decoder. The derived reference sample filters may also be pre-stored in a lookup table and stored in a memory (e.g., Fig.25 In the memory 2530).
[0185] return Fig.21 In step 2104, in some alternative implementations, the processor may determine a reference sample filter from a plurality of reference sample filters adaptively derived based on the video frame.
[0186] For example, Fig.23 23 is a flowchart illustrating an exemplary method for determining a reference sample filter based on an adaptive block size according to some embodiments of the present disclosure. In step 2302, the processor may apply a default reference sample filter (e.g., [1 2 1] / 4) to the reference samples of each video block in the video frame to obtain filtered reference samples. In step 2304, the processor may perform intra prediction (e.g., planar prediction, angular intra prediction) on each video block using the filtered reference samples of the video block to obtain intra prediction samples of each video block. In step 2306, the processor may further derive a reference sample filter for the video block size of the video frame using a least squares method (e.g., equation (18)) based on the reference samples of each video block of each size of the video frame and the corresponding prediction samples of the video block (predicted in step 2304). The derived reference sample filter is signaled to the decoder in the bitstream.
[0187] return Fig.21In step 2106, the processor applies the determined reference sample filter to the reference samples of the video block according to the size of the current video block. When the processor is in a decoder, if the determined reference sample filter is pre-programmed, the reference sample filter is retrieved from the processor or memory, or if the reference sample filter is adaptively derived based on the video frame, the reference sample filter is obtained from a signal in the bitstream.
[0188] In step 2108 , the processor performs intra prediction (eg, planar prediction) on samples in the video block using the reference samples filtered in step 2106 .
[0189] In some embodiments, block-size based reference sample filtering (e.g., method 2100) may be performed in conjunction with directional plane interpolation (e.g., method 1600). For example, method 2100 may be performed between step 1608 and step 1610 of method 1600. However, it is contemplated that block-size based reference sample filtering and directional plane interpolation may also be applied separately and independently of each other. Both techniques provide improvements to intra prediction performance.
[0190] Fig.24A and Fig. 24B A diagram showing reference samples for PDPC in upper right diagonal mode and lower left diagonal mode respectively according to some embodiments of the present disclosure. The prediction sample pred(x, y) is located at (x, y) of the prediction block. The reference sample R x’,-1 With horizontal coordinates x'=x+y+1 and vertical coordinates -1, and reference sample point R -1,y’ It has horizontal coordinate -1 and vertical coordinate y'=x+y+1.
[0191] Assuming that pred(x, y) is the predicted sample of the current sample, the modified predicted sample pred'(x, y) obtained by performing PDPC can be calculated according to equation (19):
[0192] pred'(x,y)=Clip3(0,(1<<BitDepth)–1,(wL×R -1,y’ +wT×R x’,-1 +(64-wL-wT)×pred(x,y)+32)>>6) (19),
[0193] Where Bitdepth represents the bit depth of the sample point; R x’,-1 and R -1,y’ They respectively represent the reference samples located at the upper and left boundaries of the reference samples; wL and wT are weights adaptively selected according to the intra mode and the block size of the current block; ">">" represents a bit-by-bit right shift operation, and "<<" represents a bit-by-bit left shift operation.
[0194] The function Clip3(x, y, z) in the above equation can be defined in equation (20) as:
[0195]
[0196] Fig.25 A computing environment 2510 is shown coupled to a user interface 2550. The computing environment 2510 may be part of a data processing server. Consistent with the present disclosure, the computing environment 2010 may be used to implement the video encoder 20 and the video decoder 30. The computing environment 2510 includes a processor 2520, a memory 2530, and an input / output (I / O) interface 2540.
[0197] The processor 2520 generally controls the overall operation of the computing environment 2510, such as operations associated with display, data acquisition, data communication, and image processing. The processor 2520 may include one or more processors to execute instructions to perform all or some steps in the above method. In addition, the processor 2520 may include one or more modules that facilitate interaction between the processor 2520 and other components. The processor 2520 may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a graphics processing unit (GPU), etc.
[0198] The memory 2530 is configured to store various types of data to support the operation of the computing environment 2510. The memory 2530 may include predetermined software 2532. Examples of such data include instructions for any application or method operating on the computing environment 2510, video data sets, image data, etc. The memory 2530 may be implemented by using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.
[0199] I / O interface 2540 provides an interface between processor 2520 and peripheral interface modules such as a keyboard, click wheel, buttons, etc. Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. I / O interface 2540 may be coupled to an encoder and a decoder.
[0200] In some embodiments, a non-transitory computer-readable storage medium is also provided, which includes a plurality of programs, for example, included in the memory 2530, and can be executed by the processor 2520 in the computing environment 2510 to perform the above method. Optionally, the non-transitory computer-readable storage medium may store therein a bit stream or a data stream, the bit stream or the data stream including the code generated by the encoder (for example, Figure 2 The video encoder 20 in FIG. 1 generates a video signal for a decoder (eg, Figure 3 The non-transitory computer-readable storage medium may be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0201] In some embodiments, a computing device is also provided, comprising: one or more processors (e.g., processor 2520); and a non-transitory computer-readable storage medium or memory 2530 storing therein a plurality of programs executable by the one or more processors, wherein the one or more processors are configured to perform the above-mentioned method when executing the plurality of programs.
[0202] In some embodiments, a computer program product is also provided, which includes multiple programs in the memory 2530, for example, and the multiple programs can be executed by the processor 2520 in the computing environment 2510 to perform the above method. For example, the computer program product may include a non-transitory computer-readable storage medium.
[0203] In some embodiments, the computing environment 2510 may be implemented using one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.
[0204] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the present disclosure. Many modifications, variations, and alternative embodiments will be apparent to one of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.
[0205] Unless otherwise specifically stated, the order of the steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but may be changed according to actual conditions. In addition, at least one step of the method according to the present disclosure may be adjusted, combined or deleted according to actual requirements.
[0206] The examples are chosen and described in order to explain the principles of the present disclosure and to enable other persons skilled in the art to understand the various embodiments of the present disclosure and to best utilize the basic principles and various embodiments with various modifications suitable for the specific use contemplated. Therefore, it should be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and modifications and other embodiments are intended to be included within the scope of the present disclosure.
Claims
1. A video processing method for performing intra-frame prediction on a video block, comprising: receiving, by a processor, reference samples from a video frame of a video including a video block; Determining, by a processor, a reference sample filter based on a size of each video block, wherein the reference sample filter is among a plurality of reference sample filters derived for different video block sizes; applying, by the processor, the determined reference sample filter to the received reference samples; and The processor performs intra prediction on the video block using the filtered reference samples.
2. The video processing method according to claim 1, wherein: The plurality of reference sample filters are trained offline and pre-programmed in the processor.
3. The video processing method according to claim 2, wherein: The plurality of reference sample filters are trained using training data including reference samples as input training data obtained by compressing original samples at different video block sizes using an encoder, and corresponding original samples as target training data.
4. The video processing method according to claim 3, wherein: For each different video block size, the encoder compresses the original samples using different quantization parameters (QP) to obtain reference samples.
5. The video processing method according to claim 4, wherein: The reference sample filters for each different video block size include a set of reference sample filters, each reference sample filter being derived using training data corresponding to each different QP.
6. The video processing method according to claim 4, wherein: The reference sample filters for each different video block size are derived together using the training data corresponding to all different QPs.
7. The video processing method according to claim 2, wherein: Each reference sample filter coefficient is pre-programmed in the processor, wherein the filter coefficient is derived by optimizing a least squares cost function.
8. The video processing method according to claim 1, wherein: The processor is in the encoder, wherein the video processing method further comprises: deriving, by a processor, the plurality of reference sample filters for different video block sizes; and The derived reference sample filter is signaled by the processor in the bitstream to the decoder.
9. The video processing method according to claim 1, wherein: The plurality of reference sample filters are adaptively derived by a processor for each video frame of the video, wherein the derived reference sample filters are signaled for each video frame.
10. The video processing method according to claim 9, wherein: The processor derives the plurality of reference sample filters for different video block sizes, further comprising: applying a default reference sample filter to the received reference samples; Performing a first pass compression on the video block using the filtered reference samples; and The plurality of reference sample filters are derived for different video block sizes using the first pass compression results.
11. The video processing method according to claim 10, wherein: The default reference sample filter has a filter coefficient of [12 1] / 4.
12. The video processing method according to claim 8, wherein: The intra prediction is a planar intra prediction.
13. The video processing method according to claim 10, wherein: Intra prediction is angular intra prediction.
14. The video processing method according to claim 13, wherein: Perform a first pass of compression on each video block, including: Performing angular intra prediction on the video block using the filtered reference samples to obtain prediction samples of the video block; and Position dependent intra prediction combination (PDPC) is performed on the prediction samples of the video block to obtain the intra prediction samples of the video block.
15. The video processing method according to claim 1, wherein: The processor is in the encoder, wherein the video processing method further comprises: A bitstream including encoded video blocks and signaling of the plurality of reference sample filters for different video block sizes is received.
16. The video processing method according to claim 1, wherein: A video block is a coding unit (CU).
17. A video processing apparatus for performing intra-frame prediction on a video block, comprising: a memory coupled to one or more processors, The one or more processors are configured to: receiving reference samples from a video frame of a video including a video block; Determine a reference sample filter based on the size of each video block, wherein the reference sample filter is among a plurality of reference sample filters derived for different video block sizes; applying the determined reference sample filter to the received reference samples; Intra prediction is performed on the video block using the filtered reference samples.
18. The video processing device according to claim 17, wherein: The plurality of reference sample filters are trained offline and pre-programmed in the one or more processors.
19. The video processing device according to claim 18, wherein: The plurality of reference sample filters are trained using training data including reference samples as input training data obtained by compressing original samples at different video block sizes using an encoder, and corresponding original samples as target training data.
20. The video processing device according to claim 19, wherein: For each different video block size, the encoder compresses the original samples using different quantization parameters (QP) to obtain reference samples.
21. The video processing device according to claim 20, wherein: The reference sample filters for each different video block size include a set of reference sample filters, each reference sample filter being derived using training data corresponding to each different QP.
22. The video processing device according to claim 20, wherein: The reference sample filters for each different video block size are derived together using the training data corresponding to all different QPs.
23. The video processing device according to claim 18, wherein: Filter coefficients for each reference sample filter are pre-programmed in the one or more processors, wherein the filter coefficients are derived by optimizing a least squares cost function.
24. The video processing device according to claim 17, wherein: The one or more processors are in an encoder, wherein the one or more processors are further configured to: deriving the plurality of reference sample filters for different video block sizes; and The derived reference sample filter is signaled to the decoder in the bitstream.
25. The video processing device according to claim 17, wherein: The plurality of reference sample filters are adaptively derived by the one or more processors for each video frame of a video, wherein the derived reference sample filters are signaled for each video frame.
26. The video processing device according to claim 24, wherein: To derive the plurality of reference sample filters for different video block sizes, the one or more processors are further configured to: applying a default reference sample filter to the received reference samples; Performing a first pass compression on the video block using the filtered reference samples; and The plurality of reference sample filters are derived for different video block sizes using the first pass compression results.
27. The video processing device according to claim 26, wherein: The default reference sample filter has a filter coefficient of [12 1] / 4.
28. The video processing device according to claim 24, wherein: The intra prediction is a planar intra prediction.
29. The video processing device according to claim 26, wherein: Intra prediction is angular intra prediction.
30. The video processing device according to claim 29, wherein: To perform a first pass compression on each video block, the one or more processors are further configured to: Performing angular intra prediction on the video block using the filtered reference samples to obtain prediction samples of the video block; and Position dependent intra prediction combination (PDPC) is performed on the prediction samples of the video block to obtain the intra prediction samples of the video block.
31. The video processing device according to claim 17, wherein: The one or more processors are in an encoder, wherein the one or more processors are further configured to: A bitstream including encoded video blocks and signaling of the plurality of reference sample filters for different video block sizes is received.
32. The video processing device according to claim 17, wherein: A video block is a coding unit (CU).
33. A non-transitory computer-readable storage medium having stored therein instructions, which when executed by one or more processors cause the one or more processors to perform a video processing method for performing intra-frame prediction on a video block, the video processing method comprising: receiving reference samples from a video frame of a video including a video block; Determine a reference sample filter based on the size of each video block, wherein the reference sample filter is among a plurality of reference sample filters derived for different video block sizes; applying the determined reference sample filter to the received reference samples; Intra prediction is performed on the video block using the filtered reference samples.
