Method and apparatus for cross-component prediction for video coding

Abstract

The present disclosure provides a method for decoding video data, comprising: obtaining a bitstream; obtaining, from the bitstream, information related to one or more convolutional cross-component models (CCMCs), each of the one or more CCMCs comprising a filter shape and a set of weighting coefficients corresponding to the filter shape for predicting each of chroma sample values of a video block from the bitstream based on a plurality of corresponding luma sample values; and decoding the video data based on the information related to the one or more CCCMs.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based upon and claims the benefit of priority from provisional application No. 63 / 431,359, filed on December 9, 2022, the entire contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present application relates to video coding and compression, and more particularly to a method and apparatus for improving coding efficiency of image / video blocks using cross-component prediction techniques. Background Art

[0004] Digital video is supported by a variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc. Electronic devices transmit and receive or otherwise transmit digital video data across a communication network, and / or store the digital video data on a storage device. Due to the limited bandwidth capacity of a communication network and the limited memory resources of a storage device, video codecs can be used to compress video data according to one or more video codec standards before the video data is transmitted or stored. For example, video codec standards include Versatile Video Codec (VVC), Joint Exploration Test Model (JEM), High Efficiency Video Codec (HEVC / H.265), Advanced Video Codec (AVC / H.264), Moving Picture Experts Group (MPEG) codecs, etc. Video codecs typically utilize prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that utilize the redundancy inherent in video data. Video codecs are intended to compress video data into a form using a lower bit rate while avoiding or minimizing degradation of video quality. Summary of the Invention

[0005] Embodiments of the present disclosure provide methods and apparatuses for improving encoding and decoding efficiency of image / video blocks to which cross-component prediction techniques are applied.

[0006] The following presents a simplified summary of one or more aspects of the present disclosure in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects and is neither intended to identify the primary or critical elements of all aspects nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that will be presented later.

[0007] According to one aspect of the present disclosure, a method for decoding video data is provided. The method includes: receiving an encoded block of luma samples for a block of the video data; decoding the encoded block of luma samples to obtain reconstructed luma samples for the block; classifying the luma samples of the block into one of a plurality of sample groups based on edge information of the luma samples, wherein the luma samples are obtained from one or more of the reconstructed luma samples to correspond to chroma samples of the block; and predicting the chroma samples by applying one of a plurality of linear prediction models corresponding to the classified sample groups to the luma samples.

[0008] According to one aspect of the present disclosure, a computer system is provided that includes one or more processors and one or more storage devices storing computer-executable instructions that, when executed, cause the one or more processors to perform operations including: receiving an encoded block of luma samples for a block of video data; decoding the encoded block of luma samples to obtain reconstructed luma samples for the block; classifying the luma samples for the block into one of a plurality of sample groups based on edge information of the luma samples, wherein the luma sample is obtained from one or more of the reconstructed luma samples to correspond to chroma samples of the block; and predicting the chroma samples by applying one of a plurality of linear prediction models corresponding to the classified sample group to the luma samples.

[0009] According to one aspect of the present disclosure, a method for video decoding using an edge-classified linear model (ELM) is provided. The method may include: receiving an encoded block of luma samples for a first block of a video signal; decoding the encoded block of luma samples to obtain reconstructed luma samples; classifying the reconstructed luma samples into a plurality of sample groups based on direction and strength of edge information; applying different linear prediction models to the reconstructed luma samples in different sample groups; and predicting chroma samples for the first block of the video signal based on the applied linear prediction models.

[0010] According to one aspect of the present disclosure, a method for video decoding using a filter-based linear model (FLM) is provided. The method may include: receiving an encoded block of luma samples for a first block of a video signal; decoding the encoded block of luma samples to obtain reconstructed luma samples; determining a luma sample region and a chroma sample region to derive a multiple linear regression (MLR) model; deriving the MLR model through pseudo-inverse matrix calculation; applying the MLR model to the reconstructed luma samples; and predicting chroma samples for the first block of the video signal based on the applied MLR model.

[0011] According to one aspect of the present disclosure, a method for video decoding using a gradient linear model (GLM) is provided. The method may include: receiving an encoded block of luma samples for a first block of a video signal; decoding the encoded block of luma samples to obtain reconstructed luma samples; exploiting the correlation between luma AC information and chroma intensity using the sample gradients; determining a luma sample region and a chroma sample region to derive a multiple linear regression (MLR) model; deriving the MLR model through pseudo-inverse matrix calculation; applying the MLR model to the reconstructed luma samples; and predicting chroma samples for the first block of the video signal based on the applied MLR model.

[0012] According to one aspect of the present disclosure, a method for video encoding and decoding without utilizing a downsampling process in a convolutional cross-component model (CCCM) is provided. The method may include: receiving an encoded block of luma samples for a first block of a video signal; decoding the encoded block of luma samples to obtain reconstructed luma samples; utilizing a non-subsampled luma reference value and / or different selections of a non-subsampled luma reference; determining a luma sample region and a chroma sample region to derive a convolutional cross-component model (CCCM); deriving CCCM parameters by LDL decomposition; applying the CCCM to the reconstructed luma samples; and predicting chroma samples for the first block of the video signal based on the applied CCCM.

[0013] According to one aspect of the present disclosure, a method for video encoding and decoding using LDL decomposition in a cross-component linear model (CCLM) / multi-model LM (MMLM) is provided. The method may include: receiving an encoded block of luma samples for a first block of a video signal; decoding the encoded block of luma samples to obtain reconstructed luma samples; determining a luma sample region and a chroma sample region to derive a cross-component linear model (CCLM) / multi-model LM (MMLM); deriving CCLM / MMLM parameters by LDL decomposition; applying the CCLM / MMLM to the reconstructed luma samples; and predicting chroma samples for the first block of the video signal based on the applied CCLM / MMLM.

[0014] According to one aspect of the present disclosure, a method for video encoding and decoding with a minimum sample limit in FLM / GLM / ELM / CCCM is provided. The method may include determining whether to apply the FLM / GLM / ELM / CCCM scheme in the intra prediction, wherein the number of samples is greater than or equal to a predefined number in the encoded block.

[0015] According to one aspect of the present disclosure, a method for performing video encoding and decoding using a non-subsampled luma reference value and a subsampled luma reference value in CCCM is provided.

[0016] According to one aspect of the present disclosure, a method for performing video encoding and decoding using a plurality of modes of a combination of FLM / GLM / ELM / CCCM / CCLM is provided.

[0017] According to one aspect of the present disclosure, a method for decoding video data is provided, comprising: obtaining a bitstream; obtaining information associated with one or more convolutional cross-component models (CCCMs) from the bitstream, wherein each of the one or more CCCMs includes a filter shape and a set of weighting coefficients corresponding to the filter shape, for predicting each of chroma sample values ​​of a video block from the bitstream based on multiple corresponding luminance sample values; and decoding the video data based on the information associated with the one or more CCCMs.

[0018] According to one aspect of the present disclosure, a method for encoding video data is provided, comprising: obtaining information associated with one or more convolutional cross-component models (CCCMs), wherein each of the one or more CCCMs comprises a filter shape and a set of weighting coefficients corresponding to the filter shape for predicting each of chroma sample values ​​of a video block based on a plurality of corresponding luminance sample values; encoding the video data based on the information associated with the one or more CCCMs; and obtaining a bitstream comprising the encoded video data and the information associated with the one or more CCCMs.

[0019] According to one aspect of the present disclosure, a method for decoding video data is provided, comprising: obtaining a video block from a bitstream; predicting each of chroma sample values ​​of the video block using a convolutional cross-component model (CCCM), wherein the CCCM includes a filter shape and a set of weighting coefficients determined based on external luma samples and external chroma samples of an external region of the video block, wherein the set of weighting coefficients includes a weighting coefficient corresponding to a distance value representing a distance between a center luma sample of the filter and a reference luma sample; and obtaining a predicted video block using the predicted internal chroma sample values.

[0020] According to one aspect of the present disclosure, a method for encoding video data is provided, comprising: obtaining a video block; predicting each of chroma sample values ​​of the video block using a convolutional cross-component model (CCCM), wherein the CCCM includes a filter shape and a set of weighting coefficients determined based on external luma samples and external chroma samples of an external region of the video block, wherein the set of weighting coefficients includes a weighting coefficient corresponding to a distance value representing a distance between a center luma sample of the filter and a reference luma sample; and generating a bitstream including the encoded video block by using the predicted chroma sample values.

[0021] According to one aspect of the present disclosure, a computer system is provided, comprising: one or more processors; and one or more storage devices storing computer-executable instructions, which, when executed, cause the one or more processors to perform the operations of the method of the present disclosure.

[0022] According to one aspect of the present disclosure, there is provided a computer program product storing computer-executable instructions that, when executed, cause one or more processors to perform the operations of the method of the present disclosure.

[0023] According to one aspect of the present disclosure, a computer-readable storage medium storing instructions is provided, which, when executed by a computing device having one or more processors, causes the one or more processors to perform the decoding method of the present disclosure and store a bit stream to be decoded by the decoding method of the present disclosure.

[0024] According to one aspect of the present disclosure, a computer-readable storage medium storing instructions is provided, which, when executed by a computing device having one or more processors, causes the one or more processors to perform the encoding method of the present disclosure and store a bit stream generated by the encoding method of the present disclosure.

[0025] According to one aspect of the present disclosure, a computer-readable medium storing a bitstream is provided, wherein the bitstream is to be decoded by performing the operations of the method of the present disclosure.

[0026] According to one aspect of the present disclosure, a computer-readable medium storing a bitstream is provided, wherein the bitstream is obtained by executing the operation of the method of the present disclosure.

[0027] It is to be understood that both the foregoing general description and the following detailed description are examples only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0029] Figure 1 is a block diagram illustrating an exemplary system for encoding and decoding video blocks according to some implementations of the present disclosure.

[0030] Figure 2 is a block diagram illustrating an exemplary video encoder according to some implementations of the present disclosure.

[0031] Figure 3 is a block diagram illustrating an exemplary video decoder according to some implementations of the present disclosure.

[0032] Figures 4A to 4E is a block diagram illustrating how a frame is recursively partitioned into multiple video blocks of different sizes and shapes, according to some implementations of the present disclosure.

[0033] Figure 5 Shown is an overall diagram of a block-based video encoder for VVC.

[0034] Figures 6A to 6E is a schematic diagram showing block partitioning in VVC.

[0035] Figure 7 Shows an overall diagram of a video decoder for VVC.

[0036] Figure 8 is a schematic diagram showing the positions of sample points used to derive α and β.

[0037] Figures 9A to 9C is a diagram illustrating examples of MDLM, MDLM_L, and MDLM_T.

[0038] Figure 10 is a schematic diagram showing an example of classifying adjacent samples into two groups.

[0039] Figure 11 Schematic diagram showing the inflection point T.

[0040] Figures 12A to 12B is a schematic diagram illustrating the effect of the slope adjustment parameter "u".

[0041] Figure 13 is a schematic diagram showing the used co-located reconstructed luminance samples.

[0042] Figure 14 is a schematic diagram showing the adjacent reconstruction samples used.

[0043] Figures 15A to 15D is a schematic diagram illustrating the process of intra mode derivation at the decoder side.

[0044] Figure 16 is a diagram illustrating an example of four reference lines adjacent to a prediction block.

[0045] Figure 17 is a schematic diagram showing the spatial part of a convolution filter.

[0046] Figure 18 is a schematic diagram showing the reference area (with its filling) used for deriving filter coefficients.

[0047] Figures 19A to 19B is a schematic diagram showing that chroma samples can be associated with multiple luma samples simultaneously.

[0048] Figure 20 is a schematic diagram showing that coefficients / offsets of multiple (eg, 6) luma samples relative to one chroma sample are trained to linearly predict chroma samples within a CU.

[0049] Figure 21 is a schematic diagram showing that different chroma types / color formats may have different predefined filter shapes / taps.

[0050] Figure 22 is a diagram showing that FLM can use only the top or left luma / chroma samples (extended) for parameter derivation.

[0051] Figure 23 is a schematic diagram showing that FLM can use different lines for parameter derivation.

[0052] Figures 24A to 24D is a schematic diagram showing a preliminary operation before applying the MLR model (GLM 1-tap / 2-tap).

[0053] Figure 25 is a diagram illustrating examples of different shapes / numbers of filter taps.

[0054] Figure 26 is a diagram illustrating examples of different shapes / numbers of filter taps.

[0055] Figures 27A to 27B is a diagram illustrating examples of different shapes / numbers of filter taps.

[0056] Figures 28A to 28G is a diagram illustrating examples of different filter tap sets.

[0057] Figures 29A to 29B is a schematic diagram illustrating 2-fold training for implicit filter shape derivation.

[0058] Figure 30is a schematic diagram showing non-subsampled luma samples.

[0059] Figure 31 A workflow of a method for decoding video data according to one or more aspects of the present disclosure is shown.

[0060] Figure 32 A workflow of a method for encoding video data according to one or more aspects of the present disclosure is shown.

[0061] Figure 33 A workflow of a method for decoding video data according to one or more aspects of the present disclosure is shown.

[0062] Figure 34 A workflow of a method for encoding video data according to one or more aspects of the present disclosure is shown.

[0063] Figure 35 is a schematic diagram illustrating a computing environment coupled with a user interface according to some implementations of the present disclosure. DETAILED DESCRIPTION

[0064] Reference will now be made in detail to specific implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, the subject matter presented herein may be implemented on many types of electronic devices with digital video capabilities.

[0065] It should be noted that the terms "first," "second," and the like used in the description, claims, and drawings of the present disclosure are used to distinguish between objects and are not used to describe any particular order or sequence. It should be understood that the terms used in this manner can be interchanged under appropriate conditions, such that the embodiments of the present disclosure described herein can be implemented in an order other than that shown in the drawings or described in the present disclosure.

[0066] Figure 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some implementations of the present disclosure. Figure 1As shown in , system 10 includes a source device 12 that generates and encodes video data to be decoded at a later time by a destination device 14. Source device 12 and destination device 14 may include any of a wide variety of electronic devices, including a cloud server, a server computer, a desktop or laptop computer, a tablet computer, a smartphone, a set-top box, a digital television, a camera, a display device, a digital media player, a video game console, a video streaming device, etc. In some implementations, source device 12 and destination device 14 are equipped with wireless communication capabilities.

[0067] In some implementations, the target device 14 may receive the encoded video data to be decoded via a link 16. The link 16 may include any type of communication medium or device capable of moving the encoded video data from the source device 12 to the target device 14. In one example, the link 16 may include a communication medium so that the source device 12 can send the encoded video data directly to the target device 14 in real time. The encoded video data may be modulated and sent to the target device 14 according to a communication standard such as a wireless communication protocol. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other device that can be used to facilitate communication from the source device 12 to the target device 14.

[0068] In some other implementations, the encoded video data can be sent from the output interface 22 to a storage device 32. The encoded video data in the storage device 32 can then be accessed by the target device 14 via the input interface 28. The storage device 32 can include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a digital versatile disc (DVD), a compact disc read-only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device 32 can correspond to a file server or another intermediate storage device that can hold the encoded video data generated by the source device 12. The target device 14 can access the stored video data from the storage device 32 via streaming or downloading. The file server can be any type of computer capable of storing and sending the encoded video data to the target device 14. Exemplary file servers include a network server (e.g., for a website), a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive. Target device 14 may access the encoded video data through any standard data connection suitable for accessing encoded video data stored on a file server, including a wireless channel (e.g., a Wireless Fidelity (Wi-Fi) connection), a wired connection (e.g., a Digital Subscriber Line (DSL), a cable modem, etc.), or a combination of both. The transmission of the encoded video data from storage device 32 may be a streaming transmission, a download transmission, or a combination of both.

[0069] like Figure 1 As shown in , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include a source such as a video capture device, for example, a video camera, a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as the source video, or a combination of such sources. As an example, if video source 18 is a camera of a security surveillance system, source device 12 and target device 14 may form a camera phone or video phone. However, the implementations described in this application may be generally applicable to video codecs and may be applied to wireless and / or wired applications.

[0070] The captured, pre-captured, or computer-generated video may be encoded by a video encoder 20. The encoded video data may be sent directly to the target device 14 via the output interface 22 of the source device 12. The encoded video data may also (or alternatively) be stored on a storage device 32 for later access by the target device 14 or other devices for decoding and / or playback. The output interface 22 may also include a modem and / or a transmitter.

[0071] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data sent over a communication medium, stored on a storage medium, or stored on a file server.

[0072] In some implementations, the target device 14 may include a display device 34, which may be an integrated display device or an external display device, configured to communicate with the target device 14. The display device 34 displays the decoded video data to a user and may include any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0073] The video encoder 20 and the video decoder 30 can operate according to proprietary or industry standards (such as VVC, HEVC, MPEG-4 Part 10, AVC or an extension of such a standard). It should be understood that the present application is not limited to a specific video encoding / decoding standard and can be applied to other video encoding / decoding standards. It is generally expected that the video encoder 20 of the source device 12 can be configured to encode the video data according to any one of these current or future standards. Similarly, it is also generally expected that the video decoder 30 of the target device 14 can be configured to decode the video data according to any one of these current or future standards.

[0074] The video encoder 20 and the video decoder 30 can each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When implemented partially in software, the electronic device can store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 can be included in one or more encoders or decoders, either of which can be integrated into the corresponding device as part of a combined encoder / decoder (CODEC).

[0075] In some implementations, at least a portion of the components of source device 12 (e.g., video source 18, as described below with reference to Figure 2 The video encoder 20 described herein or components included in the video encoder 20 and the output interface 22) and / or at least a portion of the components of the target device 14 (eg, the input interface 28, as described below with reference to Figure 3 The video decoder 30 described herein or the components included in the video decoder 30 and the display device 34 may be operated in a cloud computing service network that may provide software, platforms, and / or infrastructure, such as software as a service (SaaS), platform as a service (PaaS), or infrastructure as a service (IaaS). In some implementations, one or more components of the source device 12 and / or the target device 14 that are not included in the cloud computing service network may be provided in one or more client devices, and the one or more client devices may communicate with a server computer in the cloud computing service network via a wireless communication network (e.g., a cellular communication network, a short-range wireless communication network, or a global navigation satellite system (GNSS) communication network) or a wired communication network (e.g., a local area network (LAN) communication network or a power line communication (PLC) network). In an embodiment, at least a portion of the operations described herein may be implemented as a cloud-based service provided by one or more server computers, the one or more server computers being implemented by at least a portion of the components of the source device 12 and / or at least a portion of the components of the target device 14 in the cloud computing service network; and one or more other operations described herein may be implemented by one or more client devices. In some implementations, the cloud computing service network can be a private cloud, a public cloud, or a hybrid cloud. Terms such as "cloud," "cloud computing," and "cloud-based" may be used interchangeably herein, as appropriate, without departing from the scope of this disclosure. It should be understood that this disclosure is not limited to implementation within the aforementioned cloud computing service network. Rather, this disclosure may also be implemented within any other type of computing environment currently known or developed in the future.

[0076] Figure 2 is a block diagram illustrating an exemplary video encoder 20 according to some implementations described herein. Video encoder 20 can perform intra- and inter-frame predictive coding of video blocks within video frames. Intra-frame predictive coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame predictive coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that the term "frame" may be used as a synonym for the term "image" or "picture" in the field of video coding.

[0077] like Figure 2As shown in FIG, video encoder 20 includes video data memory 40, prediction processing unit 41, decoded picture buffer (DPB) 64, summer 50, transform processing unit 52, quantization unit 54, and entropy coding unit 56. Prediction processing unit 41 further includes motion estimation unit 42, motion compensation unit 44, segmentation unit 45, intra prediction processing unit 46, and intra block copy (BC) unit 48. In some implementations, video encoder 20 also includes inverse quantization unit 58 for video block reconstruction, inverse transform processing unit 60, and summer 62. A loop filter 63 (such as a deblocking filter) can be positioned between summer 62 and DPB 64 to filter block boundaries and thereby remove blocking artifacts from the reconstructed video. In addition to the deblocking filter, another loop filter such as a sample adaptive offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF) can also be used to filter the output of summer 62. It should be noted that, with respect to the CCSAO technique, the present application is not limited to the embodiments described herein, and alternatively, the present application may be applied to situations in which an offset for any other of the luma, Cb, and Cr chroma components is selected based on any of the luma, Cb, and Cr chroma components to modify the other components based on the selected offset. Furthermore, it should be noted that the first component referred to herein may be any one of the luma, Cb, and Cr chroma components, the second component referred to herein may be any other of the luma, Cb, and Cr chroma components, and the third component referred to herein may be the remaining luma, Cb, and Cr chroma components. In some examples, the loop filter may be omitted, and the decoded video block may be provided directly to the DPB 64 by the summer 62. The video encoder 20 may take the form of fixed or programmable hardware units, or may be divided between one or more of the fixed or programmable hardware units shown.

[0078] Video data memory 40 may store video data to be encoded by components of video encoder 20. The video data in video data memory 40 may be obtained, for example, from Figure 1 . The DPB 64 is a buffer that stores reference video data (e.g., reference frames or pictures) for use by the video encoder 20 (e.g., in intra-frame or inter-frame predictive coding modes) when encoding the video data. Video data memory 40 and DPB 64 can be formed from any of a variety of memory devices. In various examples, video data memory 40 can be on-chip with other components of the video encoder 20, or off-chip relative to those components.

[0079] like Figure 2 As shown in FIG, after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into slices, tiles (e.g., sets of video blocks), or other larger coding units (CUs) according to a predefined segmentation structure, such as a quadtree (QT) structure associated with the video data. A video frame is or can be considered a two-dimensional array or matrix of samples having sample values. The samples in the array may also be referred to as pixels or basic units of an image. The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the video frame. A video frame may be divided into multiple video blocks, for example, using QT segmentation. A video block is again or can be considered a two-dimensional array or matrix of samples having sample values, but with smaller dimensions than a video frame. The number of samples in the horizontal and vertical directions (or axes) of a video block defines the size of the video block. A video block may also be segmented into one or more block partitions or sub-blocks (which may again form blocks), for example, by iteratively using QT segmentation, binary tree (BT) segmentation, ternary tree (TT) segmentation, or any combination thereof. It should be noted that the term "block" or "video block" as used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. For example, with reference to HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU) or a transform unit (TU), and / or may be or correspond to a corresponding block, such as a coding tree block (CTB), a coding block (CB), a prediction block (PB) or a transform block (TB) and / or a subblock.

[0080] Prediction processing unit 41 may select one of multiple possible prediction coding modes for the current video block based on the error results (e.g., coding rate and distortion level), such as one of multiple intra-frame predictive coding modes or one of multiple inter-frame predictive coding modes. Prediction processing unit 41 may provide the resulting intra-frame or inter-frame predicted coded block to summer 50 to generate a residual block and to summer 62 to reconstruct the coded block for subsequent use as part of a reference frame. Prediction processing unit 41 also provides syntax elements such as motion vectors, intra-frame mode indicators, partition information, and other such syntax information to entropy coding unit 56.

[0081] To select an appropriate intra-frame predictive coding mode for the current video block, intra-frame prediction processing unit 46 within prediction processing unit 41 may perform intra-frame predictive coding of the current video block relative to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-frame predictive coding of the current video block relative to one or more predictive blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate coding mode for each block in the video data.

[0082] In some implementations, motion estimation unit 42 determines the inter-prediction mode for the current video frame by generating motion vectors according to a predetermined pattern within a sequence of video frames, the motion vectors indicating the displacement of video blocks within the current video frame relative to predictive blocks within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors that estimate the motion of video blocks. For example, a motion vector may indicate the displacement of a video block within the current video frame or picture relative to a predictive block within a reference frame relative to the current block being encoded within the current frame. The predetermined pattern may designate video frames in the sequence as P-frames or B-frames. Intra BC unit 48 may determine vectors (e.g., block vectors) for intra BC coding in a manner similar to the motion vectors determined by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine block vectors.

[0083] The predictive block for the video block may be or may correspond to a block of a reference frame or reference block that is considered to closely match the video block to be encoded in terms of pixel difference, which may be determined by sum of absolute difference (SAD), sum of squared difference (SSD), or other difference metrics. In some implementations, video encoder 20 may calculate values ​​for sub-integer pixel positions of the reference frame stored in DPB 64. For example, video encoder 20 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference frame. Thus, motion estimation unit 42 may perform a motion search relative to full pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

[0084] Motion estimation unit 42 calculates a motion vector for a video block in an inter-prediction encoded frame by comparing the position of the video block to the positions of predictive blocks of a reference frame selected from a first reference frame list (List 0) or a second reference frame list (List 1), each of which identifies one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to motion compensation unit 44 and then to entropy encoding unit 56.

[0085] Motion compensation performed by motion compensation unit 44 may involve extracting or generating a predictive block based on the motion vector determined by motion estimation unit 42. Upon receiving the motion vector for the current video block, motion compensation unit 44 may locate the predictive block pointed to by the motion vector in one of the reference frame lists, retrieve the predictive block from DPB 64, and forward the predictive block to summer 50. Summer 50 then forms a residual video block of pixel difference values ​​by subtracting the pixel values ​​of the predictive block provided by motion compensation unit 44 from the pixel values ​​of the current video block being encoded. The pixel difference values ​​forming the residual video block may include luma component differences, chroma component differences, or both. Motion compensation unit 44 may also generate syntax elements associated with the video block of the video frame for use by video decoder 30 when decoding the video block of the video frame. The syntax elements may include, for example, syntax elements defining the motion vector used to identify the predictive block, any flags indicating a prediction mode, or any other syntax information described herein. It should be noted that motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are illustrated separately for conceptual purposes.

[0086] In some implementations, the intra BC unit 48 may generate a vector and extract a predictive block in a manner similar to that described above in conjunction with the motion estimation unit 42 and the motion compensation unit 44, except that the predictive block is in the same frame as the current block being encoded and the vector is referred to as a block vector rather than a motion vector. Specifically, the intra BC unit 48 may determine the intra prediction mode to use to encode the current block. In some examples, the intra BC unit 48 may encode the current block using various intra prediction modes, for example, during separate encoding passes, and test their performance using rate-distortion analysis. The intra BC unit 48 may then select an appropriate intra prediction mode to use from among the various tested intra prediction modes and generate an intra mode indicator accordingly. For example, the intra BC unit 48 may use rate-distortion analysis to calculate rate-distortion values ​​for the various tested intra prediction modes and select the intra prediction mode with the best rate-distortion characteristics from among the tested modes as the appropriate intra prediction mode to use. Rate-distortion analysis typically determines the amount of distortion (or error) between an encoded block and the original, unencoded block that was encoded to produce the encoded block, as well as the bit rate (i.e., the number of bits) used to produce the encoded block. Intra BC unit 48 may calculate ratios based on the distortion and rates of the various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0087] In other examples, intra BC unit 48 may use, in whole or in part, motion estimation unit 42 and motion compensation unit 44 to perform such functions for intra BC prediction in accordance with implementations described herein. In either case, for intra block copying, the predictive block may be a block that is deemed to closely match the block to be encoded in terms of pixel difference, which may be determined by SAD, SSD, or other difference metrics, and identification of the predictive block may include calculation of values ​​for sub-integer pixel positions.

[0088] Regardless of whether the predictive block is from the same frame according to intra-frame prediction or a different frame according to inter-frame prediction, video encoder 20 can form a residual video block by subtracting the pixel values ​​of the predictive block from the pixel values ​​of the current video block being encoded, thereby forming pixel difference values. The pixel difference values ​​forming the residual video block may include both luma component differences and chroma component differences.

[0089] Intra-prediction processing unit 46 may intra-predict the current video block as an alternative to inter-prediction performed by motion estimation unit 42 and motion compensation unit 44 or intra-block copy prediction performed by intra BC unit 48, as described above. In particular, intra-prediction processing unit 46 may determine an intra-prediction mode to use to encode the current block. To do so, intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, a mode selection unit) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode into the bitstream.

[0090] After prediction processing unit 41 determines the predictive block for the current video block via inter-prediction or intra-prediction, summer 50 forms a residual video block by subtracting the predictive block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform.

[0091] Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of the matrix comprising the quantized transform coefficients. Alternatively, entropy coding unit 56 may perform the scan.

[0092] After quantization, entropy encoding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be encoded as Figure 1 is sent to the video decoder 30 as shown in Figure 1 , is archived in storage device 32 for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also entropy encode the motion vectors and other syntax elements for the current video frame being encoded.

[0093] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain to generate a reference block used for prediction of other video blocks. As mentioned above, motion compensation unit 44 may generate a motion compensated predictive block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the predictive block to calculate sub-integer pixel values ​​used in motion estimation.

[0094] Summer 62 adds the reconstructed residual block to the motion compensated predictive block produced by motion compensation unit 44 to produce a reference block for storage in DPB 64. The reference block may then be used by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 as a predictive block to inter-predict another video block in a subsequent video frame.

[0095] Figure 3 is a block diagram illustrating an exemplary video decoder 30 according to some implementations of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, a summer 90, and a DPB 92. The prediction processing unit 81 also includes a motion compensation unit 82, an intra prediction unit 84, and an intra BC unit 85. The video decoder 30 may perform operations generally similar to those described above with respect to the combined Figure 2 The encoding process is the reciprocal decoding process described for video encoder 20. For example, motion compensation unit 82 may generate prediction data based on the motion vector received from entropy decoding unit 80, and intra-prediction unit 84 may generate prediction data based on the intra-prediction mode indicator received from entropy decoding unit 80.

[0096] In some examples, units of the video decoder 30 may be tasked with performing implementations of the present disclosure. Furthermore, in some examples, implementations of the present disclosure may be divided among one or more of the units of the video decoder 30. For example, the intra BC unit 85 may perform implementations of the present disclosure alone or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functionality of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0097] The video data memory 79 may store video data (such as an encoded video bitstream) to be decoded by other components of the video decoder 30. The video data stored in the video data memory 79 may be obtained, for example, from a storage device 32, via a wired or wireless network communication of video data, or from a local video source (such as a camera) by accessing a physical data storage medium (e.g., a flash drive or hard disk). The video data memory 79 may include a coded picture buffer (CPB) that stores encoded video data from the encoded video bitstream. The DPB 92 of the video decoder 30 stores reference video data for use when decoding video data by the video decoder 30 (e.g., in intra-frame or inter-frame predictive coding modes). The video data memory 79 and the DPB 92 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For illustrative purposes, the video data memory 79 is shown in FIG. Figure 3 , video data memory 79 and DPB 92 are depicted as two separate components of video decoder 30. However, those skilled in the art will appreciate that video data memory 79 and DPB 92 can be provided by the same memory device or by separate memory devices. In some examples, video data memory 79 can be on-chip with other components of video decoder 30, or off-chip relative to those components.

[0098] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video frame and associated syntax elements. The video decoder 30 may receive syntax elements at the video frame level and / or the video block level. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors or intra-frame prediction mode indicators and other syntax elements to the prediction processing unit 81.

[0099] When a video frame is encoded as an intra-predictively coded (I) frame or for intra-coded predictive blocks in other types of frames, intra-prediction unit 84 of prediction processing unit 81 may generate prediction data for a video block of the current video frame based on a signaled intra-prediction mode and reference data from previously decoded blocks of the current frame.

[0100] When the video frame is encoded as an inter-frame predictive coded (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 generates one or more predictive blocks for the video block of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each of the predictive blocks can be generated from a reference frame within one of the reference frame lists. Video decoder 30 can construct the reference frame lists (List 0 and List 1) using a default construction technique based on the reference frames stored in DPB 92.

[0101] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a predictive block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The predictive block may be within the same reconstructed region of the picture as the current video block defined by video encoder 20.

[0102] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for a video block of the current video frame by parsing motion vectors and other syntax elements, and then uses the prediction information to generate a predictive block for the current video block being decoded. For example, motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode (e.g., intra or inter prediction) used to encode the video block of the video frame, the inter-prediction frame type (e.g., B or P), construction information for one or more of the reference frame lists for the frame, the motion vector for each inter-predictively encoded video block of the frame, the inter-prediction state for each inter-predictively encoded video block of the frame, and other information used to decode the video block in the current video frame.

[0103] Similarly, the intra BC unit 85 may use some of the received syntax elements (e.g., flags) to determine whether the current video block is predicted using the intra BC mode, construction information of which video blocks of the frame are within the reconstructed region and should be stored in the DPB 92, block vectors for each intra BC predicted video block of the frame, the intra BC prediction state of each intra BC predicted video block of the frame, and other information used to decode the video block in the current video frame.

[0104] Motion compensation unit 82 may also perform interpolation to calculate interpolated values ​​for sub-integer pixels of a reference block using interpolation filters as used during encoding of the video block by video encoder 20. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 from the received syntax elements and use the interpolation filters to produce the predictive block.

[0105] The quantized transform coefficients provided in the bitstream are dequantized by inverse quantization unit 86 and entropy decoded by entropy decoding unit 80 using the same quantization parameters calculated for each video block in the video frame by video encoder 20 to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to reconstruct the residual block in the pixel domain.

[0106] After the motion compensation unit 82 or the intra BC unit 85 generates a predictive block for the current video block based on the vector and other syntax elements, the summer 90 reconstructs the decoded video block for the current video block by summing the residual block from the inverse transform processing unit 88 and the corresponding predictive block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter 91 (such as a deblocking filter, an SAO filter, a CCSAO filter, and / or an ALF) can be positioned between the summer 90 and the DPB 92 to further process the decoded video block. In some examples, the loop filter 91 can be omitted and the decoded video block can be provided directly to the DPB 92 by the summer 90. The decoded video block in a given frame is then stored in the DPB 92, which stores reference frames that are used for subsequent motion compensation of the next video block. The DPB 92, or a memory device separate from the DPB 92, can also store the decoded video for later display on a display device such as Figure 1 is presented on a display device 34).

[0107] In a typical video encoding and decoding process, a video sequence typically consists of an ordered set of frames or pictures. Each frame may include three sample arrays, denoted as SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In other cases, a frame may be monochrome and therefore consist only of a two-dimensional array of luma samples.

[0108] like Figure 4AAs shown in , the video encoder 20 (or more specifically, the partition unit 45) generates an encoded representation of a frame by first partitioning the frame into a set of CTUs. A video frame may include an integer number of CTUs ordered consecutively in a raster scan order from left to right and from top to bottom. Each CTU is the largest logical coding unit, and the width and height of the CTU are signaled by the video encoder 20 in a sequence parameter set so that all CTUs in a video sequence have the same size of one of 128×128, 64×64, 32×32, and 16×16. However, it should be noted that the present application is not necessarily limited to a particular size. Figure 4B As shown in , each CTU may include one CTB for luma samples, two corresponding coding tree blocks for chroma samples, and syntax elements used to encode the samples of the coding tree blocks. The syntax elements describe the properties of different types of units of the coded pixel blocks and how the video sequence can be reconstructed at the video decoder 30, including inter-frame or intra-frame prediction, intra-frame prediction mode, motion vectors, and other parameters. In a monochrome picture or a picture with three separate color planes, a CTU may include a single coding tree block and syntax elements used to encode the samples of the coding tree block. The coding tree block may be an N×N sample block.

[0109] To achieve better performance, the video encoder 20 may recursively perform tree partitioning, such as binary tree partitioning, ternary tree partitioning, quadtree partitioning, or a combination thereof, on the coding tree block of the CTU and divide the CTU into smaller CUs. Figure 4C As depicted in FIG, a 64x64 CTU 400 is first divided into four smaller CUs, each with a block size of 32x32. Of the four smaller CUs, CU 410 and CU 420 are each divided into four CUs with a block size of 16x16. The two 16x16 CUs 430 and 440 are each further divided into four CUs with a block size of 8x8. Figure 4D Depicts the following Figure 4C The quadtree data structure of the final result of the CTU 400 partitioning process is depicted in FIG, where each leaf node of the quadtree corresponds to a CU of a corresponding size ranging from 32x32 to 8x8. Figure 4B In the CTU depicted in FIG, each CU may include a CB of luma samples and two corresponding coding blocks of chroma samples of the same size frame, and syntax elements used to encode the samples of the coding blocks. In a monochrome picture or a picture with three separate color planes, a CU may include a single coding block and syntax structures used to encode the samples of the coding block. It should be noted that Figure 4C and Figure 4DThe quadtree partitioning depicted in FIG is for illustrative purposes only, and a CTU can be partitioned into CUs to adapt to varying local characteristics based on quadtree / ternary / binary tree partitioning. In multiple types of tree structures, a CTU is partitioned by a quadtree structure, and each quadtree leaf CU can be further partitioned by a binary tree and a ternary tree structure. Figure 4E As shown in , there are five possible partitioning types of a coding block with width W and height H, namely, quadruple partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal ternary partitioning, and vertical ternary partitioning.

[0110] In some implementations, the video encoder 20 may further partition the coding block of a CU into one or more MxN PBs. A PB is a rectangular (square or non-square) block of samples on which the same prediction (inter or intra) is applied. A PU of a CU may include a PB of luma samples, two corresponding PBs of chroma samples, and syntax elements used to predict the PBs. In a monochrome picture or a picture with three separate color planes, a PU may include a single PB and a syntax structure used to predict the PB. The video encoder 20 may generate predictive luma, Cb, and Cr blocks for the luma, Cb, and Cr PBs of each PU of the CU.

[0111] Video encoder 20 may use intra prediction or inter prediction to generate the predictive blocks of a PU. If video encoder 20 uses intra prediction to generate the predictive blocks of a PU, video encoder 20 may generate the predictive blocks of the PU based on decoded samples of a frame associated with the PU. If video encoder 20 uses inter prediction to generate the predictive blocks of a PU, video encoder 20 may generate the predictive blocks of the PU based on decoded samples of one or more frames different from the frame associated with the PU.

[0112] After the video encoder 20 generates the predictive luma, Cb, and Cr blocks for one or more PUs of a CU, the video encoder 20 may generate a luma residual block for the CU by subtracting the predictive luma block of the CU from its original luma coding block, such that each sample in the luma residual block of the CU indicates the difference between a luma sample in one of the predictive luma blocks of the CU and a corresponding sample in the original luma coding block of the CU. Similarly, the video encoder 20 may generate a Cb residual block and a Cr residual block for the CU, respectively, such that each sample in the Cb residual block of the CU indicates the difference between a Cb sample in one of the predictive Cb blocks of the CU and a corresponding sample in the original Cb coding block of the CU, and each sample in the Cr residual block of the CU may indicate the difference between a Cr sample in one of the predictive Cr blocks of the CU and a corresponding sample in the original Cr coding block of the CU.

[0113] In addition, if Figure 4CAs shown in , the video encoder 20 can use quadtree partitioning to decompose the luma, Cb and Cr residual blocks of a CU into one or more luma, Cb and Cr transform blocks, respectively. A transform block is a rectangular (square or non-square) block of samples on which the same transform is applied. A TU of a CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements used to transform the transform block samples. Therefore, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In a monochrome picture or a picture with three separate color planes, a TU may include a single transform block and a syntax structure used to transform the samples of the transform block.

[0114] Video encoder 20 may apply one or more transforms to the luma transform block of a TU to generate a luma coefficient block for the TU. A coefficient block may be a two-dimensional array of transform coefficients. A transform coefficient may be a scalar. Video encoder 20 may apply one or more transforms to the Cb transform block of a TU to generate a Cb coefficient block for the TU. Video encoder 20 may apply one or more transforms to the Cr transform block of a TU to generate a Cr coefficient block for the TU.

[0115] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), video encoder 20 may quantize the coefficient block. Quantization generally refers to a process in which transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. After video encoder 20 quantizes the coefficient block, video encoder 20 may entropy encode syntax elements indicating the quantized transform coefficients. For example, video encoder 20 may perform CABAC on the syntax elements indicating the quantized transform coefficients. Finally, video encoder 20 may output a bitstream comprising a sequence of bits forming a representation of the encoded frame and associated data, which is stored in storage device 32 or sent to target device 14.

[0116] After receiving the bitstream generated by the video encoder 20, the video decoder 30 can parse the bitstream to obtain syntax elements from the bitstream. The video decoder 30 can reconstruct a frame of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing video data is generally the inverse of the encoding process performed by the video encoder 20. For example, the video decoder 30 can perform an inverse transform on the coefficient blocks associated with the TUs of the current CU to reconstruct the residual blocks associated with the TUs of the current CU. The video decoder 30 also reconstructs the coding blocks of the current CU by adding samples of the predictive blocks for the PUs of the current CU to corresponding samples of the transform blocks of the TUs of the current CU. After reconstructing the coding blocks of each CU of the frame, the video decoder 30 can reconstruct the frame.

[0117] As mentioned above, video codecs primarily implement video compression using two modes: intra-frame prediction (or intra-prediction) and inter-frame prediction (or inter-prediction). It should be noted that IBC can be considered intra-frame prediction or a third mode. Between the two modes, inter-frame prediction contributes more to codec efficiency than intra-frame prediction because it uses motion vectors to predict the current video block from a reference video block.

[0118] However, with ever-improving video data capture techniques and finer video block sizes for preserving details in video data, the amount of data required for the motion vector representing the current frame has also increased significantly. One way to overcome this challenge is to benefit from the fact that not only do groups of neighboring CUs in both the spatial and temporal domains have similar video data for prediction purposes, but the motion vectors between these neighboring CUs are also similar. Therefore, it is possible to use the motion information of spatially neighboring CUs and / or temporally co-located CUs as an approximation of the motion information (e.g., motion vector) of the current CU by exploiting the spatial and temporal correlation of the current CU, which is also referred to as the "motion vector predictor (MVP)" of the current CU.

[0119] Instead, it will be combined as above Figure 2 The actual motion vector of the current CU determined by the motion estimation unit 42 is encoded into the video bitstream, and the motion vector prediction value of the current CU is subtracted from the actual motion vector of the current CU to generate a motion vector difference (MVD) of the current CU. By doing so, the motion vector determined by the motion estimation unit 42 for each CU of the frame does not need to be encoded into the video bitstream, and the amount of data used to represent motion information in the video bitstream can be significantly reduced.

[0120] Similar to the process of selecting a predictive block in a reference frame during inter-frame prediction of a code block, both the video encoder 20 and the video decoder 30 need to adopt a set of rules to construct a motion vector candidate list (also called a "merge list") for the current CU using those potential candidate motion vectors associated with the current CU's spatially neighboring CUs and / or temporally co-located CUs, and then select one member from the motion vector candidate list as the motion vector predictor for the current CU. By doing so, the motion vector candidate list itself does not need to be sent from the video encoder 20 to the video decoder 30, and the index of the selected motion vector predictor within the motion vector candidate list is sufficient for the video encoder 20 and the video decoder 30 to use the same motion vector predictor within the motion vector candidate list to encode and decode the current CU.

[0121] introduce

[0122] Various video codec technologies can be used to compress video data. Video coding and decoding are performed according to one or more video codec standards. For example, video codec standards include Versatile Video Codec (VVC), High Efficiency Video Codec (H.265 / HEVC), Advanced Video Codec (H.264 / AVC), Moving Picture Experts Group (MPEG) codec, etc. Video codecs typically utilize prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that utilize redundancy in video images or sequences. An important goal of video coding and decoding technology is to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation of video quality.

[0123] The first version of the VVC standard was completed in July 2020, which provides approximately 50% bitrate savings or equivalent perceptual quality compared to the existing generation video codec standard HEVC. Although the VVC standard provides significant codec improvements over its predecessor, there is evidence that additional codec tools can be used to achieve excellent codec efficiency. Recently, the Joint Video Exploration Team (JVET), in collaboration with ITU-T VCEG and ISO / IEC MPEG, began exploring advanced technologies that can achieve substantial enhancements in codec efficiency over VVC. In April 2021, a software code base called the Enhanced Compression Model (ECM) was established for future video codec exploration work. The ECM reference software is based on the VVC Test Model (VTM) developed by JVET for VVC, in which several existing modules (e.g., intra / inter prediction, transform, loop filtering, etc.) are further extended and / or improved. In the future, any new codec tools beyond the VVC standard can be integrated into the ECM platform and tested using the JVET Common Test Conditions (CTC).

[0124] Similar to all the aforementioned video codec standards, ECM is built on a block-based hybrid video codec framework. Figure 5 The block diagram of a hybrid video coding system based on a general block is shown. The input video signal is processed block by block (called a coding unit (CU)). In ECM-1.0, a CU can be up to 128x128 pixels. However, like VVC, a coding tree unit (CTU) is split into CUs to adapt to varying local characteristics based on quadtree / binarytree / ternarytree. In the multi-type tree structure, a CTU is first partitioned by a quadtree structure. Then, each quadtree leaf node can be further partitioned by a binary tree and a ternary tree structure. As shown in FIG. Figure 6A 、 Figure 6B 、 Figure 6C 、 Figure 6D and Figure 6E As shown, there are five types of splits: quadruple split, vertical binary split, horizontal binary split, vertical extended quadruple split, and horizontal extended quadruple split.

[0125] exist Figure 5In

[15] , spatial prediction and / or temporal prediction can be performed. Spatial prediction (or "intra-frame prediction") uses pixels from samples of coded neighboring blocks (which are called reference samples) in the same video picture / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also known as "inter-frame prediction" or "motion compensated prediction") uses reconstructed pixels from coded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs) indicating the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index is additionally sent, which is used to identify which reference picture in the reference picture store stores the temporal prediction signal. After spatial and / or temporal prediction, the mode decision block in the encoder selects the best prediction mode, for example based on a rate-distortion optimization method. The prediction block is then subtracted from the current video block; and the prediction residual is decorrelated and quantized using a transform. The quantized residual coefficients are dequantized and inverse transformed to form a reconstructed residual, which is then added back into the prediction block to form the reconstructed signal of the CU. Further loop filtering (such as a deblocking filter, sample adaptive offset (SAO), and adaptive loop filter (ALF)) may be applied to the reconstructed CU before it is placed in a reference picture store and used to encode future video blocks. To form the output video bitstream, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit to be further compressed and packed to form the bitstream. It should be noted that the term "block" or "video block" as used herein can be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. For example, with reference to HEVC and VVC, a block or video block can be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU), and / or can be or correspond to a corresponding block, such as a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB) and / or a sub-block.

[0126] Figure 7 A general block diagram of a block-based video decoder is shown. The video bitstream is first entropy decoded at the entropy decoding unit. The coding mode and prediction information are sent to the spatial prediction unit (if intra-coded) or the temporal prediction unit (if inter-coded) to form a prediction block. The residual transform coefficients are sent to the inverse quantization unit and the inverse transform unit to reconstruct the residual block. The prediction block and the residual block are then added together. The reconstructed block can further undergo loop filtering before being stored in the reference picture store. The reconstructed video in the reference picture store is then sent out to drive the display device and is used to predict future video blocks.

[0127] The primary focus of this disclosure is to further enhance the codec efficiency of the codec tools used for cross-component prediction in ECM, the Cross-Component Linear Model (CCLM). Below, we briefly review some relevant codec tools in ECM. This is followed by a discussion of some shortcomings in existing CCLM designs. Finally, we provide solutions to improve existing CCLM prediction designs.

[0128] Cross-component linear model prediction

[0129] To reduce cross-component redundancy, the Cross-Component Linear Model (CCLM) prediction mode is used in VVC, for which chroma samples are predicted based on the reconstructed luma samples of the same CU by using the following linear model:

[0130] pred C (i,j)=α·rec L ′(i,j)+β (1)

[0131] where pred C (i, j) represents the predicted chroma sample in the CU, and rec L ′(i,j) represents the reconstructed luminance sample rec of the same CU L The downsampled reconstructed luma samples are obtained by downsampling (i, j). The above α and β are linear model parameters, which are derived from up to four adjacent chroma samples and their corresponding downsampled luma samples, which can be called adjacent luma-chroma sample pairs. Assuming that the current chroma block has a size of W×H, W' and H' are obtained as follows:

[0132] – When LM mode is applied, W'=W, H'=H;

[0133] – When LM-A mode is applied, W'=W+H;

[0134] – When LM-L mode is applied, H'=H+W;

[0135] In LM mode, both the upper and left samples of the CU are used to calculate the linear model coefficients; in LM_A mode, only the upper samples of the CU are used to calculate the linear model coefficients; and in LM_L mode, only the left samples of the CU are used to calculate the linear model coefficients.

[0136] If the positions of the upper neighboring samples of the chroma block are denoted as S[0, -1] ... S[W'-1, -1], and the positions of the left neighboring samples of the chroma block are denoted as S[-1, 0] ... S[-1, H'-1], the positions of the four neighboring chroma samples are selected as follows:

[0137] – When LM mode is applied and both the upper and left adjacent samples are available, S[W' / 4, -1], S[3*W' / 4, -1], S[-1, H' / 4], S[-1, 3*H' / 4] are selected as the positions of the four adjacent chroma samples;

[0138] – When LM-A mode is applied or only upper adjacent samples are available, S[W' / 8, -1], S[3*W' / 8, -1], S[5*W' / 8, -1], S[7*W' / 8, -1] are selected as the positions of the four adjacent chroma samples;

[0139] – When LM-L mode is applied or only left adjacent samples are available, S[-1, H' / 8], S[-1, 3*H' / 8], S[-1, 5*H' / 8], S[-1, 7*H' / 8] are selected as the positions of the four adjacent chroma samples.

[0140] The four adjacent luminance samples corresponding to the selected position are obtained by downsampling operation, and the obtained four adjacent luminance samples are compared four times to find the two larger values: x 0 A and x 1 A , and two smaller values: x 0 B and x 1 B The chrominance sample values ​​corresponding to the two larger and two smaller values ​​are represented as y 0 A 、y 1 A 、y 0 B and y 1 B Then, X a 、X b 、Y a and Y b is derived as:

[0141] X a =(x 0 A +x 1 A +1)>>1;

[0142] X b =(x 0 B +x 1 B +1)>>1;

[0143] Y a =(y 0A +y 1 A +1)>>1;

[0144] Y b =(y 0 B +y 1 B +1)>>1 (2)

[0145] Finally, the linear model parameters α and β are obtained according to the following equations.

[0146]

[0147] β=Y b -α·X b (4)

[0148] Figure 8 Examples of the positions of the left and top samples involved in the CCLM mode and the samples of the current block are shown, including the positions of the left and top samples of the N×N chroma block in the CU and the left and top samples of the 2N×2N luminance block in the CU.

[0149] The division operation for calculating the parameter α is implemented using a lookup table. To reduce the memory required to store the table, the diff value (the difference between the maximum and minimum values) and the parameter α are represented by exponential notation. For example, diff is approximated as a 4-bit significand and an exponent. Thus, for 16 values ​​of the significand as follows, the table for 1 / diff is reduced to 16 elements:

[0150] DivTable[]={0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} (5)

[0151] This will have the benefit of reducing both the complexity of the calculations and the memory size required to store the required tables.

[0152] In addition to the upper and left templates being used together to calculate the linear model coefficients, they can also be used alternatively in the other two LM modes (referred to as LM_A and LM_L modes).

[0153] In LM_T mode, only the upper template is used to calculate the linear model coefficients. To obtain more samples, the upper template is expanded to (W+H) samples. In LM_L mode, only the left template is used to calculate the linear model coefficients. To obtain more samples, the left template is expanded to (H+W) samples.

[0154] In LM_LT mode, the linear model coefficients are calculated using the left and upper templates.

[0155] To match the chroma sample locations of a 4:2:0 video sequence, two types of downsampling filters are applied to the luma samples to achieve a 2 to 1 downsampling ratio in both the horizontal and vertical directions. The choice of downsampling filter is specified by the SPS level flag.

[0156] The two downsampling filters are as follows, corresponding to "Type 0" and "Type 2" content respectively.

[0157]

[0158] Note that when the above reference line is located at a CTU boundary, only one luma line (common line buffer in intra prediction) is used to get the downsampled luma samples.

[0159] This parameter calculation is performed as part of the decoding process, not just as an encoder search operation. Therefore, no syntax is used to convey the values ​​of α and β to the decoder.

[0160] For chroma intra mode coding, a total of 8 intra modes are allowed for chroma intra mode coding. These modes include five traditional intra modes and three cross-component linear model modes (CCLM, LM_A and LM_L). The chroma mode signaling and derivation process are shown in Table 1. Chroma mode coding depends directly on the intra prediction mode of the corresponding luminance block. Since separate block partitioning structures for luminance components and chrominance components are enabled in I slices, one chroma block can correspond to multiple luminance blocks. Therefore, for the chroma DM mode, the intra prediction mode of the corresponding luminance block covering the center position of the current chroma block is directly inherited.

[0161] Table 1 - Chroma prediction modes derived from luma mode when CCLM is enabled

[0162]

[0163] Regardless of the value of sps_cclm_enabled_flag, a single binarization table is used, as shown in Table 2.

[0164] Table 2 - Unified binarization table for chroma prediction mode

[0165] The value of intra_chroma_pred_mode binary bit string 4 00 0 0100 1 0101 2 0110 3 0111 5 10 6 110 7 111

[0166] In Table 2, the first binary bit indicates whether it is normal mode (0) or LM mode (1). If it is LM mode, the next binary bit indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, the next binary bit indicates whether it is LM_L (0) or LM_A (1). For this case, when sps_cclm_enabled_flag is 0, the first binary bit of the binarization table for the corresponding intra_chroma_pred_mode can be discarded before entropy coding. Or, in other words, the first binary bit is inferred to be 0 and therefore not coded. This single binarization table is used for both the cases where sps_cclm_enabled_flag is equal to 0 and 1. The first two binary bits in Table 2 are context coded using their own context model, and the remaining binary bits are bypass coded.

[0167] Additionally, to reduce luma-chroma latency in dual trees, when a 64x64 luma coding tree node is partitioned with no split (and ISP is not used for 64x64 CUs) or QT, chroma CUs in 32x32 / 32x16 chroma coding tree nodes are allowed to use CCLM in the following manner:

[0168] – If a 32x32 chroma node is not split or is split by a partitioned QT, all chroma CUs in the 32x32 node can use CCLM

[0169] If a 32x32 chroma node is split with horizontal BT and the 32x16 child nodes are not split or use vertical BT split, all chroma CUs in the 32x16 chroma node can use CCLM.

[0170] Under all other luma and chroma coding tree split conditions, CCLM is not allowed for chroma CUs.

[0171] During the development of the ECM, the simplified derivation of α and β (min-max approximation) was removed. Instead, the model parameters α and β were derived using a linear least squares solution between the causal reconstructed data of the downsampled luma samples and the causal chroma samples.

[0172]

[0173] Among them, Rec C (i) and Rec' L (i) indicates the reconstructed chroma samples and downsampled reconstructed luminance samples around the target block, and I indicates the total number of samples of adjacent data.

[0174] The LM_A and LM_L models are also called multidirectional linear models (MDLM). Figure 9AAn example of the MDLM working is shown when the block content cannot be predicted from the L-shaped reconstructed region. Figure 9B MDLM_L is shown using only the left reconstructed samples to derive CCLM parameters. Figure 9C MDLM_T is shown using only the top reconstruction samples to derive CCLM parameters.

[0175] JCTVC-C206: Integration

[0176] The integration of the least mean square (LMS) discussed above (see equations (8)-(9)) has been proposed as an improvement to CCLM. The initial integration design of the LMS CCLM was first proposed in JCTVC-C206. The method was then improved through a series of simplifications, including the reduction of the α accuracy n α JCTVC-F0233 / I0178 that reduced the maximum multiplier bit width from 13 to 7, JCTVC-I0151 that reduced the maximum multiplier bit width, and JCTVC-H0490 / I0166 that reduced the division LUT entries from 64 to 32, finally leading to the ECM LMS version.

[0177] Basic Algorithm

[0178] As discussed in equation (1), the integrated design utilizes a linear relationship to model the correlation of the luma and chroma signals. The chroma values ​​are predicted from the reconstructed luma values ​​of the co-located blocks.

[0179] The luminance component and the chrominance component have different sampling rates in YUV420 sampling. The sampling rate of the chrominance component is half of the sampling rate of the luminance component and has a phase difference of 0.5 pixels in the vertical direction. The reconstructed luminance needs to be downsampled in the vertical direction and subsampled in the horizontal direction to match the size of the chrominance signal. For example, downsampling can be achieved by:

[0180] Rec L ′(i,j)=(rec L (2i,2j)+rec L (2i,2j+1))>>1 (10)

[0181] Integer implementation

[0182] Floating-point operations are required to calculate the linear model parameter α in equation (8) to maintain high data accuracy. And when α is represented by floating-point values, floating-point multiplication is involved in equation (1). In this section, an integer implementation of the algorithm is designed. Specifically, using n α The fractional part of the parameter α is quantized with a bit data accuracy. The value of the parameter α is obtained by expanding and rounding the integer value α' and a'=a×(1<<n α). Then, the linear model of equation (1) is changed to:

[0183] pred C [x,y]=(α'·Rec L '[x,y]>>n α )+β' (11)

[0184] where β′ is the rounded value of floating point β, and α′ can be calculated as follows.

[0185]

[0186] It is proposed to replace the division operation of equation (12) by table lookup and multiplication. First, A2 is reduced to reduce the table size. A1 is also reduced to avoid product overflow. Then, in A2, only the The most significant bit of the value is defined, while the other bits are set to zero. The approximate value A2' can be calculated as:

[0187]

[0188] where [...] means rounding operation, and can be calculated as:

[0189]

[0190] where bdepth(A2) refers to the bit depth of value A2.

[0191] Perform the same operation on A1 as follows:

[0192]

[0193] Considering the quantized representations of A1 and A2, equation (12) can be rewritten as follows.

[0194]

[0195] in is represented by a length of to avoid division.

[0196] In the simulation, the constant parameters are set as:

[0197] ·n α Equal to 13, this value is a compromise between data accuracy and computational cost.

[0198] · Equal to 6, resulting in a lookup table size of 64. When bdepth(A2)<6 (e.g., A2<32), the table size can be further reduced to 32 by dilation.

[0199] ·n table Equal to 15, resulting in a 16-bit data representation of the table elements.

[0200] · Set to 15 to avoid product overflow and maintain 16-bit multiplication.

[0201] Finally, limit α' to [-2 -15 ,2 15 -1] to maintain 16-bit multiplication in equation (11). With this clipping, when n α When it is equal to 13, the actual a value is limited to [-4, 4), which is useful for preventing error amplification.

[0202] Using the calculated parameter α', the parameter β' is calculated as follows:

[0203]

[0204] Here, the division in the above equation can be simply replaced by a shift since the value I is a power of 2.

[0205] JCTVC-I0166: Simplified parameter calculation

[0206] Similar to the discussion above with respect to equation (1), in HM6.0, an intra prediction mode called LM is applied to predict the chroma PU based on a linear model using reconstruction of the co-located luma PU. The parameters of the linear model consist of the slope (a>>k) and y-intercept (b) derived from neighboring luma and chroma pixels using a least mean square solution. The values ​​of the predicted samples predSamples[x,y] are derived as follows, where x,y=0...nS-1, where nS specifies the block size of the current chroma PU:

[0207] predSamples[x,y]=Clip1 C (((p Y '[x,y]*a)>>k)+b), where, x,y=0...nS-1 (17)

[0208] Among them, P Y '[x,y] is the reconstructed pixel from the corresponding luminance component. When the coordinates x and y are equal to or greater than 0, P Y ' is the reconstructed pixel from the same luma PU. When x or y is less than 0, P Y ' is the reconstructed neighboring pixel of the same-position luma PU.

[0209] Some intermediate variables L, C, LL, LC, k2 and k3 in the derivation process are derived as follows:

[0210]

[0211] k2=Log2((2*nS)>>k3) (18-5)

[0212] k3=Max(0,BitDepth C +Log2(nS)-14) (18-6)

[0213] Therefore, the variables a, b, and k can be derived as:

[0214] a1=(LC< <k2)–L*C (19-1)

[0215] a2=(LL< <k2)–L*L (19-2)

[0216] k1=Max(0,Log2(abs(a2))-5)–Max(0,Log2(abs(a1))-14)+2(19-3)

[0217] a1s=a1>>Max(0,Log2(abs(a1))-14) (19-4)

[0218] a2s=abs(a2>>Max(0,Log2(abs(a2))-5)) (19-5)

[0219] a3=a2s<1?0:Clip3(-2 15 ,2 15 -1,a1s*lmDiv+(1<<(k1-1))>>k1)(19-6)

[0220] a=a3>>Max(0,Log2(abs(a3))-6) (19-7)

[0221] k=13–Max(0,Log2(abs(a))-6) (19-8)

[0222] b=(L–((a*C)>>k1)+(1<<(k2-1)))>>k2 (19-9)

[0223] where lmDiv is specified in a 63-entry lookup table (i.e., Table 3), which is generated online as follows:

[0224] lmDiv(a2s)=((1<<15)+a2s / 2) / a2s (20)

[0225] Table 3 - lmDiv Specifications

[0226] a2s 1 2 3 4 5 6 7 8 9 10 11 12 13 lmDiv 32768 16384 10923 8192 6554 5461 4681 4096 3641 3277 2979 2731 2521 a2s 14 15 16 17 18 19 20 21 22 23 24 25 26 lmDiv 2341 2185 2048 1928 1820 1725 1638 1560 1489 1425 1365 1311 1260 a2s 27 28 29 30 31 32 33 34 35 36 37 38 39 lmDiv 1214 1170 1130 1092 1057 1024 993 964 936 910 886 862 840 a2s 40 41 42 43 44 45 46 47 48 49 50 51 52 lmDiv 819 799 780 762 745 728 712 697 683 669 655 643 630 a2s 53 54 55 56 57 58 59 60 61 62 63 64 lmDiv 618 607 596 585 575 565 555 546 537 529 520 512

[0227] In equation (19-6), a1s is a 16-bit signed integer and lmDiv is a 16-bit unsigned integer. Therefore, a 16-bit multiplier and 16-bit storage are required. It is proposed to reduce the bit depth of the multiplier to the internal bit depth and reduce the size of the lookup table, as described in detail below.

[0228] Reduced multiplier bit depth

[0229] The bit depth of a1s is reduced to the internal bit depth by changing equation (19-4) to the following:

[0230] a1s=a1>>Max(0,Log2(abs(a1))–(BitDepth C –2)) (21)

[0231] The value of lmDiv with internal bit depth is implemented using the following equation (22) and stored in a lookup table:

[0232] lmDiv(a2s)=((1<<(BitDepth C -1))+a2s / 2) / a2s (22)

[0233] Table 4 shows an example of an internal bit depth of 10.

[0234] Table 4 - Specification of lmDiv with internal bit depth equal to 10

[0235] a2s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 lmDiv 512 256 171 128 102 85 73 64 57 51 47 43 39 37 34 32 a2s 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 lmDiv 30 28 27 26 24 23 22 21 20 20 19 18 18 17 17 16 a2s 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 lmDiv 16 15 15 14 14 13 13 13 12 12 12 12 11 11 11 11 a2s 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 lmDiv 10 10 10 10 10 9 9 9 9 9 9 9 8 8 8

[0236] Also modify equations (19-3) and (19-8) as follows:

[0237] k1=Max(0,Log2(abs(a2))-5)–Max(0,Log2(abs(a1))–(BitDepth C –2)) (23-1)

[0238] k = BitDepth C –1–Max(0,Log2(abs(a))-6) (23-2)

[0239] Reduced lookup table entries

[0240] It is also proposed to reduce the number of entries from 63 to 32 and the bits per entry from 16 to 10, as shown in Table 5. By doing so, a memory saving of almost 70% can be achieved. The corresponding changes to Equation (19-6), Equation (20), and Equation (19-8) are as follows:

[0241] a3=a2s<32?0:Clip3(-2 15 ,2 15 -1,a1s*lmDiv+(1<<(k1-1))>>k1) (24-1)

[0242] lmDiv(a2s)=((1<<(BitDepth C +4))+a2s / 2) / a2s (24-2)

[0243] k = BitDepth C +4–Max(0,Log2(abs(a))-6) (24-3)

[0244] Table 5 - Specification of lmDiv with internal bit depth equal to 10

[0245] a2s 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 lmDiv 512 496 482 468 455 443 431 420 410 400 390 381 372 364 356 349 a2s 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 lmDiv 341 334 328 321 315 309 303 298 293 287 282 278 273 269 264 260

[0246] Multi-model linear model prediction

[0247] In ECM-1.0, the Multi-Model LM (MMLM) prediction mode is proposed, for which chrominance samples are predicted based on the reconstructed luma samples of the same CU by using the following two linear models:

[0248]

[0249] where pred C (i, j) represents the predicted chroma sample in the CU, and rec L ′(i, j) represents the downsampled reconstructed luma sample of the same CU. Threshold is calculated as the average of adjacent reconstructed luma samples. Figure 10 An example of classifying adjacent samples into two groups based on the value Threshold is shown. For each group, the parameters αi and βi (where i is equal to 1 and 2 respectively) are obtained from two samples from the inside of the group (which is the minimum brightness sample A(X)). A 、Y A ) and the maximum brightness sample B(X B 、Y B )) is derived from the linear relationship between the brightness value and the chromaticity value. Here, X A 、Y A is the x-coordinate (i.e., luminance value) and y-coordinate (i.e., chrominance value) value of sample point A, and X B 、Y B are the x-coordinate and y-coordinate values ​​of sample point B. The linear model parameters α and β are obtained according to the following equations.

[0250]

[0251] β=y A -αx A (26)

[0252] Such a method is also known as the min-max method. The division in the above equation can be avoided and replaced by multiplication and shift.

[0253] For coding blocks with a square shape, the above two equations apply directly.For non-square coding blocks, the neighboring samples of the longer boundary are first subsampled to have the same number of samples as for the shorter boundary.

[0254] In addition to the scenario where the upper template and the left template are used together to calculate the linear model coefficients, the two templates may alternatively be used in the other two MMLM modes (referred to as MMLM_A and MMLM_L modes).

[0255] In MMLM_A mode, only pixel samples in the upper template are used to calculate the linear model coefficients. To obtain more samples, the upper template is expanded to a size of (W+W). In MMLM_L mode, only pixel samples in the left template are used to calculate the linear model coefficients. To obtain more samples, the left template is expanded to a size of (H+H).

[0256] Note that when the above reference line is at a CTU boundary, only one luma line (which is stored in the line buffer for intra prediction) is used to get the downsampled luma samples.

[0257] For chroma intra mode coding, a total of 11 intra modes are allowed for chroma intra mode coding. These modes include five traditional intra modes and six cross-component linear model modes (CCLM, LM_A, LM_L, MMLM, MMLM_A and MMLM_L). The chroma mode signaling and derivation process are shown in Table 6. Chroma mode coding directly depends on the intra prediction mode of the corresponding luminance block. Since the separate block partitioning structures for luminance components and chrominance components are enabled in I slices, one chroma block can correspond to multiple luminance blocks. Therefore, for the chroma DM mode, the intra prediction mode of the corresponding luminance block covering the center position of the current chroma block is directly inherited.

[0258] Table 6 - Chroma prediction modes derived from luma mode when MMLM_ is enabled

[0259]

[0260] Adaptive Enabling of LM and MMLM for Forecasting

[0261] MMLM and LM models can also be used together in an adaptive manner. For MMLM, the two linear models are as follows:

[0262]

[0263] where pred C (i, j) represents the predicted chroma sample in the CU, and rec L ′(i, j) represents the downsampled reconstructed luma sample of the same CU. Threshold can be simply determined based on the luma and chroma average values ​​and their minimum and maximum values. Figure 11 An example of classifying adjacent samples into two groups based on the inflection point T indicated by the arrow is shown. The linear model parameters α1 and β1 are obtained from two samples (which are the minimum brightness sample A(X A 、Y A )) between the brightness and chromaticity values ​​and the Threshold (X T 、Y T ) is derived. The linear model parameters α2 and β2 are derived from the two samples (which are the maximum brightness samples B(X B 、Y B )) between the brightness and chromaticity values ​​and the Threshold (X T 、Y T ) is derived. Here, X A 、Y A is the x-coordinate (i.e., luminance value) and y-coordinate (i.e., chrominance value) value of sample point A, and X B 、Y B is the x-coordinate and y-coordinate value of sample point B. The linear model parameter α of each group is obtained according to the following equation i and β i , where i is equal to 1 and 2 respectively.

[0264]

[0265] β2=Y T -α2X T (28)

[0266] For coding blocks with a square shape, the above equations apply directly. For non-square coding blocks, the neighboring samples of the longer boundary are first subsampled to have the same number of samples as for the shorter boundary.

[0267] In addition to the scenario where the upper template and the left template are used together to determine the linear model coefficients, the two templates may alternatively be used in the other two MMLM modes (referred to as MMLM_A and MMLM_L modes, respectively).

[0268] In MMLM_A mode, only pixel samples in the upper template are used to calculate the linear model coefficients. To obtain more samples, the upper template is expanded to a size of (W+W). In MMLM_L mode, only pixel samples in the left template are used to calculate the linear model coefficients. To obtain more samples, the left template is expanded to a size of (H+H).

[0269] Note that when the above reference line is at a CTU boundary, only one luma line (which is stored in the line buffer for intra prediction) is used to get the downsampled luma samples.

[0270] For chroma intra mode coding, there is a condition check that is used to select LM mode (CCLM, LM_A and LM_L) or multi-model LM mode (MMLM, MMLM_A and MMLM_L). The condition check is as follows:

[0271]

[0272] Where BlkSizeThres LM Indicates the minimum block size in LM mode, and BlkSizeThres MM Indicates the minimum block size of the MMLM mode. The symbol d represents a predetermined threshold value. In one example, d can be set to 0. In another example, d can be set to 8.

[0273] For chroma intra mode coding, a total of 8 intra modes are allowed for chroma intra mode coding. These modes include five traditional intra modes and three cross-component linear model modes. The chroma mode signaling and derivation process are shown in Table 1. It is worth noting that for a given CU, if it is encoded in linear model mode, whether it is a conventional single model LM mode or an MMLM mode is determined based on the above conditional check. Unlike the case shown in Table 6, there is no separate MMLM mode to be signaled. Chroma mode coding depends directly on the intra prediction mode of the corresponding luminance block. Since separate block partitioning structures for luminance components and chrominance components are enabled in I slices, one chroma block can correspond to multiple luminance blocks. Therefore, for the chroma DM mode, the intra prediction mode of the corresponding luminance block covering the center position of the current chroma block is directly inherited.

[0274] Slope adjustment for CCLM

[0275] During the development of the ECM, scale (slope) adjustment for the CCLM was proposed as a further improvement, for example, as described in JVET-Y0055 / Z0049.

[0276] As discussed above, CCLM uses a 2-parameter model to map luma values ​​to chroma values. The scale parameter "a" and the bias parameter "b" define the following mapping:

[0277] chromaVal=a*lumaVal+b (30)

[0278] It is proposed to signal the adjustment of the scale parameter "u" to update the model to the following form:

[0279] chromaVal=a'*lumaVal+b' (31)

[0280] Where a' = a + u, and b' = bu * y r .

[0281] With this choice, the mapping function is centered around the value y with the brightness r It is recommended to use the average value of the reference brightness samples used in model creation as the y r , in order to provide meaningful modifications to the model. Figures 12A to 12B shows the effect of the rescaling parameter "u", where Figure 12A shows the model created without the scaling parameter "u", and Figure 12B The model created with the scaling parameter "u" is shown.

[0282] In one example, the rescaling parameter is provided as an integer between -4 and 4 (inclusive) and signaled in the bitstream. The unit of the rescaling parameter is 1 / 8 of the chroma sample value per luma sample value (for 10-bit content).

[0283] In one example, the CCLM model that is used for both the reference samples above and to the left of the block ("LM_CHROMA_IDX" and "MMLM_CHROMA_IDX") but not for "one-sided" mode is adjusted. This choice is based on the codec efficiency versus complexity tradeoff.

[0284] When applying rescaling for a multi-mode CCLM model, both models may be rescaled and thus up to two scale updates may be signaled for a single chroma block.

[0285] To implement rescaling at the encoder, the encoder may perform a SATD-based search for the optimal value of the rescaling update for Cr and a similar SATD-based search for Cb. If either result is a non-zero rescaling parameter, the combined rescaling pair (SATD-based update for Cr, SATD-based update for Cb) is included in the RD check list for the TU.

[0286] Fusion of chroma intra prediction modes

[0287] During the development of ECM, JVET-Y0092 / Z0051 proposed the fusion of chroma intra mode.

[0288] The intra prediction modes enabled for chroma components in ECM-4.0 are six cross-component linear model (LM) modes (including CCLM_LT, CCLM_L, CCLM_T, MMLM_LT, MMLM_L, and MMLM_T modes), direct mode (DM), and four default chroma intra prediction modes. The four default modes are given by the list {0, 50, 18, 1}, and if the DM mode already belongs to the list, the mode in the list will be replaced with mode 66.

[0289] The decoder-side intra mode derivation (DIMD) method for luma intra prediction is included in ECM-4.0. First, the horizontal and vertical gradients are calculated for each reconstructed luma sample of the L-shaped template of the second adjacent row and column of the current block to construct a histogram of gradients (HoG). Then, the two intra prediction modes with the largest and second largest histogram magnitude values ​​are blended with the planar mode to generate the final prediction value for the current luma block.

[0290] In order to improve the coding efficiency of chroma intra prediction, two methods are proposed, including the decoder-side derived chroma intra prediction mode (DIMD chroma) and the fusion of non-LM mode and MMLM_LT mode.

[0291] DIMD color mode

[0292] In the first embodiment, a DIMD chroma mode is proposed. The proposed DIMD chroma mode uses the DIMD derivation method to derive the chroma intra prediction mode of the current block based on the co-located reconstructed luma samples. Specifically, Figure 13 As shown in , the horizontal gradient and vertical gradient are calculated for each co-located reconstructed luma sample of the current chroma block to construct the HoG. Then, the intra prediction mode with the largest histogram magnitude value is used to perform chroma intra prediction of the current chroma block.

[0293] When the intra prediction mode derived from the DIMD chroma mode is the same as the intra prediction mode derived from the DM mode, the intra prediction mode having the second largest histogram magnitude value is used as the DIMD chroma mode.

[0294] A CU level flag is signaled to indicate whether the proposed DIMD chroma mode is applied, as shown in Table 7.

[0295] Table 7. Binarization process of intra_chroma_pred_mode used in the proposed method

[0296] intra_chroma_pred_mode binary bit string Chroma Intra mode 0 1100 List[0] 1 1101 List[1] 2 1110 List[2] 3 1111 List[3] 4 10 DIMD chromaticity 5 0 DM

[0297] Fusion of chroma intra prediction modes

[0298] In a second embodiment, a fusion of chroma intra prediction modes is proposed, wherein the DM mode and the four default modes can be fused with the MMLM_LT mode as follows:

[0299] pred=(w0*pred0+w1*pred1+(1<<(shift-1)))>>shift

[0300] Where pred0 is the prediction value obtained by applying the non-LM mode, pred1 is the prediction value obtained by applying the MMLM_LT mode, and pred is the final prediction value of the current chroma block. The two weights w0 and w1 are determined by the intra prediction mode of the adjacent chroma blocks, and shift is set to be equal to 2. Specifically, when the upper adjacent block and the left adjacent block are both encoded in the LM mode, {w0, w1} = {1, 3}; when the upper adjacent block and the left adjacent block are both encoded in the non-LM mode, {w0, w1} = {3, 1}; otherwise, {w0, w1} = {2, 2}.

[0301] For syntax design, if non-LM mode is selected, a flag is signaled to indicate whether fusion is applied, and the proposed fusion is only applied to I slices.

[0302] Combination of DIMD chroma mode and chroma intra prediction mode fusion

[0303] In the third embodiment, the DIMD chroma mode is combined with the fusion of the chroma intra prediction mode. Specifically, the DIMD chroma mode described in the first embodiment is applied, and for I slices, the DM mode, the four default modes, and the DIMD chroma mode can be fused with the MMLM_LT mode using the weights described in the second embodiment, while for non-I slices, only the DIMD chroma mode can be fused with the MMLM_LT mode using equal weights.

[0304] Combination of DIMD chroma mode and chroma intra prediction mode fusion with reduced processing

[0305] In a fourth embodiment, the DIMD chroma mode with reduced processing is combined with a fusion of chroma intra prediction modes. Specifically, the DIMD chroma mode with reduced processing derives an intra mode based on adjacent reconstructed Y, Cb, and Cr samples in the second adjacent row and column, as shown in FIG. Figure 14 The other parts are the same as those of the third embodiment.

[0306] Decoder-side intra mode derivation (DIMD)

[0307] In one embodiment, when DIMD is applied, two intra modes are derived from reconstructed neighboring samples and these two predictions are combined with the planar mode prediction using weights derived from gradients as described in JVET-00449, as 15A to 15D The division operation in the weight derivation is performed using the same LUT-based integration scheme used by CCLM. For example, the division operation in the orientation calculation Orient=G y / G x It is calculated using the following LUT-based scheme:

[0308] x=Floor(Log2(Gx))

[0309] normDiff=((Gx<<4)>>x)&15

[0310] x+=(3+(normDiff!=0)?1:0)

[0311] Orient=(Gy*(DivSigTable[normDiff]|8)+(1<<(x-1)))>>x

[0312] in

[0313] DivSigTable

[16] ={0, 7, 6, 5, 5, 4, 4, 3, 3, 2, 2, 1, 1, 1, 1, 0}.

[0314] The derived intra modes are included in the main list of intra most probable modes (MPMs), so the DIMD process is performed before building the MPM list. The main derived intra modes of a DIMD block are stored with the block and are used for MPM list construction of neighboring blocks.

[0315] Figures 15A to 15D The steps of intra mode derivation at the decoder side are shown, where the intra prediction direction is estimated without intra mode signaling. Figure 15A The first step shown in consists of estimating for each sample point (for Figure 15A The gradient of the light gray sample points shown in Figure 15B The second step shown in consists of mapping the gradient value to the closest prediction direction within [2, 66]. Figure 15C The third step shown in comprises selecting two prediction directions, wherein for each prediction direction, all absolute gradients Gx and Gy of the neighboring pixels having that direction are summed, and the first two directions are selected. Figure 15D The fourth step shown in comprises implementing weighted intra prediction using the selected direction.

[0316] Multiple Reference Line (MRL) Intra Prediction

[0317] Multiple reference line (MRL) intra prediction uses more reference lines for intra prediction. Figure 16 In

[15] , an example of 4 reference lines is depicted, where the samples of segments A and F are not extracted from the reconstructed neighboring samples, but are filled with the closest samples from segments B and E, respectively. HEVC intra-picture prediction uses the nearest reference line (i.e., reference line 0). In MRL, 2 additional lines (reference line 1 and reference line 3) are used.

[0318] The index of the selected reference line (mrl_idx) is signaled and used to generate intra prediction values. For reference line indices greater than 0, only additional reference line modes in the MPM list are included, and only the mpm index is signaled if there are no remaining modes. The reference line index is signaled before the intra prediction mode, and if a non-zero reference line index is signaled, planar mode is excluded from the intra prediction mode.

[0319] MRL is disabled for the first line of blocks inside a CTU to prevent the use of extended reference samples outside the current CTU line. In addition, PDPC is disabled when additional lines are used. For MRL mode, the derivation of the DC value in DC intra prediction mode for non-zero reference line index is consistent with the derivation of the DC value for reference line index 0. MRL requires the storage of 3 adjacent luma reference lines for a CTU to generate the prediction. The Cross Component Linear Model (CCLM) tool also requires 3 adjacent luma reference lines for its downsampling filter. The definition of MRL using the same 3 lines is aligned with CCLM to reduce the storage requirements for the decoder.

[0320] Convolutional Cross-Component Model (CCCM) for Intra Prediction

[0321] During the development of ECM, the Convolutional Cross-Component Model (CCCM) for Chroma Intra Mode was proposed.

[0322] It is proposed to apply a convolutional cross-component model (CCCM) to predict chroma samples from reconstructed luma samples in a similar spirit to that done by the current CCLM mode. As with CCLM, when chroma subsampling is used, the reconstructed luma samples are downsampled to match the lower resolution chroma grid.

[0323] In addition, similar to CCLM, there is an option to use a single model or a multi-model variant of CCCM. The multi-model variant uses two models, one model is derived for samples above the average luminance reference value, and the other model is for the remaining samples (following the spirit of CCLM design). The multi-model CCCM mode can be selected for PUs with at least 128 available reference samples.

[0324] Convolution filter

[0325] The proposed convolutional 7-tap filter consists of a 5-tap plus-shape spatial component, a nonlinear term, and a bias term. The input to the spatial 5-tap component of the filter consists of the following terms: the center (C) luma sample (which is co-located with the chroma sample to be predicted) and its above / north (N), below / south (S), left / west (W), and right / east (E) neighbors, as Figure 17 As shown in .

[0326] The nonlinear term P is expressed as a power of 2 of the center luma sample C and scaled to the content's sample value range:

[0327] P=(C*C+midVal)>>bitDepth

[0328] That is, for 10-bit content, it is calculated as:

[0329] P=(C*C+512)>>10

[0330] The bias term B represents a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the middle chroma value (512 for 10-bit content).

[0331] The output of the filter is calculated as the filter coefficient c i Convolution with the input value and clipped to the range of valid chroma samples:

[0332] predChromaVal=c0C+c1N+c2S+c3E+c4W+c5P+c6B

[0333] Calculation of filter coefficients

[0334] The filter coefficients c are calculated by minimizing the MSE between the predicted chroma samples and the reconstructed chroma samples in the reference region. i . Figure 18A reference region consisting of six rows of chroma samples above and to the left of the PU is shown. The reference region extends one PU width to the right below the PU boundary and one PU height. The region is adjusted to include only available samples. The extension of the region shown in blue is required to support the "side samples" of the plus-shaped spatial filter and is padded when in unavailable areas.

[0335] MSE minimization is performed by computing the autocorrelation matrix of the luma input and the cross-correlation vector between the luma input and the chroma output. The autocorrelation matrix is ​​LDL-decomposed, and back substitution is used to calculate the final filter coefficients. This process roughly follows the calculation of the ALF filter coefficients in ECM, however, LDL decomposition is chosen instead of Cholesky decomposition to avoid the use of square root operations. The proposed method uses only integer arithmetic.

[0336] Bitstream signaling

[0337] The use of the mode is signaled using a CABAC coded PU-level flag. A new CABAC context is included to support this. When it comes to signaling, CCCM is considered a submode of CCLM. That is, the CCCM flag is only signaled when the intra prediction mode is LM_CHROMA_IDX (to enable single-mode CCCM) or MMLM_CHROMA_IDX (to enable multi-mode CCCM).

[0338] Encoder Operation

[0339] The encoder performs two new RD checks in the chroma prediction mode loop, one for checking single-model CCCM mode and one for checking multi-model CCCM mode.

[0340] In existing CCLM or MMLM designs, adjacent reconstructed luma-chroma samples are classified into one or more sample groups based on a value called Threshlod that considers only the luma DC value. That is, luma-chroma sample pairs are classified by considering only the intensity of the luma samples. However, the luma component typically retains a large amount of texture, and the current luma sample can be highly correlated with neighboring luma samples. Such inter-sample correlation (AC correlation) can benefit the classification of luma-chroma sample pairs and bring additional codec efficiency.

[0341] like Figure 19A As shown in , CCLM assumes that a given chroma sample is only related to the corresponding luma sample (L0.5, which can be regarded as a fractional luma sample position) and uses simple linear regression (SLR) with ordinary least squares (OLS) estimation to predict the given chroma sample. Figure 19BAs shown in , in some video contents, one chroma sample may be correlated with multiple luma samples simultaneously (AC or DC correlation), so the multiple linear regression (MLR) model can further improve the prediction accuracy.

[0342] Although CCCM can enhance intra-frame prediction efficiency, there is room for further improvement. At the same time, some parts of the existing CCCM mode need to be simplified for efficient codec hardware implementation or improved for better codec efficiency. Furthermore, the trade-off between implementation complexity and codec efficiency benefits needs to be further improved.

[0343] Marginal Linear Model (ELM)

[0344] To improve the encoding and decoding efficiency of luma and chroma components, a classifier that considers luma edge or AC information is introduced, in contrast to the above-mentioned implementation method that only considers luma DC values. In addition to existing classified MMLMs, the present disclosure provides exemplary classifiers. The process of generating linear prediction models for different groups of samples can be similar to CCLM or MMLM (e.g., via least squares or a simplified min-max method), but with different classification metrics. Different classifiers can be used to classify adjacent luma samples (e.g., of adjacent luma-chroma sample pairs) and / or luma samples corresponding to the chroma sample to be predicted. Luma samples corresponding to chroma samples can be obtained through a downsampling operation to match the positions of corresponding chroma samples in a 4:2:0 video sequence. For example, luma samples corresponding to chroma samples can be obtained by performing a downsampling operation on more than one (e.g., four) reconstructed luma samples corresponding to (e.g., located around) the chroma sample. Alternatively, for example, in the case of a 4:4:4 video sequence, the luma samples may be obtained directly from the reconstructed luma samples. Alternatively, the luma samples may be obtained from corresponding ones of the reconstructed luma samples at corresponding co-located positions of the corresponding chroma samples. For example, the luma sample to be classified may be obtained from one of the four reconstructed luma samples corresponding to the chroma sample at the upper left position of the four reconstructed luma samples, which may be considered as the co-located position for the chroma samples.

[0345] The first classifier can classify luma samples based on their edge strength. For example, a direction (e.g., 0 degrees, 45 degrees, or 90 degrees, etc.) can be selected to calculate the edge strength. The direction can be formed by the current sample and the adjacent samples along the direction (e.g., the adjacent sample located to the upper right of the current sample at 45 degrees). The edge strength can be calculated by subtracting the adjacent samples from the current sample. The edge strength can be quantized into one of M segments using M-1 thresholds, and the first classifier can use M categories to classify the current sample. Alternatively or additionally, N directions can be formed by the current sample and N adjacent samples along N directions. N edge strengths can be calculated by subtracting the N adjacent samples from the current sample, respectively. Similarly, if each of the N edge strengths can be quantized into one of M segments using M-1 thresholds, the first classifier can use MN categories to classify the current sample.

[0346] The second classifier can be used to classify based on local patterns. For example, the current luma sample Y0 can be compared with its N adjacent luma samples Yi. If the value of Y0 is greater than the value of Yi, the score can be increased by 1, otherwise, the score can be decreased by 1. The score can be quantized to form K categories. The second classifier can classify the current sample into one of the K categories. For example, the adjacent luma samples can be obtained from the four neighbors located above, to the left, to the right, and below (i.e., without diagonal neighbors) the current luma sample.

[0347] It is contemplated that multiple first classifiers, second classifiers, or different instances of the first classifier or the second classifier or other classifiers described herein may be combined. For example, the first classifier may be combined with an existing MMLM threshold-based classifier. For another example, an instance A of the first classifier may be combined with another instance B of the first classifier, where instances A and B employ different orientations (e.g., vertical and horizontal, respectively).

[0348] It will be appreciated by those skilled in the art that, although the existing CCLM design in the VVC standard is used as the basic CCLM method in the description, the proposed cross-component method described in this disclosure can also be applied to other predictive codecs with similar design spirit. For example, for Chroma from Luma (CfL) in the AV1 standard, the proposed method can also be applied by dividing the luma-chroma sample pairs into multiple sample groups.

[0349] It will be understood by those skilled in the art that Y / Cb / Cr can also be represented as Y / U / V in the field of video coding. If the video data is in RGB format, the proposed method can also be applied by, for example, simply mapping the YUV representation to GBR.

[0350] Filter-based linear model (FLM)

[0351] A filter-based linear model (FLM) utilizing the MLR model is introduced below to account for the possibility that one chrominance sample may be simultaneously correlated with multiple luma samples.

[0352] For chroma samples to be predicted, the reconstructed co-located and adjacent luma samples can be used to predict the chroma samples to capture the inter-sample correlation between the co-located luma samples, adjacent luma samples, and chroma samples. The reconstructed luma samples are linearly weighted and combined with an "offset" to generate predicted chroma samples (C: predicted chroma samples, L i : the i-th reconstructed co-located or adjacent brightness sample, α i : filter coefficient, β: offset, N: filter tap), as shown in the following equation (32-1). Note that the linear weighting plus the offset value directly forms the predicted chroma samples (which can be low-pass or high-pass adaptively according to the video content), and then it is added to the residual to form the reconstructed chroma samples.

[0353]

[0354] In some implementations like CCCM, the offset term may also be implemented as the intermediate chroma value B (512 for 10-bit content) multiplied by another coefficient, as shown in the following equation (32-2).

[0355]

[0356] For a given CU, the top and left reconstructed luma and chroma samples can be used to derive or train FLM parameters (α i ,,β). As with CCLM, α i and β can be derived via OLS. The top and left training samples are collected and a pseudo-inverse matrix is ​​computed at both the encoder and decoder sides to derive the parameters, which are then used to predict the chroma samples in a given CU. Let N denote the number of filter taps applied to the luma samples, M denote the total top and left reconstructed luma-chroma sample pairs used for training the parameters, represents the luma sample with the i-th sample pair and the j-th filter tap, C i Denotes the chrominance sample with the i-th sample pair, the following equation shows the pseudo-inverse matrix A + The derivation and parameters. Figure 20 An example is shown where N is 6 (6-tap), M is 8, and the top 2 rows and left 3 columns of luma samples and the top 1 row and left 1 column of chroma samples are used to derive or train parameters.

[0357]

[0358]

[0359] b=Ax

[0360] x=(A T A) -1 A T b=A + b (33)

[0361] Please note that it is possible to i Instead, there is no offset β to predict the chroma samples, which can be a subset of the proposed method.

[0362] It should be noted that although the existing CCLM design in the VVC standard is used as the basic CCLM method in the following description, it is clear to those skilled in the art of video coding that the proposed cross-component method described in this disclosure can also be applied to other predictive codecs with similar design spirit. For example, for Chroma from Luma (CfL) in the AV1 standard, the proposed FLM can also be applied by including multiple luma samples into the MLR model.

[0363] The proposed ELM / FLM / GLM (as discussed below) can be directly extended to the CfL design in the AV1 standard, which explicitly sends the model parameters (α, β). For example, (1-tap case) α and / or β are derived at the encoder at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level and signaled to the decoder for CfL mode.

[0364] Filter shape

[0365] In order to further improve the encoding and decoding performance, additional designs can be used in FLM prediction. Figure 20 As shown in and discussed above, a 6-tap luma filter is used for FLM prediction. However, although a multi-tap filter can fit the training data well (e.g., the top and left adjacent reconstructed luma and chroma samples), in some cases, the training data does not capture all the characteristics of the test data, which may lead to overfitting and may not predict the test data well (i.e., the chroma block samples to be predicted). Moreover, different filter shapes can adapt well to different video block contents, resulting in more accurate predictions.

[0366] To address this issue, the filter shape and the number of filter taps can be predefined, or signaled or switched in the sequence parameter set (SPS), adaptive parameter set (APS), picture parameter set (PPS), picture header (PH), slice header (SH), region, CTU, CU, sub-block or sample level. The filter shape candidate set can be predefined, and the selection of the filter shape candidate set can be signaled or switched in the SPS, APS, PPS, PH, SH, region, CTU, CU, sub-block or sample level. Different components (e.g., U and V) can have different filter switching controls. For example, the filter shape candidate set (e.g., indicated by indices 0 to 5) can be predefined, and the filter shape (1, 2) can represent a 2-tap luminance filter, the filter shape (1, 2, 4) can represent a 3-tap luminance filter, and so on, as in Figure 20 As shown in . The filter shape selection for U and V components can be switched in PH or in CU or CTU level. Note that N taps can represent N taps with or without offset β as described herein. An example is given in Table 8 below.

[0367] Table 8 - Example signaling and switching for different filter shapes

[0368]

[0369]

[0370] The FLM or CCCM filter shape can include nonlinear terms. For example, for a CCCM filter, the following equations can be used to predict the chroma sample values:

[0371] predChromaVal=c0C+c1N+c2S+c3E+c4W+c5P+c6B,P=(C*C+midVal)>>bitDepth

[0372] The filter corresponds to the weighting coefficients c0, c1, ..., c6 of the center (C) luma sample value (which is co-located with the chroma sample to be predicted), its upper / north (N), lower / south (S), right / east (E), and left / west (W) neighbors, the nonlinear term P, and the bias term B. The value used to derive the nonlinear term P can be a combination of the current and neighboring luma samples, but is not limited to C*C. For example, P can be derived as follows:

[0373] P=(Q*R+midVal)>>bitDepth

[0374] Here, Q and R represent the values ​​used to derive the nonlinear term P.

[0375] Q and R can be linear combinations of the current and adjacent luma samples in the downsampled domain (eg, Q and R are pre-operation luma samples obtained by a weighted average operation) or without any downsampling process.

[0376] For example, each of Q and R can be selected from one of the luma sample values ​​of N, S, E, W, and C, for example, Q*R=C*N, C*S, C*E, C*W, S*N, or N*N, etc.; or both Q and R can be equal to the average of the luma sample values ​​of N, S, E, and W, that is, Q=R=(N+S+E+W) / 4; or Q is equal to the luma sample value of C, and R is equal to the average of the luma sample values ​​of N, S, E, and W, that is, Q=C, and R=(N+S+E+W) / 4.

[0377] Different values ​​(Q and R) used to derive the nonlinear term are considered different filter shapes and can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. The filter shape candidate set can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0378] Different chroma types and / or color formats may have different predefined filter shapes and / or taps. For example, a predefined filter shape (1, 2, 4, 5) may be used for 4:2:0 type 0, a predefined filter shape (0, 1, 2, 4, 7) may be used for 4:2:0 type 2, and a predefined filter shape (1, 4) may be used for 4:2:2, and a predefined filter shape (0, 1, 2, 3, 4, 5) may be used for 4:4:4, as shown in FIG. Figure 21 As shown in .

[0379] In another aspect of the present disclosure, unavailable luma and chroma samples used to derive the MLR model can be filled in from available reconstruction samples. Figure 21 In the 6-tap (0, 1, 2, 3, 4, 5) filter, for the CU located at the left picture boundary, the left column including the sample (0, 3) is unavailable (outside the picture boundary), so the sample (0, 3) is repeated from the sample (1, 4) to apply the 6-tap filter. Note that the padding process can be applied to both the training data (top and left adjacent reconstructed luma and chroma samples) and the test data (luma and chroma samples in the CU).

[0380] One or more shapes / number of filter taps can be used for FLM prediction, for example in Figure 25 、 Figure 26 and FIG. 27A to FIG. 27B One or more sets of filter taps can be used for FLM prediction, for example in Figures 28A to 28G Shown in.

[0381] Implicit filter shape derivation

[0382] The filter shape candidate may be derived implicitly without explicit signaling bits. For example, the filter shape candidate may be a filter shape candidate for FLM or GLM (as discussed below). In another example, the filter shape candidate may be a cross-shaped filter for CCCM, Figure 25 、 Figure 26 、 Figure 27A 、 Figure 27B as well as Figures 28A to 28G Any of the filters shown in or other filters mentioned in this disclosure. Since longer filter taps theoretically always fit better in the training data (template area), but may overfit, the "N-fold cross validation" technique well known in the field of machine learning can be used to train the filter coefficients. This technique divides the available training data into N sets and uses some of the sets for training and the other sets for validation.

[0383] The following example involves implicit filter shape derivation for FLM prediction:

[0384] Step 1: Determine M filter shape candidates for predicting the chroma sample values ​​of the current CU;

[0385] Step 2: Divide the available L-shaped template area outside the CU into N regions, denoted as R0, R1, ... R N-1 , that is, the training data is divided into N sets for N-fold training, where the luminance sample values ​​and chrominance sample values ​​of the available template area are known values;

[0386] Step 3: Apply each of the M filter shape candidates to a portion of the available template region, i.e., all N regions R0, R1, ... R N-1 One or more areas in

[0387] Step 4: Derive M sets of filter coefficients corresponding to the M filter shape candidates, denoted as F0, F1, ... F M- 1;

[0388] Step 5: The derived F0, F1, ... F M-1 Applying the set of filter coefficients to another portion of the available template area to predict chrominance sample values ​​based on corresponding luma sample values, wherein the another portion of the available template area is different from the portion of the available template area mentioned in step 3;

[0389] Step 6: Accumulate the errors (denoted as E0, E1, ..., E1) between the predicted chroma sample values ​​and the known chroma sample values ​​in another part of the available template area by sum of absolute differences (SAD), sum of squared differences (SSD), or sum of absolute transformed differences (SATD) for each of the M filter shapes, respectively. M-1 );

[0390] Step 7: Sort and select the K smallest errors (denoted as E'0, E'1, ... E' K-1 ), which corresponds to K filter shapes and K sets of filter coefficients; and

[0391] Step 8: Select a filter shape candidate from the K filter shape candidates to apply to the current CU for chroma prediction. If K is greater than 1, the decoder can still receive a signal from the encoder indicating the filter to be applied. However, if K is 1, the signaling can be omitted and the filter with the smallest accumulated error is determined as the applied filter.

[0392] Figure 29A and Figure 29B An example of 2-fold training for implicit filter shape derivation is shown. For the current chroma CU prediction (blue area), Figure 29A It is shown that in the template area, the even-numbered row area R0 (yellow) is used for training / deriving 4 sets of filter coefficients, and the odd-numbered row area R1 (red) is used for verification / comparison of the 4 sets of filter coefficients and sorting the costs; Figure 29B Shown in the template area, R0 (yellow) and R1 (red) are interleaved. It should be understood that R0 and R1 can be swapped in these examples.

[0393] In one example, to select one of four filter shape candidates as the filter to be applied, the L-shaped template area is divided into even-numbered and odd-numbered rows or columns. The steps include:

[0394] Four filter shape candidates are predefined for the current CU (e.g., from Figure 25 4 filter shape candidates of );

[0395] The available L-shaped template area (e.g., 6 chroma rows and columns for CCCM, noting that in CCCM design, each chroma sample refers to 6 luma samples for downsampling) is divided into 2 regions denoted as R0, R1, where, for example, R0 consists of even rows or columns, and R1 consists of odd rows or columns ( Figure 29AThe following example is shown: in the template area, the even-numbered row area R0 is used to train / derive 4 sets of filter coefficients, and the odd-numbered row area R1 is used to verify / compare and sort the costs of the 4 sets of filter coefficients);

[0396] Apply the four filter shape candidates independently to a portion of the available template region (e.g., region R0);

[0397] Four sets of filter coefficients are derived for each of the four filter shapes, denoted as F0, F1, ..., F3;

[0398] Applying the derived F0, F1, ... F3 filter coefficient sets to another portion of the available template area (e.g., area R1) to predict corresponding chrominance sample values;

[0399] Accumulating the errors (denoted as E0, E1, ... E3) between the predicted chroma sample values ​​and the known chroma sample values ​​in another part of the available template area (e.g., area R1) by SAD, SSD, or SATD, respectively, for each of the four filters; and

[0400] A filter shape candidate (denoted as E'0) with the minimum accumulated error among E0, E1, ... E3 among the four filter shape candidates is selected, which corresponds to one filter shape and one filter coefficient set. In this example, only the filter shape candidate with the minimum accumulated error will be determined as the applied filter, and then the decoder does not need to receive a signal indicating the applied filter.

[0401] In one example, the L-shaped template region can be divided into staggered parts when K = 2. These steps include:

[0402] Determine 4 filter shape candidates for the current CU;

[0403] The available L-shaped template area (e.g., 6 chroma rows or columns for CCCM) is divided into 2 regions denoted as R0, R1, where the luma samples in R0 and R1 are interleaved, for example, as shown in the following table:

[0404]

[0405] or

[0406] <![CDATA[R1]]> <![CDATA[R0]]> <![CDATA[R0]]> <![CDATA[R1]]>

[0407] as well as Figure 29B The following example is shown: in the template area, R0 and R1 are interleaved;

[0408] Apply the four filter shape candidates independently to a portion of the available template region (e.g., region R0);

[0409] derive four sets of filter coefficients (denoted as F0, F1, ..., F3) for each of the four filter shapes;

[0410] Applying the derived F0, F1, ... F3 filter coefficient sets to another portion of the available template area (e.g., area R1) to predict corresponding chrominance sample values;

[0411] Accumulate the errors (denoted as E0, E1, ..., E3) between the predicted chroma sample values ​​and the known chroma sample values ​​in another part of the available template area (e.g., area R1) by SAD, SSD, or SATD, respectively, for each of the four filter shapes;

[0412] sorting and selecting the two smallest accumulated errors (denoted as E'0, E'1) among E0, E1, ... E3, which correspond to the two filter shapes and the two sets of filter coefficients; and

[0413] Based on the signal received from the encoder, one of the two filter shape candidates is selected to be applied to the current CU for chroma prediction.

[0414] Note that the implicit filter shape derivation method can also be used to determine whether to introduce nonlinear terms in the CCCM filter coefficients (using / not using nonlinear terms is treated as different filter shapes).

[0415] Although the examples above are shown for CCCM filters, it should be understood that the nonlinear term P can also be included in FLM filters (e.g., as Figure 20 ) and is derived in a similar manner as discussed above.

[0416] In one example, the step of dividing the available L-shaped template area can be omitted. In this example, M sets of filter coefficients can be derived based on the sample values ​​from the available template area and then applied back to the available template area to predict the corresponding chrominance sample values ​​for accumulating the error.

[0417] Matrix derivation

[0418] As mentioned above, the MLR model (linear equation) must be derived at both the encoder and the decoder. According to one or more aspects of the present disclosure, several methods are proposed to derive the pseudo-inverse matrix A. + , or solve the linear equation directly. Other known methods include Newton's method, Cayley-Hamilton method and eigendecomposition (as in https: / / en.wikipedia.org / wiki / Invertible_matrix ) can also be applied.

[0419] In this disclosure, for simplicity, A + It can be represented as A -1 . Linear equations can be solved as follows:

[0420] 1. Solve A using the adjoint matrix (adjA), closed form, and analytical solution -1 :

[0421] The following shows an nxn general form, a 2x2 and a 3x3 case. If FLM uses 3x3, then 2 scalers plus an offset need to be solved.

[0422] b=Ax,x=(A T A) -1 A T b=A + b, is represented by A -1 b

[0423]

[0424] By removing the (n-1)x(n-1) submatrix with the jth row and ith column

[0425]

[0426]

[0427] 2. Gauss-Jordan elimination method

[0428] We can use Gauss-Jordan elimination, by augmenting the matrix [AI n ] and a series of basic row operations to solve the linear equation to obtain the reduced row echelon form matrix [I|X]. 2x2 and 3x3 examples are shown below.

[0429]

[0430] 3. Cholesky decomposition

[0431] To solve Ax=b, A can first be decomposed by the Cholesky-Crout algorithm, resulting in an upper triangular matrix and a lower triangular matrix, and a forward substitution followed by a backward substitution can be applied serially to obtain the solution. A 3x3 example is shown below.

[0432]

[0433]

[0434] In addition to the above examples, some conditions require special handling. For example, if some conditions result in the inability to solve a linear equation, default values can be used to fill the chrominance prediction values. For example, when 1<<(bitDepth-1), meanC, meanL, or mean C-meanL (average current chrominance or other chrominance, from available luminance values, or a subset of the FLM reconstructed adjacent regions) is predefined, the default value can be predefined, signaled, or switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0435] The following example shows the case when matrix A cannot be solved, where the default prediction value can be assigned to the entire current block:

[0436] 1. Solved by closed form (analytical, adjoint matrix), but A is singular (i.e., detA = 0); <H

[0437] 2. Solved by Cholesky decomposition, but A cannot be Cholesky decomposed, g jj <REG_SQR, where REG_SQR is a small value that can be predefined, signaled, or switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0438] Applied area

[0439] Figure 20 Illustrates a typical case of deriving FLM parameters using the top 2 luminance lines and / or the left 3 luminance lines and the top 1 chrominance line and / or the left 1 chrominance line. However, as mentioned above, due to different block contents and different reconstruction qualities of adjacent samples, using different regions for parameter derivation can bring coding and decoding benefits. The following presents several ways to select the regions applied for parameter derivation:

[0440] 1. Similar to MDLM, FLM derivation can use only the top or left luminance and / or chrominance samples to derive parameters. Whether to use FLM, FLM_L, or FLM_T can be predefined, signaled, or switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. Assuming the current chrominance block has a size of W×H, then W' and H' are obtained as follows:

[0441] – When the FLM mode is applied, W’ = W, H’ = H;

[0442] – When FLM_T mode is applied, W'=W+We; where We represents the extended top luma / chroma sample;

[0443] When the FLM_L mode is applied, H'=H+He, where He represents the extended left luma / chroma samples.

[0444] The number of extended luma / chroma samples (We, He) may be predefined, or signaled or switched in SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0445] For example, (We, He) = (H, W) is predefined as a VVC CCLM, or (W, H) is predefined as an ECM CCLM. Unavailable (We, He) luminance / chrominance samples can be repeatedly filled from the nearest (horizontal, vertical) luminance / chrominance sample.

[0446] Figure 22 The specification of FLM_L and FLM_T is shown (eg, less than 4 taps). When FLM_L or FLM_T is applied, only H' or W' luma / chroma samples, respectively, are used for parameter derivation.

[0447] 2. Similar to MRL, different line indices can be predefined, or signaled or switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level to indicate the selected luma-chroma sample pair lines. This can benefit from different reconstruction qualities of different line samples.

[0448] Figure 23 It is shown that similar to MRL, FLM can use different lines for parameter derivation (e.g., lower than 4 taps). For example, FLM can use light blue / yellow luma and / or chroma samples of index 1.

[0449] 3. Expand the CCLM region and employ all top N rows and / or left M rows for parameter derivation. Figure 23 It is shown that all dark blue and light blue as well as dark yellow and light yellow areas can be used at once. Training using larger areas (data) can lead to more robust MLR models.

[0450] It should be understood that luma sample values ​​of an outer region of a video block to be decoded may be referred to as “outer luma sample values” and chroma sample values ​​of an outer region may be referred to as “outer chroma sample values” throughout the disclosure.

[0451] grammar

[0452] The corresponding syntax may be defined as follows for FLM prediction in Table 9, where FLC denotes fixed length code, TU denotes truncated unary code, EGk denotes exponential Golomb code with order k, where k may be fixed or signaled / switched in SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level, SVLC denotes signed EG0, and UVLC denotes unsigned EG0.

[0453] Table 9 - Example of FLM syntax

[0454]

[0455] Note that the binarization of each syntax element can be changed.

[0456] Gradient Linear Model (GLM)

[0457] Based on the existing linear model design, a new method for cross-component prediction is proposed to further improve the encoding and decoding accuracy and efficiency. The main aspects of the proposed method are detailed as follows.

[0458] While the FLM discussed above offers the best flexibility (leading to the best performance), if the number of filter taps increases, many unknown parameters need to be solved. When the inverse matrix is ​​larger than 3×3, closed-form derivation is not suitable (too many multipliers) and requires iterative methods like Cholesky, which burdens the decoder processing cycle. In this section, a pre-operation before applying the linear model is proposed, involving the use of sample gradients to exploit the correlation between luma AC information and chroma intensity. With the help of gradients, the number of filter taps can be effectively reduced.

[0459] Please note that the methods / examples in this section can be combined / reused from any of the designs discussed above, including but not limited to classification, filter shapes, matrix derivation (with special processing), application areas, syntax. In addition, the methods / examples listed in this section can also be applied to any of the designs discussed above to have better performance and certain complexity trade-offs.

[0460] Please note that the reference samples / training templates / reconstructed neighboring regions used in this article generally refer to the luma samples used to derive the MLR model parameters, which are then applied to the internal luma samples in a CU to predict the chroma samples in the CU.

[0461] Filter shape

[0462] According to the proposed method, instead of directly using luma sample intensity values ​​as input to the linear model, pre-operations (e.g., pre-linear weighting, sign, scaling / abs, thresholding, ReLU) can be applied to reduce the dimensionality of unknown parameters. In one example, the pre-operations can include calculating sample differences based on the luma sample values. As understood by those skilled in the art, the sample differences can be characterized as gradients, and therefore in some embodiments the new method is also referred to as a gradient linear model (GLM).

[0463] Please note that the following detailed description discusses scenarios in which the proposed pre-operation can be reused for the SLR model (also known as the 1-tap case) / combined with the SLR model, and reused for the MLR model (also known as the multi-tap case, e.g., 2-tap) / combined with the MLR model.

[0464] For example, instead of applying a 2-tap to the 2 luma samples, the 2 luma samples can be pre-operated on and then a simpler 1-tap can be applied to reduce complexity. Figures 24A to 24D Some examples for 1-tap / 2-tap (with offset) pre-operation are shown, where the 2-tap coefficients are represented as (a, b). Note that, as Figures 24A to 24D Each circle shown in represents an illustrative chroma position for the YUV 4:2:0 format. As discussed above, in the YUV 4:2:0 format, a luma sample corresponding to a chroma sample may be obtained by performing a downsampling operation on more than one (e.g., 4) reconstructed luma samples corresponding to (e.g., located around) the chroma sample. In other words, a chroma position may correspond to one or more luma samples including a co-located luma sample. Different 1-tap patterns are designed for gradient computation for different gradient directions and using different "interpolated" luma samples (weighted to different luma positions). For example, in Figure 24A 、 24C A typical filter [1, 0, -1; 1, 0, -1] is shown in FIG and 24D, which represents the following operation:

[0465]

[0466] Among them, rec L Represents the reconstructed luminance sample value and Rec L ″(i,j) represents the pre-operation brightness sample value. Also note that, as in Figure 24A 、 Figure 24C and Figure 24D The 1-tap filter shown in can be understood as an alternative to the downsampling filter with changed filter coefficients as used in CCLM (please refer to equations (6)-(7)).

[0467] The pre-operation can be based on gradient, edge direction (detection), pixel intensity, pixel change, pixel variance, Roberts / Prewitt / Compass / Sobel / Laplacian operator, high pass filter (by calculating gradient or other related operators), low pass filter (by performing weighted average operation), etc. The edge direction detector listed in the example can be extended to different edge directions. For example, 1 tap (1, -1) or 2 taps (a, b) are applied along different directions to detect different edge gradients. The filter shape / coefficient can be symmetric about the chrominance position, such as Figures 24A to 24D Example (420 Type-0 case).

[0468] Pre-operation parameters (coefficients, sign, scaling / absolute value, thresholding, ReLU) can be fixed or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. Note that in the example, if multiple coefficients apply to one sample (e.g., -1, 4), they can be combined (e.g., 3) to reduce operations.

[0469] In one example, the pre-operation may involve calculating sample differences of the luma sample values. Alternatively, the pre-operation may include performing downsampling by a weighted average operation. In some cases, the pre-operation may be applied repeatedly. For example, a template filter may be applied to the template using a low-pass smoothing FIR filter [1, 2, 1] / 4 or [1, 2, 1; 1, 2, 1] / 8 (i.e., downsampling) to remove outliers, and then a 1-tap GLM filter may be applied to calculate sample differences to derive a linear model. It is conceivable that sample differences may also be calculated and then downsampling may be implemented.

[0470] In one example, pre-operation coefficients (either ultimately applied (eg, 3) or intermediately applied (eg, -1, 4) to each luma sample) may be limited to power-of-2 values ​​to save multipliers.

[0471] In one aspect of the present disclosure, the proposed new method can be reused for / combined with the CCLM discussed above, which utilizes a simple linear regression (SLR) model and uses one corresponding luma sample value to predict the chroma sample value. This is also referred to as the 1-tap case. In this case, deriving the linear model also includes deriving the scale parameter α and the offset parameter β by using the pre-operated adjacent luma sample values ​​and adjacent chroma sample values. Alternatively, the linear model can be rewritten as:

[0472] C=α·L+β (35)

[0473] Where L here represents the "pre-operation" luma sample. The parameter derivation of the 1-tap GLM can reuse the CCLM design, but consider the directional gradient (possibly with a high-pass filter). In one example, the scale parameter α can be derived by using a division lookup table as detailed below to achieve simplification.

[0474] In one example, when the GLM is combined with the SLR model, the scale parameter α and the offset parameter β can be derived by utilizing the minimum-maximum method discussed above. Specifically, the scale parameter α and the offset parameter β can be derived by comparing the adjacent luma sample values ​​of the pre-operation to determine the minimum luma sample value Y A and the maximum brightness sample value Y B ; respectively for the minimum brightness sample value Y A and the maximum brightness sample value Y B Determine the corresponding chroma sample value X A and X B ; and based on the minimum brightness sample value Y A , maximum brightness sample value Y B and the corresponding chroma sample value X A and X B , the scale parameter α and the offset parameter β are derived according to the following equations:

[0475]

[0476] β=Y A -αX A (36)

[0477] In one example, when combining the GLM with the SLR model, the rescaling discussed above can be reused. In this case, the encoder can determine the rescaling value (e.g., "u") to be signaled in the bitstream and add the rescaling value to the derived scale parameter α. The decoder can determine the rescaling value (e.g., "u") from the bitstream and add the rescaling value to the derived scale parameter α. The added value is ultimately used to predict the intra chroma sample values.

[0478] In one aspect of the present disclosure, the proposed new method can be reused / combined with FLM, which uses a multiple linear regression (MLR) model and uses multiple luma sample values ​​to predict chroma sample values. This is also called a multi-tap case, for example, 2 taps. In this case, the linear model can be rewritten as:

[0479]

[0480] In this case, multiple scale parameters α and offset parameters β can be derived using adjacent luma sample values ​​and adjacent chroma sample values ​​from the pre-operation. In one example, the offset parameter β is optional. In one example, at least one of the multiple scale parameters α can be derived using sample differences. In addition, another of the multiple scale parameters α can be derived using downsampled luma sample values. In one example, at least one of the multiple scale parameters α can be derived using horizontal or vertical sample differences calculated based on downsampled adjacent luma sample values. In other words, the linear model can combine multiple scale parameters α associated with different pre-operations.

[0481] Implicit filter shape derivation

[0482] In one example, instead of explicitly signaling the selected filter shape index, the directional orientation filter shape used can be derived at the decoder to save bit overhead. For example, at the decoder, several directional gradient filters can be applied to each reconstructed luminance sample of the L-shaped template of the i-th adjacent row and column of the current block. Then, the filtered values ​​(gradients) can be accumulated for each direction of the several directional gradient filters respectively. In the example, the accumulated values ​​are the accumulated values ​​of the absolute values ​​of the corresponding filtered values. After accumulation, the direction of the directional gradient filter for which the accumulated value is the largest can be determined as the derived (luminance) gradient direction. For example, a gradient histogram (HoG) can be constructed to determine the maximum value. The derived direction can be further applied as the direction for predicting the chrominance samples in the current block.

[0483] The following example involves reusing the decoder-side intra mode derivation (DIMD) method for luma intra prediction included in ECM-4.0:

[0484] Step 1: Apply two directional gradient filters (3x3 horizontal / vertical Sobel) to each reconstructed luminance sample of the L-shaped template of the second adjacent row and column of the current block;

[0485] Step 2: Accumulate the filtered values ​​(gradients) of each of the directional gradient filters by SAD (Sum of Absolute Difference);

[0486] Step 3: Constructing a Histogram of Gradients (HoG) based on the accumulated filtered values; and

[0487] Step 4: The maximum value in HoG is determined as the derived (luminance) gradient direction, based on which the GLM filter can be determined.

[0488] In one example, if the shape candidates are [-1, 0, 1; -1, 0, 1] (horizontal) and [1, 2, 1; -1, -2, -1] (vertical), when the maximum value is associated with the horizontal shape, the shape [-1, 0, 1; -1, 0, 1] is used for GLM-based chrominance prediction.

[0489] The gradient filter used to derive the gradient direction can be the same or different in shape from the GLM filter. For example, the two filters can be horizontal [-1, 0, 1; -1, 0, 1], or the two filters can have different shapes, and the GLM filter can be determined based on the gradient filter.

[0490] Classification

[0491] The proposed GLM can be combined with the MMLM or ELM discussed above. When combined with classification, each group can share or have its own filter shape, where the syntax indicates the shape of each group. For example, as an exemplary classifier, the horizontal gradient grad_hor can be classified into the first group corresponding to the first linear model, and the vertical gradient grad_ver can be classified into the second group corresponding to the second linear model. In one example, the horizontal brightness pattern can be generated only once.

[0492] A further possible classifier is provided below. Using the classifier, adjacent and internal luma-chroma sample pairs of the current video block can be classified into a plurality of groups based on one or more thresholds. Note that, as discussed above, each adjacent / internal chroma sample and its corresponding luma sample can be referred to as a luma-chroma sample pair. One or more thresholds are associated with the intensity of the adjacent / internal luma samples. In this case, each of the plurality of groups corresponds to a respective one of the plurality of linear models.

[0493] When combined with the MMLM classifier, the following operations can be performed: adjacent reconstructed luminance-chrominance sample pairs of the current video block are classified into two groups based on a threshold; different linear models are derived for different groups, wherein the derivation process can be GLM-simplified, that is, the number of taps is reduced by utilizing the above-mentioned pre-operation; similarly, the luminance-chrominance sample pairs within the CU (intra-luminance-chrominance sample pairs, wherein each intra-luminance-chrominance sample pair in the intra-luminance-chrominance sample pairs includes an intra-chrominance sample value to be predicted using the derived linear model) are classified into two groups based on a threshold; different linear models are applied to the reconstructed luminance samples in different groups; and chrominance samples in the CU are predicted based on different classified linear models.

[0494]

[0495] Among them, rec L ′(i,j) can be the reconstructed brightness sample after downsampling; rec C (i, j) can be reconstructed chroma samples (note: only neighbors are available); Threshold can be the average of neighboring reconstructed luma samples. Note that the number of categories (2) can be expanded to multiple categories by increasing the number of thresholds (e.g., equally divided based on the min / max values ​​of neighboring reconstructed (downsampled) luma samples, fixed, or signaled / switched at SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / subblock / sample level).

[0496] In one example, instead of the MMLM luma DC intensity, the filtered values ​​of the FLM / GLM applied to adjacent luma samples are used for classification. For example, if a 1-tap (1, -1) GLM is applied, the average AC value (physical meaning) is used. The processing can be: based on one or more filter shapes, one or more filtered values ​​and K-1 thresholds Ti, adjacent reconstructed luma-chroma sample pairs are classified into K groups; different MLR models are derived for different groups, wherein the derivation process can be GLM simplified, that is, the number of taps is reduced by the above-mentioned pre-operation; similarly, based on one or more filter shapes, one or more filtered values ​​and K-1 thresholds Ti, the luma-chroma sample pairs within the CU (intra-luma-chroma sample pairs, wherein each intra-luma-chroma sample pair in the intra-luma-chroma sample pairs includes an intra-chroma sample value to be predicted using the derived linear model) are classified into K groups; different linear models are applied to the reconstructed luma samples in different groups; and the chroma samples in the CU are predicted based on different classified linear models. Among them, the Threshold can be predefined (e.g., 0, or can be a table), or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. For example, the Threshold can be the average AC value (filtered value) of adjacent reconstructed (possibly downsampled) luma samples (2 groups), or equally divided based on the minimum / maximum AC (K groups).

[0497] It is also proposed to combine GLM with ELM classifiers. Figures 24A to 24D As shown in , a filter shape (e.g., 1 tap) can be selected to calculate the edge strength. The direction is determined as the direction along which the sample difference between the current sample and N adjacent samples (e.g., all 6 luma samples) is calculated. For example, Figure 24AThe filter in the upper middle of (shape [1, 0, -1; 1, 0, -1]) indicates the horizontal direction, because the sample differences between the samples in the horizontal direction can be calculated, and the filter below it (shape [1, 2, 1; -1, -2, -1]) indicates the vertical direction, because the sample differences between the samples in the vertical direction can be calculated. The positive coefficients and negative coefficients in each of the filters implement the calculation of the sample differences. Then, the processing may include: calculating an edge intensity through the filtered value (e.g., equivalent); quantizing the edge intensity into M segments through M-1 thresholds Ti; classifying the current sample using K categories (e.g., K == M); deriving different MLR models for different groups, wherein the derivation process may be GLM simplified, that is, using the above-mentioned pre-operation to reduce the number of taps; classifying the luminance-chrominance sample pairs within the CU into K groups; applying different MLR models to the reconstructed luminance samples in different groups; and predicting the chrominance samples in the CU based on the differently classified MLR models. Note that the filter shape used for classification can be the same as or different from the filter shape used for MLR prediction. Both the number of thresholds M-1 and the threshold value Ti can be fixed or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. In addition, other classifiers / combinations of classifiers as discussed in ELM can also be used for FLM and / or GLM.

[0498] If the number of classification samples in a group is less than a certain number (e.g., a predefined 4), the default values ​​mentioned when discussing the matrix derivation of the MLR model can be applied to the group parameters (α i , β). If the corresponding adjacent reconstructed samples are not available for the selected LM mode, then a default value may be applied, for example, when the MMLM_L mode is selected but the left sample is invalid.

[0499] Simplify and unify

[0500] In order to further improve the encoding and decoding efficiency, several methods related to the simplification of GLM are introduced as follows.

[0501] The matrix / parameter derivation in FLM requires floating point operations (e.g., closed form division), which is expensive for the decoder hardware, so a fixed point design is needed. For the 1-tap GLM case, it can be viewed as a modified luminance reconstruction sample generation of CCLM (e.g., horizontal gradient direction, from CCLM [1, 2, 1; 1, 2, 1] / 8 to GLM [-1, 0, 1; -1, 0, 1]), and the original CCLM process can be reused for GLM, including fixed point operations, MDLM downsampling, division table, applying size restrictions, min-max approximation and scale adjustment. The 1-tap GLM can have its own configuration or share the same design as CCLM for all entries. For example, a simplified min-max method is used to derive parameters (instead of LMS) and combined with scale adjustment after deriving the GLM model. In this case, the center point (luminance value y r ) becomes the average value of the reference luma sample "gradient". Another example, when GLM is on for this CU, CCLM slope adjustment is inferred to be off and no syntax related to slope adjustment needs to be signaled.

[0502] For example, this section uses the typical reference sample points (top row and left column). Figure 23 As shown in , the extended reconstruction region can also use simplifications in the same spirit and can have a syntax that indicates a specific region (such as MDLM, MRL).

[0503] Note that the following aspects can be combined and applied jointly: For example, combining reference sample downsampling and a division table to perform a division process.

[0504] When applying classification (MMLM / ELM), each group can apply the same or different simplification operations. For example, before applying the right shift, the samples of each group are padded to the target number of samples, and then the same derivation process and the same division table are applied.

[0505] Note that the implicit filter shape derivation method can also be used to determine whether to disable the downsampling process in the CCCM filter coefficients (using / not using the downsampling process is treated as different filter shapes).

[0506] Fixed-point implementation

[0507] The CCLM design can be reused for the 1-tap case, division by n can be implemented by right shift, and division by A2 can be implemented by LUT. α 、 n tableThe integration parameters for the intermediate parameters used to derive the linear model (Equations (19)-(20)) can be the same as CCLM or have different values ​​to have higher accuracy. The integration parameters can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level and can be conditioned on the sequence bit depth. For example, n table =bitdepth+4.

[0508] MDLM downsampling

[0509] When GLM is combined with MDLM, the total number of samples used for parameter derivation may not be a power of 2 value and needs to be padded to a power of 2 to replace the division with a right shift operation. For example, for an 8x4 chroma CU, MDLM requires W+H=12 samples, of which MDLM_T has only 8 samples available (reconstructed), and then the downsampled 4 samples (0, 2, 4, 6) can be equally padded. The code for implementing such an operation is shown below:

[0510]

[0511] Other filling methods may also be applied, such as repeated / mirrored filling relative to the last neighboring sample point (rightmost / lowest).

[0512] The filling method used for the GLM can be the same as or different from the filling method used for the CCLM.

[0513] Note that in the ECM version, the 8x4 chroma CU MDLM_T / MDLM_L requires 2T / 2L=16 / 8 samples respectively, in which case the same padding method can be applied to meet the target power-of-two number of samples.

[0514] Division LUT

[0515] The division LUT proposed for CCLM / LIC (Local Illumination Compensation) in known standard developments such as AVC / HEVC / AV1 / VVC / AVS can be used for GLM division. For example, the LUT in JCTVC-I0166 is reused for the case of bit depth = 10 (Table 4). The division LUT can be different from CCLM. For example, CCLM uses a min-max method with a division table such as in Equation 5, but GLM uses a 32-entry LMS division LUT such as in Table 5.

[0516] When GLM is combined with MMLM, meanL values ​​may not always be positive (e.g., using filtered / gradient values ​​to classify groups), so it is necessary to extract sgn(meanL) and use abs(meanL) to look up the division LUT. Note that the division LUT used for MMLM classification and parameter derivation can be different. For example, use a lower precision LUT (such as the LUT in the min-max method) for mean classification and a higher precision LUT (such as in LMS) for parameter derivation.

[0517] Size limits and latency constraints

[0518] Similar to CCLM design, some size restrictions can be applied to ELM / FLM / GLM. For example, the same constraints can be applied to the luma-chroma delay in the dual tree.

[0519] Size limits can be based on CU area / width / height / depth. The thresholds can be predefined or signaled at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. For example, for chroma CU area, the predefined threshold can be 128.

[0520] In one example, at least one pre-operation is performed in response to determining that a video block satisfies an enablement threshold, where the enablement threshold is associated with an area, width, height, or partition depth of the video block. Specifically, the enablement threshold may define a minimum or maximum area, width, height, or partition depth of the video block. As will be appreciated by those skilled in the art, a video block may include a current chroma block and its co-located luma block. It is also proposed that the enablement thresholds for the current chroma block and its co-located luma block be applied jointly. For example, in response to determining that both the current chroma block and its co-located luma block satisfy the enablement threshold, at least one pre-operation is performed.

[0521] Line buffer reduction

[0522] Similar to the CCLM design, if the co-located luma region of the current chroma CU contains the first row inside a CTU, the top template sample generation can be limited to 1 row to reduce the CTU line buffer storage. Note that when the above reference line is at the CTU boundary, only one luma line (common line buffer in intra prediction) is used to get the downsampled luma samples.

[0523] For example, in Figure 22In

[15] , if the co-located luma region of the current chroma CU contains the first row inside a CTU, the top template can be restricted to using only 1 row (but not 2) for parameter derivation (other CUs can still use 2 rows). This saves luma sample line buffer storage when processing CTUs line by line at the decoder hardware. Several methods can be used to achieve line buffer reduction. Note that the example of limited "1" row can be extended to N rows with similar operations. Similarly, 2 taps or multiple taps can also apply such operations. When multiple taps are applied, chroma samples may also need operations applied.

[0524] For example, using Figure 24A The 1-tap filter [1, 0, -1; 1, 0, -1] shown in FIG is explained as an example. This filter can be reduced to [0, 0, 0; 1, 0, -1], that is, only the coefficients of the lower row are used. Alternatively, the limited luma samples of the upper row can be filled with luma samples from the lower row (repeated, mirrored, 0, meanL, meanC, etc.).

[0525] Take N=4 as an example, that is, the video block is located at the top boundary of the current CTU, and the adjacent luminance sample values ​​and corresponding chrominance sample values ​​of the top 4 rows are used to derive the linear model. Please note that the corresponding chrominance sample values ​​can refer to the corresponding adjacent chrominance sample values ​​of the top 4 rows (for example, for YUV 4:4:4 format). Alternatively, the corresponding chrominance sample values ​​can refer to the corresponding adjacent chrominance sample values ​​of the top 2 rows (for example, for YUV 4:2:0 format). In this case, the adjacent luminance sample values ​​and corresponding chrominance sample values ​​of the top 4 rows can be divided into two regions: a first region including valid sample values ​​(for example, the luminance sample values ​​and corresponding chrominance sample values ​​of the most recent row) and a second region including invalid sample values ​​(for example, the luminance sample values ​​and corresponding chrominance sample values ​​of the other three rows). The coefficients of the filter corresponding to the sample positions that do not belong to the first region can then be set to zero, so that only the sample values ​​from the first region are used to calculate the sample difference. For example, as discussed above, in this case, the filter [1, 0, -1; 1, 0, -1] can be reduced to [0, 0, 0; 1, 0, -1]. Alternatively, the nearest sample values ​​in the first region can be padded to the second region so that the padded sample values ​​can be used to calculate the sample difference.

[0526] Fusion of chroma intra prediction modes

[0527] In one example, since GLM can be considered as a special CCLM mode, the fusion design can be reused or have its own way. Multiple (two or more) weights can be applied to generate the final prediction value. For example,

[0528] pred=(w0*pred0+w1*pred1+(1<<(shift-1)))>>shift

[0529] where pred0 is the predicted value based on the non-LM model and pred1 is the predicted value based on the GLM, or

[0530] pred0 is the prediction based on one of the CCLMs (including all MDLMs / MMLMs), and pred1 is the prediction based on the GLM, or

[0531] pred0 is the predicted value based on GLM, and pred1 is the predicted value based on GLM.

[0532] Different I / P / B slices may have different designs for weights w0 and w1 depending on whether the neighboring blocks are coded with CCLM / GLM / other coding modes or block size / width / height.

[0533] For example, the design of the weights can be determined by the intra prediction mode of the adjacent chroma blocks, and the shift is set to be equal to 2. Specifically, when both the upper and left adjacent blocks are encoded using the LM mode, {w0, w1} = {1, 3}; when both the upper and left adjacent blocks are encoded using the non-LM mode, {w0, w1} = {3, 1}; otherwise, {w0, w1} = {2, 2}. For non-I slices, both w0 and w1 can be set to be equal to 2.

[0534] For grammar design, if non-LM mode is selected, a flag is signaled to indicate whether fusion is applied.

[0535] 1-tap linear model

[0536] As described above, the 1-tap GLM has a good gain-complexity trade-off because it can reuse the existing CCLM module without introducing additional derivations. According to one or more aspects of the present disclosure, such a 1-tap design can be further extended or generalized.

[0537] In one aspect of the present disclosure, for a chroma sample to be predicted, a single corresponding luma sample L can be generated by combining the co-located luma sample and the adjacent luma samples. For example, the combination can be a combination of different linear filters, such as a high-pass gradient filter (GLM) and a low-pass smoothing filter (e.g., a [1, 2, 1; 1, 2, 1] / 8 FIR downsampling filter that can be commonly used in CCLM); and / or a combination of a linear filter and a nonlinear filter (e.g., with n-th power, such as L n, n can be a positive number, a negative number, or a +- fraction (e.g., +1 / 2, square root, or +3, cube, which can be rounded and rescaled to the bit depth dynamic range)).

[0538] In one aspect of the present disclosure, the combination can be applied repeatedly. For example, a combination of GLM and [1, 2, 1; 1, 2, 1] / 8 FIR can be applied to the reconstructed luma samples, and then a nonlinear power of 1 / 2 can be applied. For example, the nonlinear filter can be implemented as a LUT (lookup table), for example, for bit depth = 10, nth power, n = 1 / 2, LUT[i] = (int)(sqrt(i) + 0.5) << 5, i = 0 to 1023, where 5 will be scaled to the dynamic range of bit depth = 10. When the linear filter cannot efficiently handle the luma-chroma relationship, the nonlinear filter can provide an option. Whether to use the nonlinear term can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0539] In one or more aspects of the present disclosure, GLM may refer to a generalized linear model (which may be used to linearly or nonlinearly generate a single luminance sample, and the generated single luminance sample may be fed into a CCLM linear model to derive parameters of the CCLM linear model), and the linear / nonlinear generation may be referred to as a general pattern. Different gradients or general patterns may be combined to form another pattern. For example, a gradient pattern may be combined with a CCLM downsampled value; a gradient pattern may be combined with a nonlinear L 2 Value combination; A gradient pattern can be combined with another gradient pattern. The two gradient patterns to be combined can have different directions or the same direction. For example, [1, 1, 1; -1, -1, -1] and [1, 2, 1; -1, -2, -1], which both have vertical directions, can be combined. You can also combine [1, 1, 1; -1, -1, -1] and [1, 0, -1; 1, 0, -1], which have vertical and horizontal directions, can be combined, such as Figures 24A to 24D The combination may include additive, subtractive or linear weighting.

[0540] GLM applied on the downsampled domain

[0541] As described above, the pre-operation can be repeatedly applied, and the GLM can be applied to the pre-linearly weighted / pre-operated samples. For example, like CCLM, a template filter can be applied to the luma samples to remove outliers using a low-pass smoothing FIR filter [1, 2, 1; 1, 2, 1] / 8 (i.e., a CCLM downsampling smoothing filter) and generate downsampled luma samples (one downsampled luma sample corresponding to one chroma sample). After that, a 1-tap GLM can be applied to the smoothed downsampled luma samples to derive the MLR model.

[0542] Some gradient filter patterns, such as 3x3 Sobel or Privitt operator, can be applied to the downsampled luma samples. The following table shows some of the gradient filter patterns.

[0543]

[0544]

[0545] The gradient filter pattern can be combined with other gradient / generic filter patterns in the downsampled luma domain. In one example, the combined filter pattern can be applied to the downsampled luma samples. For example, the combined filter pattern can be derived by performing addition or subtraction operations on the corresponding coefficients of the gradient filter pattern and a DC / lowpass based filter pattern such as a filter pattern [0, 0, 0; 0, 1, 0; 0, 0, 0] or [1, 2, 1; 2, 4, 1; 1, 2, 1]. In another example, the combined filter pattern can be derived by performing addition or subtraction operations on the coefficients of the gradient filter pattern and a filter pattern such as L 2 The combined filter pattern is derived by performing an addition or subtraction operation on the nonlinear values ​​of the gradient filter pattern and another gradient filter pattern with a different or the same direction. In another example, the combined filter pattern is derived by performing an addition or subtraction operation on the corresponding coefficients of the gradient filter pattern and another gradient filter pattern with a different or the same direction. In another example, the combined filter pattern is derived by performing a linear weighting operation on the coefficients of the gradient filter pattern.

[0546] A GLM applied on the downsampled domain can fit into the CCCM framework, but may sacrifice high frequency accuracy because low-pass smoothing is applied before applying the GLM.

[0547] GLM used as input to CCCM

[0548] As shown above, just like CCLM, CCCM applies luma downsampling before convolution. When chroma subsampling is used, the reconstructed luma samples are downsampled to match the lower resolution chroma grid. Since a 1-tap GLM can also be viewed as changing the CCLM downsampling filter coefficients (e.g., from [1, 2, 1; 1, 2, 1] / 8 to [1, 2, 1, -1, -2, -1], i.e., from low pass to high pass), the GLM can be used as the input to the CCCM. Specifically, the gradient filter of the GLM replaces the luma downsampling filter ([1, 2, 1; 1, 2, 1] / 8) with gradient-based coefficients (e.g., [1, 2, 1, -1, -2, -1)). In this case, the CCCM operation becomes a "linear / nonlinear combination of gradients", as shown by the following equation:

[0549] predChromaVal=c0C+c1N+c2S+c3E+c4W+c5P+c6B

[0550] Where C, N, S, E, W, P are the gradients of the current or neighboring samples (compared to the original downsampled values ​​of CCCM). The related GLM methods described in this disclosure can be applied in the same way before entering the CCCM convolution, such as classification, separate Cb / Cr control, syntax, pattern combination, PU size limit, etc.

[0551] Gradient-based coefficient replacement can be applied to specific CCCM taps. Furthermore, not only high-pass coefficient replacement can be used, but also low-pass / band-pass / all-pass coefficient replacement. Replacement can be combined with the FLM / CCCM shape switching discussed above (resulting in a different number of taps). For example, Figures 24A to 24D The gradient pattern in can be used for replacement. In one example, the operation for applying GLM as input to CCCM includes: predefining one or more coefficient candidates for CCCM / FLM downsampling; determining the CCCM / FLM filter shape and the number of filter taps for the CU; applying different CCLM downsampling coefficients to different filter taps, where the coefficients can be a high-pass filter (GLM) or a low-pass / band-pass / all-pass filter; generating downsampled luma samples for CCCM input samples (using the applied coefficients); and feeding the generated downsampled luma samples into the CCCM process.

[0552] Some examples for changing the CCLM downsampling filter coefficients are shown below:

[0553] Example 1:

[0554] The candidate filters are [1, 2, 1; 1, 2, 1] / 8 and [1, 0, -1; 1, 0, -1]; predChromaVal = c0C+c1N+c2S+c3E+c4W+c5P+c6B, using the typical CCCM cross shape, 7 taps; N, S, W, E use the filter [1, 2, 1; 1, 2, 1] / 8, keeping the original CCLM downsampling filter; and C, P use the filter [1, 0, -1; 1, 0, -1], i.e., horizontal gradient filter, and then P physically means gradient^2.

[0555] Example 2:

[0556] The candidate filters are [1, 2, 1; 1, 2, 1] / 8, [1, 0, -1; 1, 0, -1], [1, 2, 1; -1, -2, -1], [2, 1, -1; 1, 1, -1, -2] and [-1, 1, 2; -2, -1, 1]; predChromaVal = c0C0+c1C1+c2C2+c3C3+c4C4+c5P+c6B; C0 uses the filter [1, 2, 1; 1, 2, 1] / 8, keeping the original CCLM downsampling filter; C1 uses the filter [1, 0, -1; 1, 0, -1]; C2 uses filter [1, 2, 1; -1, -2, -1]; C3 uses filter [2, 1, -1; 1, -1, -2]; C4 uses filter [-1, 1, 2; -2, -1, 1]; C5 uses filter [1, 2, 1; 1, 2, 1] / 8, keeping the original CCLM downsampling filter; C0 to C5, and P have the same downsampled luminance position (in the typical CCCM cross shape == C); and C1 to C4 are generated by Sobel-based gradient filters in different directions (such as 24A to 24D ).

[0557] Example 3:

[0558] The candidate filters are [1, 2, 1; 1, 2, 1] / 8, [1, 0, -1; 1, 0, -1], [1, 2, 1; -1, -2, -1], [0, 1, 1; 0, 1, 1] and [1, 1, 0; 1, 1, 0]; redChromaVal = c0C0+c1C1+c2C2+c3C3+c4C4+c5P+c6B; C0 uses the filter [1, 2, 1; 1, 2, 1] / 8, keeping the original CCLM downsampling filter; C1 uses the filter [1, 0, -1; 1, 0, -1]; C2 uses the filter [1, 2, 1; -1, -2, -1]; C3 uses the filter [0, 1, 1; 0, 1, 1]; C4 uses the filter [1, 1, 0; 1, 1, 0]; C5 uses the filter [1, 2, 1; 1, 2, 1] / 8, keeping the original CCLM downsampling filter; C0 to C5, and P have the same downsampled brightness position (in the typical CCCM cross shape == C); C1 to C2 are generated by gradient filters based on different directional Sobels (such as Figures 24A to 24D ); and C3 to C4 are generated by a low-pass filter.

[0559] Which CCCM / FLM taps to which coefficient replacement is applied may be predefined (as in the above examples) or signaled / switched in the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0560] For each CCCM / FLM tap, coefficient candidates for CCCM / FLM downsampling may be predefined or signaled / switched in SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0561] Example 4:

[0562] The candidate filters are [1, 2, 1; 1, 2, 1] / 8, [1, 0, -1; 1, 0, -1], [1, 2, 1; -1, -2, -1], [2, 1, -1; 1, -1, -2] and [-1, 1, 2; -2, -1, 1]; predChromaVal = c0C+c1N+c2S+c3E+c4W+c5P+c6B, using the typical CCCM cross shape, 7 taps; C uses the switched downsampling filter among the 5 candidate filters; N, S, W, E use the filter [1, 2, 1; 1, 2, 1] / 8, keeping the original CCLM downsampling filter; and P uses the switched downsampling filter among the 5 candidate filters.

[0563] Example 5:

[0564] The candidate filters are: [1, 2, 1; 1, 2, 1] / 8, [1, 0, -1; 1, 0, -1], [1, 2, 1; -1, -2, -1], [0, 1, 1; 0, 1, 1] and [1, 1, 0; 1, 1, 0]; predChromaVal = c0C+c1W+c2E+c3P+c4B, i.e., horizontal minus shape, 5 taps; C uses the switched downsampling filter among the 5 candidate filters; W and E use the switched downsampling filters among the 3 candidate filters: [1, 2, 1; 1, 2, 1] / 8, [0, 1, 1; 0, 1, 1] and [1, 1, 0; 1, 1, 0]; and P uses the filter [1, 2, 1; 1, 2, 1] / 8, maintaining the original CCLM downsampling filter.

[0565] grammar

[0566] In one or more aspects of the present disclosure, one or more grammars may be introduced to indicate information about GLM. An example of a GLM grammar is shown in Table 10 below.

[0567] Table 10

[0568]

[0569]

[0570] FLC: Fixed Length Code

[0571] TU: Truncated Unary Code

[0572] EGk: Exponential Golomb code with order k, where k can be fixed or signaled / switched in SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0573] SVLC: Signed EG0

[0574] UVLC: Unsigned EG0

[0575] Note that the binarization of each syntax element can be changed.

[0576] In one aspect of the present disclosure, the GLM on / off control for Cb / Cr components can be jointly or individually. For example, at the CU level, a single flag can be used to indicate whether GLM is active for that CU. If it is active, a single flag can be used to indicate whether both Cb / Cr are active. If it is not active, the single flag indicating Cb or Cr is active. When Cb and / or Cr are active, the filter index / gradient (generic) pattern can be signaled individually. All flags can have their own context model or be bypass-coded.

[0577] In another aspect of the present disclosure, whether to signal the GLM on / off flag can depend on the luminance / chrominance coding mode and / or CU size. For example, in the ECM5 chrominance intra-frame mode syntax, when applying MMLM or MMLM_L or MMLM_T, GLM can be inferred as off; when the CU area < A, where A can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level; if combined with CCCM, when CCCM is on, GLM can be inferred as off.

[0578] Note that when GLM is combined with MMLM, different models can share the same gradient / generic pattern or have their own gradient / generic pattern.

[0579]

[0580] When GLM is combined with CCCM / FLM, if the current CU is implemented as CCCM / FLM, then the CU-level GLM enable flag can be inferred as off.

[0581] hasGlmFlag &=!pu.cccmFlag;

[0582] CCCM without downsampling [[ID=二十]]

[0583] CCCM needs to process the downsampled luminance reference value before calculating the model parameters and applying the CCCM model, which burdens the decoder processing cycle. In this section, a CCCM that does not utilize the downsampling process is proposed, including different selections that utilize non-downsampled luminance reference values and / or non-downsampled luminance references. One or more filter shapes can be used for the purposes described below.

[0584] In one example, a convolutional 7-tap filter may include a 5-tap plus-shaped spatial component, a nonlinear term, and a bias term. The input to the spatial 5-tap component of the filter includes the center (C) non-subsampled luma sample (which is co-located with the chroma sample to be predicted) and its non-subsampled above or north (N), below or south (S), left or west (W), and right or east (E) neighbors, as shown in FIG. Figure 17 As shown in .

[0585] The nonlinear term P is expressed as a power of 2 of the center luma sample C and scaled to the content's sample value range:

[0586] P=(C*C+midVal)>>bitDepth

[0587] That is, for 10-bit content, it is calculated as:

[0588] P=(C*C+512)>>10

[0589] The bias term B represents a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the middle chroma value (512 for 10-bit content).

[0590] The output of the filter is calculated as the convolution between the filter coefficients ci and the input value and is clipped to the range of valid chroma samples:

[0591] predChromaVal=c0C+c1N+c2S+c3E+c4W+c5P+c6B

[0592] In another example, a convolutional 7-tap filter may include a 6-tap rectangular shaped spatial component and a bias term. The input to the spatial 6-tap component of the filter includes the center (b) non-downsampled luma sample (which is co-located with the chroma sample to be predicted) and its downsampled lower left or southwest (d), lower right or southeast (f), lower or south (e), left or west (a), and right or east (c) neighbors, as shown in FIG. Figure 25 The shape is shown in 1.

[0593] The bias term B represents a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the middle chroma value (512 for 10-bit content).

[0594] The output of the filter is calculated as the convolution between the filter coefficients ci and the input value and is clipped to the range of valid chroma samples:

[0595] predChromaVal=c0a+c1b+c2c+c3d+c4e+c5f+c6B

[0596] In yet another example, the convolutional 8-tap filter may consist of a 6-tap rectangular shaped spatial component, a nonlinear term, and a bias term. The input to the spatial 6-tap component of the filter consists of the following terms: the center (b) non-subsampled luma sample (which is co-located with the chroma sample to be predicted) and its non-subsampled lower left or southwest (d), lower right or southeast (f), lower or south (e), left or west (a), and right or east (c) neighbors, as shown in FIG. Figure 25 The shape is shown in 1.

[0597] The nonlinear term P is expressed as a power of 2 of the center luma sample (b) and scaled to the content's sample value range:

[0598] P=(b*b+midVal)>>bitDepth

[0599] That is, for 10-bit content, it is calculated as:

[0600] P=(b*b+512)>>10

[0601] The bias term B represents a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the middle chroma value (512 for 10-bit content).

[0602] The output of the filter is calculated as the convolution between the filter coefficients ci and the input value and is clipped to the range of valid chroma samples:

[0603] predChromaVal=c0a+c1b+c2c+c3d+c4e+c5f+c6P+c7B

[0604] It should be understood that the examples shown above are merely sample examples and that other implementations may be possible without departing from the present disclosure, such as having more or fewer taps and having Figure 25 any of the shapes shown in (where the chroma samples to be predicted are represented as circles).

[0605] In yet another example, a convolutional 9-tap filter may consist of a 6-tap rectangular shape spatial component, two nonlinear terms, and a bias term. The input to the spatial 6-tap component of the filter consists of the following terms: the center (b) non-subsampled luma sample (which is co-located with the chroma sample to be predicted and is substantially located at the center of the filter shape) and its non-subsampled lower left / southwest (d), lower right / southeast (f), lower / south (e), left / west (a), and right / east (c) neighbors, as shown in FIG. Figure 25 The shape is shown in 1.

[0606] The nonlinear terms P and Q are two nonlinear luma sample values ​​expressed as powers of the luma sample value of the center (b) luma sample and the luma sample value of the bottom / south (e) luma sample, respectively, and then scaled to the sample value range of the content:

[0607] P = (b*b+midVal)>>bitDepth;

[0608] Q=(e*e+midVal)>>bitDepth.

[0609] That is, for 10-bit content, it is calculated as:

[0610] P = (b*b+512)>>10;

[0611] Q=(e*e+512)>>10.

[0612] The bias term B is a bias value representing a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the middle chroma value (512 for 10-bit content).

[0613] The output of the filter is calculated as the convolution between the filter coefficients ci and the input value and is clipped to the range of valid chroma samples:

[0614] predChromaVal=c0a+c1b+c2c+c3d+c4e+c5f+c6P+c7Q+c8B

[0615] It should be understood that the nonlinear terms P and Q can be expressed as powers of any luma sample value of the filter's non-subsampled luma samples. The two nonlinear terms P and Q are merely exemplary, and the corresponding chroma sample values ​​can be calculated based on one or more nonlinear values.

[0616] In yet another example, as in Figure 30 As shown in , the convolution 9-tap filter may include a 6-tap spatial term, two nonlinear terms, and a bias term. The 6-tap spatial term corresponds to the chrominance samples to be predicted (i.e., Figure 30 Note that the two adjacent original reconstructed luma samples used in the nonlinear term can be replaced by any two adjacent original reconstructed luma samples, i.e., Figure 30 The ones shown in do not have downsampling (ie, L0, L1, ..., L5).

[0617]

[0618] Among them, α i Yes and L iThe associated coefficients are α and β are offsets. As with the existing CCCM design, up to 6 rows / columns of chroma samples above and to the left of the current CU are used to derive the filter coefficients. The filter coefficients are derived based on the same LDL decomposition method used in CCCM. The convolutional 9-tap filter can be signaled as an additional CCCM model in addition to the existing CCCM model. For signaling, when CCCM is selected, a single flag is signaled and used for both chroma components to indicate whether the default CCCM model or the proposed additional CCCM model is applied.

[0619] In yet another example, a convolutional N-tap (N is an integer and greater than 1) filter may be composed of (N-1-M)-tap (M is an integer) spatial terms, M nonlinear terms, and bias terms. The (N-1-M)-tap spatial terms correspond to adjacent luma samples (i.e., L0, L1, ..., L1) of the chroma sample to be predicted. N-1-M ). Note that the adjacent original reconstructed luminance samples used in the spatial term and the nonlinear term can be replaced by any adjacent original reconstructed luminance samples, i.e., without downsampling (i.e., L0, L1, ..., L N-1-M ).

[0620]

[0621] where α i Yes and L i The associated coefficients are β and β is the offset. As with the existing CCCM design, up to 6 rows / columns of chroma samples above and to the left of the current CU are used to derive the filter coefficients. The filter coefficients are derived based on the same LDL decomposition method used in CCCM. This method can be signaled as an additional CCCM model in addition to the existing CCCM model. For signaling, when CCCM is selected, a single flag is signaled and used for both chroma components to indicate whether the default CCCM model or the proposed additional CCCM model is applied.

[0622] Please note that the methods / examples in this section can be combined / reused with the methods mentioned above, including but not limited to methods related to classification, filter shape, matrix derivation (with special processing), application area and syntax. In addition, the methods / examples listed in this section can also be applied together with the above methods / examples (more taps) to have better performance and certain complexity trade-offs.

[0623] In this disclosure, reference samples / training templates / reconstructed neighboring regions generally refer to luma samples used to derive MLR model parameters, which are then applied to internal luma samples in a CU to predict chroma samples in the CU.

[0624] According to one or more embodiments of the present disclosure, the reference sample / training template / reconstructed neighboring region may be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. For example, Figure 9A An example of an L-shaped reconstruction region, a left / top reconstruction region, is shown to derive parameters.

[0625] Filter shape

[0626] One or more shapes / number of filter taps may be used for CCCM prediction, such as Figure 25 、 Figure 26 and FIG. 27A to FIG. 27B One or more sets of filter taps can be used for FLM prediction, for example in Figures 28A to 28G The selected luma reference value is non-subsampled. One or more predefined shapes / number of filter taps may be used for CCCM prediction based on previously decoded information at TB / CB / slice / picture / sequence level.

[0627] Although a multi-tap filter can fit the training data (i.e., top / left adjacent reconstructed luma / chroma samples) well, in some cases, the training data does not capture all the characteristics of the test data and may lead to overfitting and may not predict the test data (i.e., the chroma block samples to be predicted) well. Moreover, different filter shapes can adapt well to different video block contents, resulting in more accurate predictions. To address this issue, the filter shape / number of filter taps can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. The filter shape candidate set can be predefined or signaled / switched at one or more coding levels (i.e., at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level). Different components (U / V) can have different filter switch controls. For example, as shown in the following table, the predefined filter shape candidate set (index = 0 to 5) and the filter shape (1, 2) represent a 2-tap luminance filter, while the filter shape (1, 2, 4) represents a 2-tap luminance filter. Figure 20 3-tap luma filter shown in . The filter shape selection for the U / V component can be switched in the PH or in the CU / CTU level. Note that, as described above, N-tap can represent N-tap with or without offset β.

[0628] In another example, a syntax element is signaled at a relatively high coding level (e.g., SPS level) to indicate whether different filter shapes and / or the number of filter shapes are enabled. If different filter shapes are enabled or the number of filter shapes is indicated, another syntax element is signaled at a lower coding level (e.g., CU level) to indicate which filter shapes and / or the number of filters are used at the lower coding level. In an example, a syntax element may be signaled at the same coding level (e.g., SPS level) to indicate which filter shapes are used.

[0629] In another example, a syntax element is signaled at a relatively high coding level (e.g., SPS level) to indicate whether CCCM is enabled and whether different filter shapes and / or the number of filter shapes are enabled (e.g., 0: CCCM off, 1: CCCM with 1 filter, 2: CCCM with 2 filters, ..., N: CCCM with N filters, where N is an integer). If CCCM is enabled and different filter shapes are disabled, another syntax element may be signaled at a lower level to indicate whether CCCM is used in the block at the lower level. If CCCM is enabled and different filter shapes are enabled or the number of filter shapes is indicated, another syntax element may be signaled at a lower or same coding level to indicate which filter shapes and / or the number of filters are used.

[0630] In another example, a syntax element is signaled at a relatively high coding level (e.g., SPS level) to indicate whether CCCM without downsampling is enabled. If CCCM without downsampling is enabled at a high coding level, a syntax element is further signaled at a lower coding level (e.g., CU level) to indicate whether CCCM without downsampling is used for blocks at the lower coding level.

[0631]

[0632] Different chroma types / color formats can have different predefined filter shapes / taps. For example, using predefined filter shapes for 420 type 0: (1, 2, 4, 5), 420 type 2: (0, 1, 2, 4, 7), 422: (1, 4), 444: (0, 1, 2, 3, 4, 5), as shown in Figure 21 As shown in .

[0633] The unavailable luminance / chrominance samples used to derive the MLR model can be filled from the available reconstructed samples. Figure 21In the 6-tap (0, 1, 2, 3, 4, 5) filter, for the CU located at the left picture boundary, the left column including (0, 3) is unavailable (outside the picture boundary), so (0, 3) is repeated from (1, 4) to apply the 6-tap filter. Note that the padding process is applied in both the training data (top / left adjacent reconstructed luma / chroma samples) and the test data (luma / chroma samples in the CU).

[0634] According to one or more embodiments of the present disclosure, unavailable luma / chroma samples for deriving the MLR model may be skipped and not used. Then, no padding process is required for the unavailable luma / chroma samples.

[0635] CCLM / MMLM with LDL decomposition

[0636] CCCM needs to process LDL decomposition to calculate the model parameters of the CCCM model, avoiding the use of square root operations and requiring only integer operations. In this section, CCLM / MMLM with LDL decomposition is proposed. As mentioned above, LDL decomposition can also be used in ELM / FLM / GLM.

[0637] Please note that the methods / examples in this section can be combined / reused with the methods mentioned above, including but not limited to methods related to classification, filter shape, matrix derivation (with special processing), application area and syntax. In addition, the methods / examples listed in this section can also be applied together with the methods / examples above to have better performance and certain complexity trade-offs.

[0638] In this disclosure, reference samples / training templates / reconstructed neighboring regions generally refer to luma samples used to derive MLR model parameters, which are then applied to internal luma samples in a CU to predict chroma samples in the CU.

[0639] CCLM / MMLM with extended range

[0640] One or more reference samples can be used for CCLM / MMLM prediction, i.e. Figure 18 As shown in , the reference region can be the same as the reference region in CCCM. Based on the previous decoding information at TB / CB / slice / picture / sequence level, different reference regions can be used for CCLM / MMLM prediction.

[0641] Although training data with multiple reference regions can be a good fit for the calculation of model parameters, in some cases, the training data does not capture all the characteristics of the test data, but it may lead to overfitting and may not predict the test data well (i.e., the chroma block samples to be predicted). In addition, different reference regions can be well adapted to different video block contents, resulting in more accurate predictions. To address this issue, the reference shape / number of reference regions can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. The reference region candidate set can be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. Different components (U / V) can have different reference region switch controls. For example, the predefined reference region candidate set (index = 0 to 4) is shown in the table below. The reference region selection for the U / V component can be switched in the PH or in the CU / CTU level. Different colorimetric types / color formats may have different predefined reference areas.

[0642]

[0643] Unavailable luma / chroma samples used to derive the MLR model can be filled from available reconstructed samples, and the filling process is applied in both training data (top / left neighboring reconstructed luma / chroma samples) and test data (luma / chroma samples in the CU).

[0644] According to one or more embodiments of the present disclosure, unavailable luma / chroma samples for deriving the MLR model may be skipped and not used. Then, no padding process is required for the unavailable luma / chroma samples.

[0645] FLM / GLM / ELM / CCCM with minimum sample point restriction

[0646] FLM needs to process the downsampled luma reference and calculate the model parameters, which burdens the decoder processing cycle, especially for small blocks. In this section, FLM with minimum sample restrictions is proposed, for example, FLM is only used for samples larger than a predefined number (such as 64, 128). One or more different restrictions can be used for this purpose, for example, FLM is only used for samples larger than a predefined number (such as 256) in a single model, and FLM is only used for samples larger than a predefined number (such as 128) in a multi-model.

[0647] According to one or more embodiments of the present disclosure, the predefined minimum number of samples for a single model may be greater than or equal to the predefined minimum number of samples for a multi-model. For example, FLM / GLM / ELM / CCCM is only used for samples greater than or equal to a predefined number (such as 128) in a single model, and FLM / GLM / ELM / CCCM is only used for samples greater than or equal to a predefined number (such as 256) in a multi-model.

[0648] According to one or more embodiments of the present disclosure, the predefined minimum number of samples for FLM / GLM / ELM may be greater than or equal to the predefined minimum number of samples for CCCM. For example, CCCM may be used only in a single model for a number of samples greater than or equal to a predefined number (such as 0), and CCCM may be used only in a multi-model for a number of samples greater than or equal to a predefined number (such as 128). FLM may be used only in a single model for a number of samples greater than or equal to a predefined number (such as 128), and FLM may be used only in a multi-model for a number of samples greater than or equal to a predefined number (such as 256).

[0649] Please note that the methods / examples in this section can be combined / reused with the methods mentioned above, including but not limited to methods related to classification, filter shape, matrix derivation (with special processing), application area and syntax. In addition, the methods / examples listed in this section can also be applied together with the above methods / examples (more taps) to have better performance and certain complexity trade-offs.

[0650] Multi-mode combination of FLM / GLM / ELM / CCCM / CCLM

[0651] According to one or more embodiments of the present disclosure, two models from the multiple modes of FLM / GLM / ELM / CCCM / CCLM can be further combined to bring additional coding efficiency. For example, the parameters of CCCM (ci) and GLM (a, b) are first derived separately, and then the weight (w) between CCCM and GLM is derived by linear regression. i ), and finally use the weighted CCCM and GLM to predict the chrominance samples from the reconstructed luma samples.

[0652] GLMpredChromaVal=a*lumaVal+b

[0653] CCCMpredChromaVal=c0*C+c1*N+c2*S+c3*E+c4*W+c5*P+c6*B

[0654] FinalpredChromaVal=w0*GLMPredChromaVal+w1*CCCMPredChromaVal

[0655] According to one or more embodiments of the present disclosure, there is a flag signaled / switched in the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level to indicate whether the combined mode is used.

[0656] According to one or more embodiments of the present disclosure, instead of explicitly signaling the selected mode flag, the mode flag may be derived at the decoder to save bit overhead.

[0657] (1) Determine M combination candidates for the current CU

[0658] (2) Divide the available L-shaped template area into N regions, which are denoted as R0, R1, ... R N-1

[0659] (Divide the training data into N sets and train N times)

[0660] (3) Apply the M combined candidates to a portion of the available template area (which can be R0, R1, ... R N-1 Single or multiple regions in

[0661] (4) Derive M sets of filter coefficients (according to the M filter shapes), denoted as F0, F1, ... F M-1

[0662] (5) The derived F0, F1, ... F M-1 The set of filter coefficients is applied to the other part of the available template area, which is different from the part of the available template in (3)

[0663] (6) The error is accumulated by SAD, SSD or SATD and is expressed as E0, E1, ... E M-1

[0664] (7) Sort and select the K smallest errors, which are denoted as E'0, E'1, ... E' K-1 , which corresponds to K combinations of filters / K sets of filter coefficients

[0665] (8) Signal and select 1 of the K combined filters to apply to the current CU for chroma prediction; if K is 1, no signaling is required (the filter shape with the smallest error is the applied combined filter)

[0666] For example,

[0667] (1) Predefine 3 filter candidates for the current CU, such as CCCM, GLM, and combined CCCM and GLM

[0668] (2) Divide the available L-shaped template area (CCCM 6 chroma rows / columns, note that in CCCM design, each chroma sample involves 6 luma samples for downsampling) into two regions, denoted as R0 and R1

[0669] For example, even rows / columns: R0, odd rows / columns: R1

[0670] Figure 29A The following example is shown: in the template area, the even-numbered row area R0 is used to train / derive 3 sets of filter coefficients, and the odd-numbered row area R1 is used to verify / compare and sort the costs of the 3 sets of filter coefficients.

[0671] (3) Independently apply the 3 filter candidates to a portion of the available template area (single R0)

[0672] (4) Derivation of three sets of filter coefficients (based on the four filter shapes), denoted as F0, F1, F2

[0673] (5) Applying the derived set of F0, F1, F2 filter coefficients to other parts (R1) of the available template area than the part of the available template in (3).

[0674] (6) Accumulated errors are expressed as E0, E1, and E2 through SAD, SSD, or SATD.

[0675] (7) Sort and select 1 minimum error, denoted as E'0, which corresponds to 1 filter shape / 1 filter coefficient set.

[0676] (8) If K is 1, no signaling is required (the filter with the smallest error is the filter shape applied)

[0677] Please note that the methods / examples in this section can be combined / reused from the methods mentioned in all sections, including but not limited to classification, filter shape, matrix derivation (with special processing), application area, syntax. In addition, the methods / examples listed in this section can also be applied to all sections to have better performance and certain complexity trade-offs.

[0678] CCCM with non-subsampled and subsampled luminance reference values

[0679] Only downsampled luma reference values ​​can be used in CCCM to calculate model parameters and apply the CCCM model. In this section, non-downsampled luma reference values ​​are also used in CCCM to calculate model parameters and apply the CCCM model, including using non-downsampled luma reference values ​​and downsampled luma reference values ​​in different or the same positions. As described above, one or more filter shapes can be used for this purpose.

[0680] In one example, a convolutional 8-tap filter may include a 6-tap rectangular shaped spatial component, downsampled luma samples, and a bias term. The input to the spatial 6-tap component of the filter includes the center (b) non-downsampled luma sample (which is co-located with the chroma sample to be predicted) and its non-downsampled lower left or southwest (d), lower right or southeast (f), lower or south (e), left or west (a), and right or east (c) neighbors, as shown in FIG. Figure 25 1, and the center (C) downsampled luma sample (which is co-located with the chroma samples to be predicted).

[0681] The bias term B represents a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the middle chroma value (512 for 10-bit content).

[0682] The output of the filter is calculated as the convolution between the filter coefficients ci and the input value and is clipped to the range of valid chroma samples:

[0683] predChromaVal=c0a+c1b+c2c+c3d+c4e+c5f+c6C+c7B

[0684] In another example, a convolutional 9-tap filter can be composed of a 6-tap rectangular shape spatial component, two nonlinear terms, and a bias term. The input to the spatial 6-tap component of the filter consists of the following terms: the center (b) non-subsampled luma sample (which is co-located with the chroma sample to be predicted and is substantially located at the center of the filter shape) and its non-subsampled lower left / southwest (d), lower right / southeast (f), lower / south (e), left / west (a), and right / east (c) neighbors, as shown in Figure 25 1, and the center (C) downsampled luma sample (which is co-located with the chroma sample to be predicted (e.g., the center (C) downsampled luma sample value is determined by a weighted averaging operation)).

[0685] The nonlinear terms P and Q are two nonlinear luma sample values ​​expressed as the power of the luma sample value of the center (b) non-subsampled luma sample and the power of the luma sample value of the center (c) subsampled luma sample, respectively, and then scaled to the sample value range of the content:

[0686] P = (b*b+midVal)>>bitDepth;

[0687] Q=(C*C+midVal)>>bitDepth.

[0688] That is, for 10-bit content, it is calculated as:

[0689] P = (b*b+512)>>10;

[0690] Q=(C*C+512)>>10.

[0691] The bias term B is a bias value representing a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the middle chroma value (512 for 10-bit content).

[0692] The output of the filter is calculated as the convolution between the filter coefficients ci and the input value and is clipped to the range of valid chroma samples:

[0693] predChromaVal=c0a+c1b+c2c+c3d+c4e+c5f+c6P+c7Q+c8B

[0694] It should be understood that the nonlinear terms P and Q can be expressed as powers of any luma sample value used to predict the downsampled / non-downsampled luma sample for the corresponding chroma sample value. The two nonlinear terms P and Q are merely exemplary, and the corresponding chroma sample values ​​can be calculated based on one or more nonlinear values.

[0695] As mentioned above, the nonlinear term Q can also be expressed as a power of the non-subsampled luma samples of the filter. In such an example, the convolutional 9-tap filter can be composed of a 6-tap rectangular shape spatial component, two nonlinear terms, and a bias term. The input to the spatial 6-tap component of the filter consists of the following terms: the center (b) non-subsampled luma sample (which is co-located with the chroma sample to be predicted and is essentially located in the center of the filter shape) and its non-subsampled lower left / southwest (d), lower right / southeast (f), lower / south (e), left / west (a), and right / east (c) neighbors, as shown in Figure 25 The shape is shown in 1.

[0696] The nonlinear terms P and Q are two nonlinear luma sample values ​​expressed as powers of the luma sample value of the center (b) luma sample and the luma sample value of the bottom / south (e) luma sample, respectively, and then scaled to the sample value range of the content:

[0697] P = (b*b+midVal)>>bitDepth;

[0698] Q=(e*e+midVal)>>bitDepth.

[0699] That is, for 10-bit content, it is calculated as:

[0700] P = (b*b+512)>>10;

[0701] Q=(e*e+512)>>10.

[0702] The bias term B is a bias value representing a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the middle chroma value (512 for 10-bit content).

[0703] The output of the filter is calculated as the convolution between the filter coefficients ci and the input value and is clipped to the range of valid chroma samples:

[0704] predChromaVal=c0a+c1b+c2c+c3d+c4e+c5f+c6P+c7Q+c8B

[0705] In yet another example, the convolutional 9-tap filter may include an original CCCM 7-tap term corresponding to the downsampled reconstructed luma sample and a 2-tap spatial term corresponding to 2 adjacent original reconstructed luma samples, i.e., Figure 30 The CCCM 7-tap filter consists of a 5-tap plus-shape spatial component, a nonlinear term, and a bias term. The input to the spatial 5-tap component of the filter consists of the following terms: the center (C) luma sample (which is co-located with the chroma sample to be predicted), and its above / north (N), below / south (S), left / west (W), and right / east (E) neighbors, as shown in FIG. Figure 17 As shown in .

[0706] The nonlinear term P is expressed as a power of 2 of the center luma sample C and scaled to the content's sample value range:

[0707] P=(C*C+midVal)>>bitDepth

[0708] That is, for 10-bit content, it is calculated as:

[0709] P=(C*C+512)>>10

[0710] The bias term B represents a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the intermediate chrominance value (512 for 10-bit content).

[0711] The output of the filter is calculated as the convolution between the filter coefficients ci and the input value, and is clipped to the range of valid chroma samples:

[0712] predChromaVal=c0C+c1N+c2S+c3E+c4W+c5P+c6B+c7L2+c8L3

[0713] Please note that L2 and L3 can be replaced by any two adjacent original reconstructed luminance samples, i.e., Figure 30The ones shown in do not have downsampling (ie, L0, L1, ..., L5).

[0714] In yet another example, a convolutional N-tap (N is an integer and greater than 7) filter may include an original CCCM 7-tap term corresponding to the downsampled reconstructed luma sample, an (N-7-M)-tap (M is an integer) spatial term corresponding to the adjacent original reconstructed luma samples of the chroma sample, and M nonlinear terms, i.e., Figure 30 , without downsampling to the chroma samples (i.e., L0, L1, ..., L5). The CCCM 7-tap filter consists of a 5-tap plus-shape spatial component, a nonlinear term, and a bias term. The input to the spatial 5-tap component of the filter consists of the following terms: the center (C) luma sample (which is co-located with the chroma sample to be predicted) and its above / north (N), below / south (S), left / west (W), and right / east (E) neighbors, as shown in Figure 17 As shown in .

[0715] The nonlinear term P is expressed as a power of 2 of the center luma sample C and is scaled to the sample value range of the content:

[0716] P=(C*C+midVal)>>bitDepth

[0717] That is, for 10-bit content, it is calculated as:

[0718] P=(C*C+512)>>10

[0719] The bias term B represents a scalar offset between the input and output (similar to the offset term in CCLM) and is set to the intermediate chrominance value (512 for 10-bit content).

[0720] The output of the filter is calculated as the convolution between the filter coefficients ci and the input value, and is clipped to the range of valid chroma samples:

[0721]

[0722]

[0723] Note that the original reconstructed luminance samples used in the spatial term and the nonlinear term can be replaced by any adjacent original reconstructed luminance samples, i.e., Figure 30 As shown in , there is no downsampling (ie, L0, L1, ..., L5).

[0724] It should be understood that the filter shape is not limited to the plus shape used by CCCM and can be any shape desired. For example, the filter shape can be as follows Figure 25 Any of the shapes shown in .

[0725] Please note that the methods / examples in this section can be combined / reused from the methods mentioned above, including but not limited to classification, filter shape, matrix derivation (with special processing), application area, syntax. In addition, the methods / examples listed in this section can also be applied to the methods mentioned above (more taps) to have better performance and some complexity trade-offs.

[0726] In this disclosure, reference samples / training templates / reconstructed neighboring regions generally refer to luma samples used to derive MLR model parameters, which are then applied to internal luma samples in a CU to predict chroma samples in the CU.

[0727] According to one or more embodiments of the present disclosure, CCCM without downsampling and CCCM with downsampling may be used with different taps or shapes. For example, the number of filter shapes / filter taps with and without downsampling values ​​may be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level.

[0728] In one example, one filter with downsampled values ​​and one filter without downsampled values ​​may be signaled / switched at the CU level. The filter with downsampled values ​​is the same as the CCCM. The filter without downsampled values ​​may consist of (N-1-M) tapped (N and M are integers) spatial terms, M nonlinear terms, and a bias term. The (N-1-M) tapped spatial terms correspond to the adjacent original reconstructed luma samples of the chroma samples to be predicted, i.e., Figure 30 As shown in , there is no downsampling (ie, L0, L1, ..., L5).

[0729]

[0730] Among them, α i Yes and L i is the associated coefficient, and β is the offset.

[0731] In another example, one filter with downsampled values ​​and two filters without downsampled values ​​can be signaled / switched at the CU level. The filter with downsampled values ​​is the same as CCCM. One filter without downsampled values ​​can be composed of (N-1-M) tap (N and M are integers) spatial terms, M nonlinear terms, and bias terms. The (N-1-M) tap spatial terms correspond to the adjacent original reconstructed luma samples of the chroma samples to be predicted, i.e., Figure 30 As shown in , there is no downsampling (ie, L0, L1, ..., L5).

[0732]

[0733] Among them, α i Yes and L i The coefficients associated with β are the coefficients associated with β, and β is the offset. Another filter without downsampling values ​​can be composed of an (N-2)-tap (N is an integer) spatial term, a nonlinear term, and a bias term. The (N-2)-tap spatial term corresponds to the adjacent original reconstructed luma samples of the chroma samples to be predicted, that is, without downsampling.

[0734]

[0735] Among them, α i Yes and L i is the associated coefficient, and β is the offset.

[0736] In yet another example, a syntax element is signaled at a relatively high coding level (e.g., SPS level) to indicate whether CCCM with a downsampling process is enabled. In an example, a syntax element is signaled at a high coding level to indicate whether CCCM without a downsampling process is enabled. In an example, if CCCM with a downsampling process is enabled at a high coding level, a syntax element is further signaled at a lower coding level (e.g., CU level) to indicate whether CCCM with a downsampling process is used for blocks at a lower level. In an example, if CCCM without a downsampling process is enabled at a high coding level, a syntax element is further signaled at a lower coding level (e.g., CU level) to indicate whether CCCM without a downsampling process is used for blocks at a lower level.

[0737] In yet another example, a syntax element is signaled at a relatively high coding level (e.g., SPS level) to indicate whether CCCM with a downsampling process is enabled and whether CCCM without a downsampling process is enabled (e.g., 0: CCCM off, 1: CCCM with a downsampling process enabled, 2: CCCM without a downsampling process enabled, 3: both CCCM with a downsampling process and CCCM without a downsampling process enabled). Then, at a lower coding level (e.g., CU level), another syntax element is signaled to indicate whether CCCM with a downsampling process is enabled and whether CCCM without a downsampling process is enabled in a block at the lower coding level. In an example, if CCCM with a downsampling process is enabled at a high coding level, another syntax element is signaled at a lower coding level (e.g., CU level) to indicate whether CCCM with a downsampling process is used in a block at the lower coding level. In an example, if both CCCM with a downsampling process and CCCM without a downsampling process are enabled in a high coding level, another syntax element is signaled in a lower coding level (e.g., CU level) to indicate whether CCCM with a downsampling process is enabled or whether CCCM without a downsampling process is enabled in blocks at the lower coding level.

[0738] According to one or more embodiments of the present disclosure, the reference samples / training templates / reconstructed neighboring regions used for derivation of multi-model parameters and the reference samples / training templates / reconstructed neighboring regions used for calculation of multi-model thresholds may be different.

[0739] According to one or more embodiments of the present disclosure, the reference sample / training template / reconstructed neighboring region may be predefined or signaled / switched at the SPS / DPS / VPS / SEI / APS / PPS / PH / SH / region / CTU / CU / sub-block / sample level. For example, Figure 9A An example of an L-shaped reconstruction region, a left / top reconstruction region, is shown to derive parameters.

[0740] CCCM with location information

[0741] When computing filter coefficients and predicting chroma sample values, position information is not used in CCCM. In this section, position information is also used in CCCM to compute filter coefficients and predict chroma sample values, including using the horizontal / vertical / diagonal distances between the center luma sample of the filter and the reference luma sample, as well as nonlinear terms for the distances.

[0742] In one example, the position-based parameter is a distance value associated with the vertical and horizontal coordinates (Xc, Yc) of the center luma sample of the filter, and is calculated relative to the coordinates of a reference luma sample (e.g., the coordinates of the luma sample to the upper left of the top left luma sample of the block (Xtl, Ytl)), e.g., Xc-Xtl+Yc-Ytl. In this example, the CCCM includes a term (Xc-Xtl+Yc-Ytl) corresponding to the sum of the horizontal coordinate difference and the vertical coordinate difference between the center luma sample and the luma sample to the upper left of the top left luma sample of the block.

[0743] In another example, the position-based parameters are distance values ​​associated with the vertical and horizontal coordinates (Xc, Yc) of the filter's center luma sample, and are calculated relative to the block's reference coordinates (Xtl, Ytl), e.g., Xc-Xtl+Yc-Ytl, Xc-Xtl, Yc-Ytl. In this example, in addition to the term corresponding to (Xc-Xtl+Yc-Ytl), the CCCM also includes a term corresponding to the horizontal coordinate difference (Xc-Xtl) and a term corresponding to the vertical coordinate difference (Yc-Ytl).

[0744] In yet another example, the position-based parameter is a distance value associated with the vertical and horizontal coordinates (Xc, Yc) of the center luma sample of the filter, and is calculated relative to the reference coordinates (Xtl, Ytl) of the block, e.g., (Xc-Xtl+Yc-Ytl) / N, where N is a predefined number, e.g., 2. In this example, the CCCM includes a term corresponding to the product of (Xc-Xtl+Yc-Ytl) and a factor 1 / N.

[0745] In yet another example, the position-based parameters are distance values ​​associated with the vertical and horizontal coordinates (Xc, Yc) of the center luma sample of the filter, and are calculated relative to the reference coordinates (Xtl, Ytl) of the block, e.g., (Xc-Xtl+Yc-Ytl) / N1, (Xc-Xtl) / N2, (Yc-Ytl) / N3, where N1-N3 are predefined numbers, e.g., 2, 3, and 4. In this example, the CCCM also includes terms corresponding to the product of (Xc-Xtl+Yc-Ytl) and a factor of 1 / N1, the product of (Xc-Xtl) and a factor of 1 / N2, and the product of (Yc-Ytl) and a factor of 1 / N3.

[0746] In yet another example, position-based nonlinear terms are expressed as powers of two of horizontal / vertical / diagonal distances, e.g., (Xc-Xtl+Yc-Ytl)*(Xc-Xtl+Yc-Ytl), (Xc-Xtl)*(Xc-Xtl), (Yc-Ytl)*(Yc-Ytl), where (Xc, Yc) are the vertical and horizontal coordinates of the center luma sample of the filter, and (Xtl, Ytl) are the reference coordinates.

[0747] Figure 31 A workflow of a method 3100 for decoding video data according to one or more aspects of the present disclosure is shown.

[0748] At step 3110 , method 3100 includes obtaining a bitstream.

[0749] At step 3120, method 3100 includes obtaining information related to one or more convolutional cross-component models (CCCMs) from a bitstream, wherein each of the one or more CCCMs includes a filter shape and a set of weighting coefficients corresponding to the filter shape for predicting each of chroma sample values ​​of a video block from the bitstream based on multiple corresponding luma sample values.

[0750] At step 3130 , method 3100 includes decoding the video data based on information associated with the one or more CCCMs.

[0751] In one example, the one or more CCCMs include one or more respective CCCMs for decoding video data at different encoding levels.

[0752] In one example, the information includes a first syntax element in a first coding level for indicating one or more of: whether CCCM with a downsampling process is enabled at the first coding level; whether CCCM without a downsampling process is enabled at the first coding level; whether different filter shapes are enabled at the first coding level; the number of different filter shapes used at the first coding level; or which filter shapes are used at the first coding level.

[0753] In one example, the information also includes a second syntax element in a second coding level lower than the first coding level for indicating one or more of the following items: whether CCCM with a downsampling process is enabled at the second coding level; whether CCCM without a downsampling process is enabled at the second coding level; whether different filter shapes are enabled at the second coding level; the number of different filter shapes used at the second coding level; or which filter shapes are used at the second coding level.

[0754] In one example, method 3100 also includes: in response to the first syntax element indicating that different filter shapes are enabled at the first coding level or the number of different filter shapes used at the first coding level, determining that the second syntax element is used to indicate: the number of different filter shapes used at the second coding level; or which filter shapes of the filter shapes are used at the second coding level.

[0755] In one example, method 3100 further includes: in response to the first syntax element indicating that CCCM without downsampling process is enabled at the first coding level, determining that the second syntax element is used to indicate: whether CCCM without downsampling process is enabled at the second coding level.

[0756] Figure 32 A workflow of a method 3200 for encoding video data according to one or more aspects of the present disclosure is shown.

[0757] At step 3210, method 3200 includes obtaining information related to one or more convolutional cross-component models (CCCMs), wherein each of the one or more CCCMs includes a filter shape and a set of weighting coefficients corresponding to the filter shape for predicting each of the chroma sample values ​​of the video block based on multiple corresponding luma sample values.

[0758] At step 3220 , method 3200 includes encoding video data based on information associated with one or more CCCMs.

[0759] At step 3230 , method 3200 includes obtaining a bitstream including encoded video data and information related to one or more CCCMs.

[0760] In one example, the one or more CCCMs include one or more respective CCCMs for encoding video data at different encoding levels.

[0761] In one example, the information includes a first syntax element in a first coding level for indicating one or more of: whether CCCM with a downsampling process is enabled at the first coding level; whether CCCM without a downsampling process is enabled at the first coding level; whether different filter shapes are enabled at the first coding level; the number of different filter shapes used at the first coding level; or which filter shapes are used at the first coding level.

[0762] In one example, the information also includes a second syntax element in a second coding level lower than the first coding level for indicating one or more of the following items: whether CCCM with a downsampling process is enabled at the second coding level; whether CCCM without a downsampling process is enabled at the second coding level; whether different filter shapes are enabled at the second coding level; the number of different filter shapes used at the second coding level; or which filter shapes are used at the second coding level.

[0763] In one example, method 3200 also includes: when the first syntax element indicates that different filter shapes are enabled at the first coding level or the number of different filter shapes used at the first coding level, setting a second syntax element to indicate: the number of different filter shapes used at the second coding level; or which filter shapes among the filter shapes are used at the second coding level.

[0764] In one example, the method 3200 further includes, when the first syntax element indicates that CCCM without downsampling process is enabled at the first coding level, setting a second syntax element to indicate whether CCCM without downsampling process is enabled at the second coding level.

[0765] Figure 33 A workflow of method 3300 for decoding video data according to one or more aspects of the present disclosure is shown.

[0766] At step 3310 , method 3300 includes obtaining a video block from a bitstream.

[0767] At step 3320, method 3300 includes predicting each of the chroma sample values ​​of the video block using a convolutional cross-component model (CCCM), wherein the CCCM includes a filter shape and a set of weighting coefficients determined based on external luma samples and external chroma samples of an external region of the video block, wherein the set of weighting coefficients includes a weighting coefficient corresponding to a distance value, wherein the distance value represents a distance between a center luma sample of the filter and a reference luma sample.

[0768] At step 3330 , method 3300 includes obtaining a predicted video block using the predicted intra chroma sample values.

[0769] In one example, the reference luma sample is the luma sample to the upper left of the upper left luma sample of the video block.

[0770] In one example, the distance value is determined as one of the following: the sum of the horizontal coordinate difference and the vertical coordinate difference between the center luma sample and the reference luma sample; the horizontal coordinate difference between the center luma sample and the reference luma sample; or the vertical coordinate difference between the center luma sample and the reference luma sample.

[0771] In one example, the distance value is determined as the square of one of the following: the sum of the horizontal coordinate difference and the vertical coordinate difference between the center luma sample and the reference luma sample; the horizontal coordinate difference between the center luma sample and the reference luma sample; or the vertical coordinate difference between the center luma sample and the reference luma sample.

[0772] In one example, the distance value is determined as a product of a factor and one of the following: the sum of the horizontal coordinate difference and the vertical coordinate difference between the center luma sample and the reference luma sample; the horizontal coordinate difference between the center luma sample and the reference luma sample; or the vertical coordinate difference between the center luma sample and the reference luma sample.

[0773] Figure 34 A workflow of a method 3400 for encoding video data according to one or more aspects of the present disclosure is shown.

[0774] At step 3410 , method 3400 includes obtaining a video block.

[0775] At step 3420, method 3400 includes predicting each of the chroma sample values ​​of the video block using a convolutional cross-component model (CCCM), wherein the CCCM includes a filter shape and a set of weighting coefficients determined based on external luma samples and external chroma samples of an outer region of the video block, wherein the set of weighting coefficients includes a weighting coefficient corresponding to a distance value, wherein the distance value represents a distance between a center luma sample of the filter and a reference luma sample.

[0776] At step 3430 , method 3400 includes generating a bitstream including the encoded video block by using the predicted chroma sample values.

[0777] In one example, the reference luma sample is the luma sample to the upper left of the upper left luma sample of the video block.

[0778] In one example, the distance value is determined as one of the following: the sum of the horizontal coordinate difference and the vertical coordinate difference between the center luma sample and the reference luma sample; the horizontal coordinate difference between the center luma sample and the reference luma sample; or the vertical coordinate difference between the center luma sample and the reference luma sample.

[0779] In one example, the distance value is determined as the square of one of the following: the sum of the horizontal coordinate difference and the vertical coordinate difference between the center luma sample and the reference luma sample; the horizontal coordinate difference between the center luma sample and the reference luma sample; or the vertical coordinate difference between the center luma sample and the reference luma sample.

[0780] In one example, the distance value is determined as a product of a factor and one of the following: the sum of the horizontal coordinate difference and the vertical coordinate difference between the center luma sample and the reference luma sample; the horizontal coordinate difference between the center luma sample and the reference luma sample; or the vertical coordinate difference between the center luma sample and the reference luma sample.

[0781] Figure 35 A computing environment 3510 is shown coupled to a user interface 3550. The computing environment 3510 may be part of a data processing server. The computing environment 3510 includes a processor 3520, a memory 3530, and an input / output (I / O) interface 3540.

[0782] The processor 3520 generally controls the overall operation of the computing environment 3510, such as operations associated with display, data acquisition, data communication, and image processing. The processor 3520 may include one or more processors to execute instructions to perform all or some of the steps in the above-described methods. In addition, the processor 3520 may include one or more modules that facilitate interaction between the processor 3520 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip machine, a graphics processing unit (GPU), etc.

[0783] The memory 3530 is configured to store various types of data to support the operation of the computing environment 3510. The memory 3530 may include predetermined software 3532. Examples of such data include instructions for any application or method operating on the computing environment 3510, video data sets, image data, etc. The memory 3530 may be implemented using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0784] The I / O interface 3540 provides an interface between the processor 3520 and peripheral interface modules (such as a keyboard, click wheel, buttons, etc.). The buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 3540 may be coupled to an encoder and a decoder.

[0785] In an embodiment, a non-transitory computer-readable storage medium is further provided, which includes, for example, a plurality of programs in a memory 3530, which can be executed by a processor 3520 in a computing environment 3510, for performing the above-mentioned method and / or storing a bit stream generated by the above-mentioned encoding method or a bit stream decoded by the above-mentioned decoding method. In one example, the plurality of programs can be executed by the processor 3520 in the computing environment 3510 to (for example, from Figure 2 The video encoder 20 in the computing environment 3510 receives a bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames and / or associated one or more syntax elements, etc.), and can also be executed by the processor 3520 in the computing environment 3510 to perform the decoding method described above according to the received bitstream or data stream. In another example, multiple programs can be executed by the processor 3520 in the computing environment 3510 to perform the above-described encoding method to encode video information (e.g., video blocks representing video frames and / or associated one or more syntax elements, etc.) into a bitstream or data stream, and can also be executed by the processor 3520 in the computing environment 3510 to send the bitstream or data stream (e.g., to a video encoder 20 in the computing environment 3510). Figure 3 Alternatively, a non-transitory computer-readable storage medium may have stored therein a bitstream or data stream comprising the video decoder 30 in FIG. 1 . Figure 2 The video encoder 20 in FIG. 1 generates a video for the decoder (eg, Figure 3 The non-transitory computer-readable storage medium may be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0786] In an embodiment, a bitstream generated by the above encoding method or a bitstream to be decoded by the above decoding method is provided. In an embodiment, a bitstream is provided, which includes encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method.

[0787] In an embodiment, a computing device is also provided that includes one or more processors (e.g., processor 3520) and a non-transitory computer-readable storage medium or memory 3530 having stored therein a plurality of programs executable by the one or more processors, wherein the one or more processors are configured to perform the above-described method when executing the plurality of programs.

[0788] In an embodiment, a computer program product having instructions for storing or transmitting a bitstream including encoded video information generated by the encoding method described above or encoded video information to be decoded by the decoding method described above is also provided. In an embodiment, a computer program product is also provided that includes, for example, a plurality of programs executable by a processor 3520 in a computing environment 3510 in a memory 3530 for performing the method described above. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0789] In an embodiment, the computing environment 3510 may be implemented using one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0790] In an embodiment, a method for storing a bitstream is further provided, comprising storing the bitstream on a digital storage medium, wherein the bitstream comprises encoded video information generated by the encoding method or encoded video information to be decoded by the decoding method.

[0791] In an embodiment, a method for transmitting a bit stream generated by the above encoder is further provided. In an embodiment, a method for receiving a bit stream to be decoded by the above decoder is further provided.

[0792] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or to limit the present disclosure. Many modifications, variations, and alternative implementations will be apparent to one of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.

[0793] Unless otherwise specifically stated, the order of the steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but can be changed according to actual conditions. In addition, at least one step of the method according to the present disclosure can be adjusted, combined or deleted according to actual requirements.

[0794] The examples are chosen and described in order to explain the principles of the present disclosure and to enable others skilled in the art to understand the present disclosure for various implementations and to best utilize the basic principles and various implementations with various modifications as are suitable for the particular use contemplated. Therefore, it should be understood that the scope of the present disclosure is not limited to the specific examples of implementations disclosed, and modifications and other implementations are intended to be included within the scope of the present disclosure.

Claims

1. A method for decoding video data, comprising: Get the bitstream; obtaining, from the bitstream, information associated with one or more convolutional cross-component models (CCCMs), wherein each of the one or more CCCMs comprises a filter shape and a set of weighting coefficients corresponding to the filter shape for predicting each of chroma sample values ​​of a video block from the bitstream based on a plurality of corresponding luma sample values; as well as The video data is decoded based on the information associated with the one or more CCCMs.

2. The method according to claim 1, wherein The one or more CCCMs include one or more corresponding CCCMs for decoding the video data at different encoding levels.

3. The method according to claim 2, wherein: The information comprises a first syntax element at a first coding level for indicating one or more of: whether CCCM with downsampling process is enabled at the first coding level; whether to enable CCCM without downsampling process at the first coding level; whether to enable different filter shapes at the first encoding level; the number of different filter shapes used at the first encoding level; or Which of the filter shapes are used at the first encoding level.

4. The method according to claim 3, wherein: The information further includes a second syntax element at a second coding level lower than the first coding level for indicating one or more of the following: whether to enable CCCM with downsampling process at the second coding level; whether to enable CCCM without downsampling process at the second coding level; whether to enable different filter shapes at the second encoding level; the number of different filter shapes used at the second encoding level; or Which of the filter shapes are used at the second encoding level.

5. The method according to claim 4, further comprising: In response to the first syntax element indicating that different filter shapes are enabled at the first encoding level or the number of different filter shapes used at the first encoding level, determining that the second syntax element is used to indicate: the number of different filter shapes used at the second encoding level; or Which of the filter shapes are used at the second encoding level.

6. The method according to claim 4, further comprising: In response to the first syntax element indicating that CCCM without a downsampling process is enabled at the first coding level, determining that the second syntax element is used to indicate: Whether to enable CCCM without downsampling process at the second encoding level.

7. A method for encoding video data, comprising: Obtaining information associated with one or more convolutional cross-component models (CCCMs), wherein each CCCM of the one or more CCCMs comprises a filter shape and a set of weighting coefficients corresponding to the filter shape for predicting each of chroma sample values ​​of a video block based on a plurality of corresponding luma sample values; encoding the video data based on the information associated with the one or more CCCMs; and A bitstream comprising encoded video data and the information associated with the one or more CCCMs is obtained.

8. The method according to claim 7, wherein: The one or more CCCMs include one or more corresponding CCCMs for encoding the video data at different encoding levels.

9. The method according to claim 8, wherein The information comprises a first syntax element at a first coding level for indicating one or more of: whether CCCM with downsampling process is enabled at the first coding level; whether to enable CCCM without downsampling process at the first coding level; whether to enable different filter shapes at the first encoding level; the number of different filter shapes used at the first encoding level; or Which of the filter shapes are used at the first encoding level.

10. The method according to claim 9, wherein: The information further includes a second syntax element at a second coding level lower than the first coding level for indicating one or more of the following: whether to enable CCCM with downsampling process at the second coding level; whether to enable CCCM without downsampling process at the second coding level; whether to enable different filter shapes at the second encoding level; the number of different filter shapes used at the second encoding level; or Which of the filter shapes are used at the second encoding level.

11. The method according to claim 10, further comprising: When the first syntax element indicates that different filter shapes are enabled at the first encoding level or the number of different filter shapes used at the first encoding level, the second syntax element is set to indicate: the number of different filter shapes used at the second encoding level; or Which of the filter shapes are used at the second encoding level.

12. The method according to claim 10, further comprising: When the first syntax element indicates that CCCM without downsampling process is enabled at the first coding level, the second syntax element is set to indicate: Whether to enable CCCM without downsampling process at the second encoding level.

13. A method for decoding video data, comprising: Get video chunks from bitstream; predicting each of the chroma sample values ​​of the video block using a convolutional cross-component model (CCCM), wherein the CCCM comprises a filter shape and a set of weighting coefficients determined based on outer luma samples and outer chroma samples of an outer region of the video block, wherein the set of weighting coefficients comprises a weighting coefficient corresponding to a distance value representing a distance between a center luma sample of the filter and a reference luma sample; and The predicted intra chroma sample values ​​are used to obtain a predicted video block.

14. The method according to claim 13, wherein The reference luma sample is a luma sample above and to the left of the above-left luma sample of the video block.

15. The method according to claim 13, wherein The distance value is determined as one of the following: The sum of the horizontal coordinate difference and the vertical coordinate difference between the central luminance sample point and the reference luminance sample point; the horizontal coordinate difference between the central luminance sample point and the reference luminance sample point; or The vertical coordinate difference between the central luminance sample point and the reference luminance sample point.

16. The method according to claim 13, wherein: The distance value is determined as the square of one of the following terms: The sum of the horizontal coordinate difference and the vertical coordinate difference between the central luminance sample point and the reference luminance sample point; the horizontal coordinate difference between the central luminance sample point and the reference luminance sample point; or The vertical coordinate difference between the central luminance sample point and the reference luminance sample point.

17. The method according to claim 13, wherein: The distance value is determined as the product of a factor and one of the following: The sum of the horizontal coordinate difference and the vertical coordinate difference between the central luminance sample point and the reference luminance sample point; the horizontal coordinate difference between the central luminance sample point and the reference luminance sample point; or The vertical coordinate difference between the central luminance sample point and the reference luminance sample point.

18. A method for encoding video data, comprising: Get the video block; predicting each of the chroma sample values ​​of the video block using a convolutional cross-component model (CCCM), wherein the CCCM comprises a filter shape and a set of weighting coefficients determined based on outer luma samples and outer chroma samples of an outer region of the video block, wherein the set of weighting coefficients comprises a weighting coefficient corresponding to a distance value representing a distance between a center luma sample of the filter and a reference luma sample; and A bitstream including the encoded video block is generated by using the predicted chroma sample values.

19. The method according to claim 18, wherein The reference luma sample is a luma sample above and to the left of the above-left luma sample of the video block.

20. The method according to claim 18, wherein The distance value is determined as one of the following: The sum of the horizontal coordinate difference and the vertical coordinate difference between the central luminance sample point and the reference luminance sample point; the horizontal coordinate difference between the central luminance sample point and the reference luminance sample point; or The vertical coordinate difference between the central luminance sample point and the reference luminance sample point.

21. The method according to claim 18, wherein The distance value is determined as the square of one of the following terms: The sum of the horizontal coordinate difference and the vertical coordinate difference between the central luminance sample point and the reference luminance sample point; the horizontal coordinate difference between the central luminance sample point and the reference luminance sample point; or The vertical coordinate difference between the central luminance sample point and the reference luminance sample point.

22. The method according to claim 18, wherein The distance value is determined as the product of a factor and one of the following: The sum of the horizontal coordinate difference and the vertical coordinate difference between the central luminance sample point and the reference luminance sample point; the horizontal coordinate difference between the central luminance sample point and the reference luminance sample point; or The vertical coordinate difference between the central luminance sample point and the reference luminance sample point.

23. A computer system comprising: one or more processors; as well as One or more storage devices storing computer-executable instructions that, when executed, cause the one or more processors to perform the operations of the method according to any one of claims 1-22.

24. A computer program product storing computer-executable instructions that, when executed, cause one or more processors to perform the operations of the method according to any one of claims 1-22.

25. A computer-readable storage medium storing instructions that, when executed by a computing device having one or more processors, cause the one or more processors to perform the decoding method according to any one of claims 1-6, 13-17, and store a bit stream to be decoded by the decoding method according to any one of claims 1-6, 13-17.

26. A computer-readable storage medium storing instructions that, when executed by a computing device having one or more processors, cause the one or more processors to perform the encoding method according to any one of claims 7-12, 18-22, and store a bitstream generated by the encoding method according to any one of claims 7-12, 18-22.

27. A computer-readable medium storing a bitstream, wherein: The bitstream is to be decoded by performing the operations of the method according to any one of claims 1-6, 13-17.

28. A computer-readable medium storing a bitstream, wherein: The bit stream is obtained by performing the operation of the method according to any one of claims 7-12, 18-22.

Images