Laplacian enhancement and / or laplacian edge as an additional source of information in alf

Abstract

An apparatus configured to: obtain one or more sample value differences associated with sample reconstruction; perform filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample comprises at least one of: at least one luma sample, or at least one chroma sample; and provide the at least one filtered sample.

Description

LAPLACIAN ENHANCEMENT AND / OR LAPLACIAN EDGE AS AN ADDITIONAL SOURCE OF INFORMATION IN ALF TECHNICAL FIELD

[0001] The example and non-limiting embodiments relate generally to filtering and, moreparticularly, to input to adaptive loop filters. BACKGROUND

[0002] It is known, in image and video encoding, to provide sample adaptive offset filteredinput to adaptive loop filters. SUMMARY

[0003] The following summary is merely intended to be illustrative. The summary is notintended to limit the scope of the claims.

[0004] In accordance with one aspect, an apparatus comprising: at least one processor; andat least one memory storing instructions that, when executed with the at least one processor, cause the apparatus at least to: obtain one or more sample value differences associated with sample reconstruction; perform filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample comprises at least one of: atleast one luma sample, or at least one chroma sample; and provide the at least one filtered sample.

[0005] In accordance with one aspect, a method comprising: obtaining, with a userequipment, one or more sample value differences associated with sample reconstruction; performing filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at leastone filtered sample, wherein the at least one sample comprises at least one of: at least one luma sample, or at least one chroma sample; and providing the at least one filtered sample.

[0006] In accordance with one aspect, an apparatus comprising means for: obtaining one ormore sample value differences associated with sample reconstruction; performing filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample comprises at least one of: at least one luma sample, or at least one chroma sample; and providing the at least one filtered sample.

[0007] In accordance with one aspect, a computer-readable medium comprising programinstructions stored thereon for performing at least the following: causing obtaining, with a user equipment, of one or more sample value differences associated with sample reconstruction; filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample comprises at least one of: at least one luma sample, or at least one chroma sample; and causing providing of the at least one filtered sample.

[0008] According to some aspects, there is provided the subject matter of the independentclaims. Some further aspects are defined in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The foregoing aspects and other features are explained in the following description,taken in connection with the accompanying drawings, wherein:

[0010] FIG.1 is a block diagram of one possible and non-limiting example system in whichthe example embodiments may be practiced;

[0011] FIG. 2 is a block diagram of one possible and non-limiting exemplary system inwhich the example embodiments may be practiced;

[0012] FIG. 3 is a diagram illustrating features as described herein;

[0013] FIG. 4 is a diagram illustrating features as described herein;

[0014] FIG. 5 is a diagram illustrating features as described herein;

[0015] FIGs. 6A-B are diagrams illustrating features as described herein;

[0016] FIG. 7 is a diagram illustrating features as described herein;

[0017] FIG. 8 is a diagram illustrating features as described herein;

[0018] FIG. 9 is a diagram illustrating features as described herein;

[0019] FIG. 10 is a diagram illustrating features as described herein;

[0020] FIG. 11 is a diagram illustrating features as described herein;

[0021] FIG. 12 is a diagram illustrating features as described herein;

[0022] FIGs. 13A-D are diagrams illustrating features as described herein;

[0023] FIGs. 14A-B are diagrams illustrating features as described herein;

[0024] FIG. 15 is a diagram illustrating features as described herein;

[0025] FIG. 16 is a diagram illustrating features as described herein;

[0026] FIG. 17 is a diagram illustrating features as described herein; and

[0027] FIG. 18 is a flowchart illustrating steps as described herein.DETAILED DESCRIPTION OF EMBODIMENTS

[0028] The following abbreviations that may be found in the specification and / or thedrawing figures are defined as follows:3GPP third generation partnership project4G fourth generation5G fifth generation5GC 5G core networkAMVP advanced motion vector predictionAMVR adaptive motion vector resolutionALF adaptive loop filterAPS adaptive parameter setAR augmented realityBCW bi-prediction with CU level weightBDOF bi-directional optical flowCCC-ALF cross-component adaptive loop filterCCCM convolutional cross-component modelCCLM cross component linear model intra predictionCDMA code division multiple accessCIIP combined inter and intra predictionCPU central processing unitcRAN cloud radio access networkCTU coding tree unitCU coding unitDCT discrete cosine transformDM direct modeECM enhanced compression modeleNB (or eNodeB) evolved Node B (e.g., an LTE base station)EN-DC E-UTRA-NR dual connectivityen-gNB or En-gNB node providing NR user plane and control plane protocolterminations towards the UE, and acting as secondary node in EN- DCE-UTRA evolved universal terrestrial radio access, i.e., the LTE radio accesstechnologyFDMA frequency division multiple accessFIFO first in first outgNB (or gNodeB) base station for 5G / NR, i.e., a node providing NR user plane andcontrol plane protocol terminations towards the UE, and connected via the NG interface to the 5GCGPU graphical processing unitGSM global systems for mobile communicationsHEVC high efficiency video codingHMD head-mounted displayIEEE Institute of Electrical and Electronics EngineersIMD integrated messaging deviceIMS instant messaging serviceIoT Internet of ThingsISP intra sub-partitionLIC local illumination compensationLM linear modelLMS least-mean-squareLTE long term evolutionMCP motion-compensated predictionME motion estimationMIP matrix weighted intra predictionMMLM multi-model liner modelMMS multimedia messaging serviceMMVD merged mode width motion vector differenceMPEG-I Moving Picture Experts Group immersive codec familyMR mixed realityMRL multiple reference lineMSE mean square errorMVD motion vector deltang or NG new generationng-eNB or NG-eNB new generation eNBNR new radioN / W or NW networkOOB out-of-boundaryO-RAN open radio access networkPC personal computerPDA personal digital assistantPDPC position dependent intra prediction combinationPU prediction unitQT quadtreeRDO rate distortion optimizationSAO sample adaptive offsetSMS short messaging serviceSPS sequence parameter setTB transport blockTCP-IP transmission control protocol-internet protocolTDMA time division multiple accessTU transform unitUE user equipment (e.g., a wireless, typically mobile device)UMTS universal mobile telecommunications systemUSB universal serial busVNR virtualized network functionVR virtual realityVTM versatile video coding test modelVVC versatile video codingWLAN wireless local area networkWP weighted prediction

[0029] The following describes suitable apparatus and possible mechanisms for practicingexample embodiments of the present disclosure. Accordingly, reference is first made to FIG. 1, which shows an example block diagram of an apparatus 50. The apparatus may be configured to perform various functions such as, for example, gathering information by one or more sensors, encoding and / or decoding information, receiving and / or transmitting information, analyzing information gathered or received by the apparatus, or the like. A device configured to encode a video scene may (optionally) comprise one or more microphones for capturing the scene and / or one or more sensors, such as cameras, for capturing information about the physical environment in which the scene is captured. Alternatively, a device configured to encode a video scene may be configured to receive information about an environment in which a scene is captured and / or a simulated environment. A device configured to decode and / or render the video scene may be configured to receive a Moving Picture Experts Group immersive codec family (MPEG-I) bitstream comprising the encoded video scene. A device configured to decode and / or render the video scene may comprise one or more speakers / audio transducers and / or displays, and / or may be configured to transmit a decoded scene or signals to a device comprising one or more speakers / audio transducers and / or displays. A device configured to decode and / or render the video scene may comprise a user equipment, a head / mounted display, or another device capable of rendering to a user an AR, VR and / or MR experience.

[0030] The electronic device 50 may for example be a mobile terminal or user equipment ofa wireless communication system. Alternatively, the electronic device may be a computer or part of a computer that is not mobile. It should be appreciated that example embodiments of the present disclosure may be implemented within any electronic device or apparatus which may process data. The electronic device 50 may comprise a device that can access a network and / or cloud through a wired or wireless connection. The electronic device 50 may comprise one or more processors 56, one or more memories 58, and one or more transceivers 52 interconnected through one or more buses. The one or more processors 56 may comprise a central processing unit (CPU) and / or a graphical processing unit (GPU). Each of the one or more transceivers 52 includes a receiver and a transmitter. The one or more buses may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. A “circuit” may include dedicated hardware or hardware in association with software executable thereon. The one or more transceivers may be connected to one or more antennas 44. The one or more memories 58 may include computer program code. The one or more memories 58 and the computer program code may be configured to, with the one or more processors 56, cause the electronic device 50 to perform one or more of the operations as described herein.

[0031] The electronic device 50 may connect to a node of a network. The network node maycomprise one or more processors, one or more memories, and one or more transceivers interconnected through one or more buses. Each of the one or more transceivers includes a receiver and a transmitter. The one or more buses may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. The one or more transceivers may be connected to one or more antennas. The one or more memories may include computer program code. The one or more memories and the computer program code may be configured to, with the one or more processors, cause the network node to perform one or more of the operations as described herein.

[0032] The electronic device 50 may comprise a microphone 36 or any suitable audio inputwhich may be a digital or analogue signal input. The electronic device 50 may further comprise an audio output device 38 which in example embodiments of the present disclosure may be any one of: an earpiece, speaker, or an analogue audio or digital audio output connection. The electronic device 50 may also comprise a battery (or in other example embodiments of the present disclosure the device may be powered by any suitable mobile energy device such as solar cell, fuel cell, or clockwork generator). The electronic device 50 may further comprise a camera 42 or other sensor capable of recording or capturing images and / or video. Additionally or alternatively, the electronic device 50 may further comprise a depth sensor. The electronic device 50 may further comprise a display 32. The electronic device 50 may further comprise an infrared port for short range line of sight communication to other devices. In other example embodiments of the present disclosure the apparatus 50 may further comprise any suitable short-range communication solutionsuch as for example a BLUETOOTH™ wireless connection or a USB / firewire wired connection.

[0033] It should be understood that an electronic device 50 configured to perform exampleembodiments of the present disclosure may have fewer and / or additional components, which may correspond to what processes the electronic device 50 is configured to perform. For example, an apparatus configured to encode a video might not comprise a speaker or audio transducer and may comprise a microphone, while an apparatus configured to render the decoded video might not comprise a microphone and may comprise a speaker or audio transducer.

[0034] Referring now to FIG. 1, the electronic device 50 may comprise a controller 56,processor or processor circuitry for controlling the apparatus 50. The controller 56 may be connected to memory 58 which in example embodiments of the present disclosure may store both data in the form of image and audio data and / or may also store instructions for implementation on the controller 56. The controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and / or decoding of audio and / or video data or assisting in coding and / or decoding carried out by the controller.

[0035] The electronic device 50 may further comprise a card reader 48 and a smart card 46,for example a UICC and UICC reader, for providing user information and being suitable for providing authentication information for authentication and authorization of the user / electronic device 50 at a network. The electronic device 50 may further comprise an input device 34, such as a keypad, one or more input buttons, or a touch screen input device, for providing information to the controller 56.

[0036] The electronic device 50 may comprise radio interface circuitry 52 connected to thecontroller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system, or a wireless local area network. The apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and / or for receiving radio frequency signals from other apparatus(es).

[0037] The electronic device 50 may comprise a microphone 38, camera 42, and / or othersensors capable of recording or detecting audio signals, image / video signals, and / or other information about the local / virtual environment, which are then passed to the codec 54 or the controller 56 for processing. The electronic device 50 may receive the audio / image / video signals and / or information about the local / virtual environment for processing from another device prior to transmission and / or storage. The electronic device 50 may also receive either wirelessly or by a wired connection the audio / image / video signals and / or information about the local / virtual environment for encoding / decoding. The structural elements of electronic device 50 described above represent examples of means for performing a corresponding function.

[0038] The memory 58 may be of any type suitable to the local technical environment andmay be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The memory 58 may be a non-transitory memory. The memory 58 may be means for performing storage functions. The controller 56 maybe or comprise one or more processors, which may be of any type suitable to the local technical environment, and may include one or more of general-purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on a multi- core processor architecture, as non-limiting examples. The controller 56 may be means for performing functions.

[0039] The electronic device 50 may be configured to perform capture of a volumetric sceneaccording to example embodiments of the present disclosure. For example, the electronic device 50 may comprise a camera 42 or other sensor capable of recording or capturing images and / or video. The electronic device 50 may also comprise one or more transceivers 52 to enable transmission of captured content for processing at another device. Such an electronic device 50 may or may not include all the modules illustrated in FIG.1.

[0040] The electronic device 50 may be configured to perform processing of volumetricvideo content according to example embodiments of the present disclosure. For example, the electronic device 50 may comprise a controller 56 for processing images to produce volumetric video content, a controller 56 for processing volumetric video content to project 3D information into 2D information, patches, and auxiliary information, and / or a codec 54 for encoding 2D information, patches, and auxiliary information into a bitstream for transmission to another device with radio interface 52. Such an electronic device 50 may or may not include all the modules illustrated in FIG. 1.

[0041] The electronic device 50 may be configured to perform encoding or decoding of 2Dinformation representative of volumetric video content according to example embodiments of the present disclosure. For example, the electronic device 50 may comprise a codec 54 for encoding or decoding 2D information representative of volumetric video content. Such an electronic device 50 may or may not include all the modules illustrated in FIG.1.

[0042] The electronic device 50 may be configured to perform rendering of decoded 3Dvolumetric video according to example embodiments of the present disclosure. For example, theelectronic device 50 may comprise a controller for projecting 2D information to reconstruct 3D volumetric video, and / or a display 32 for rendering decoded 3D volumetric video. Such an electronic device 50 may or may not include all the modules illustrated in FIG.1.

[0043] With respect to FIG. 2, an example of a system within which example embodimentsof the present disclosure can be utilized is shown. The system 10 comprises multiple communication devices which can communicate through one or more networks. The system 10 may comprise any combination of wired or wireless networks including, but not limited to awireless cellular telephone network (such as a GSM, UMTS, E-UTRA, LTE, CDMA, 4G, 5Gnetwork etc.), a wireless local area network (WLAN) such as defined by any of the IEEE 802.xstandards, a BLUETOOTH™ personal area network, an Ethernet local area network, a token ringlocal area network, a wide area network, and / or the Internet. A wireless network may implement network virtualization, which is the process of combining hardware and software network resources and network functionality into a single, software-based administrative entity, a virtual network. Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is categorized as either external, combining many networks, or parts of networks, into a virtual unit, or internal, providing network-like functionality to software containers on a single system. For example, a network may be deployed in a tele cloud, with virtualized network functions (VNF) running on, for example, data center servers. For example, network core functions and / or radio access network(s) (e.g. CloudRAN, O-RAN, edge cloud) may be virtualized. Note that the virtualized entities that result from the network virtualization are still implemented, at some level, using hardware such as processors and memories, and also such virtualized entities create technical effects.

[0044] It may also be noted that operations of example embodiments of the present disclosuremay be carried out by a plurality of cooperating devices (e.g. cRAN).

[0045] The system 10 may include both wired and wireless communication devices and / orelectronic devices suitable for implementing example embodiments of the present disclosure.

[0046] For example, the system shown in FIG. 2 shows a mobile telephone network 11 anda representation of the internet 28. Connectivity to the internet 28 may include, but is not limitedto, long range wireless connections, short range wireless connections, and various wiredconnections including, but not limited to, telephone lines, cable lines, power lines, and similar communication pathways.

[0047] The example communication devices shown in the system 10 may include, but arenot limited to, an apparatus 15, a combination of a personal digital assistant (PDA) and a mobile telephone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22, and a head-mounted display (HMD) 17. The electronic device 50 may comprise any of those example communication devices. In an example embodiment of the present disclosure, more than one of these devices, or a plurality of one or more of these devices, may perform the disclosed process(es). These devices may connect to the internet 28 through a wireless connection 2.

[0048] The example embodiments of the present disclosure may also be implemented in aset-top box; i.e. a digital TV receiver, which may / may not have a display or wireless capabilities, in tablets or (laptop) personal computers (PC), which have hardware and / or software to process neural network data, in various operating systems, and in chipsets, processors, DSPs and / or embedded systems offering hardware / software based coding. The example embodiments of the present disclosure may also be implemented in cellular telephones such as smart phones, tablets, personal digital assistants (PDAs) having wireless communication capabilities, portable computers having wireless communication capabilities, image capture devices such as digital cameras having wireless communication capabilities, gaming devices having wireless communication capabilities, music storage and playback appliances having wireless communication capabilities, Internet appliances permitting wireless Internet access and browsing, tablets with wireless communication capabilities, as well as portable units or terminals that incorporate combinations of such functions.

[0049] Some or further apparatus may send and receive calls and messages and communicatewith service providers through a wireless connection 25 to a base station 24, which may be, forexample, an eNB, gNB, access point, access node, other node, etc. The base station 24 may be connected to a network server 26 that allows communication between the mobile telephone network 11 and the internet 28. The system may include additional communication devices and communication devices of various types.

[0050] The communication devices may communicate using various transmissiontechnologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), BLUETOOTH™, IEEE 802.11, 3GPP Narrowband IoT and any similar wireless communication technology. A communications device involved in implementing various example embodiments of the present disclosure may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.

[0051] In telecommunications and data networks, a channel may refer either to a physicalchannel or to a logical channel. A physical channel may refer to a physical transmission medium such as a wire, whereas a logical channel may refer to a logical connection over a multiplexed medium, capable of conveying several logical channels. A channel may be used for conveying an information signal, for example a bitstream, which may be a MPEG-I bitstream, from one or several senders (or transmitters) to one or several receivers.

[0052] Having thus introduced one suitable but non-limiting technical context for the practiceof the example embodiments of the present disclosure, example embodiments will now be described with greater specificity.

[0053] Features as described herein may generally relate to encoding and / or decoding ofvideo and / or image information. A video codec consists of an encoder that transforms an input video into a compressed representation suited for storage / transmission, and a decoder that candecompress the compressed video representation back into a viewable form. Typically, the encoder discards some information in the original video sequence to be able to represent the video in a more compact form (that is, at a lower bitrate).

[0054] Typical hybrid video codecs, such as H.264 / AVC, H.265 / HEVC and H.266 / VVC,encode the video information in two phases. Firstly, pixel values in a certain picture area (or “block”) (310) may be predicted (320) for example by motion compensation means (e.g. finding and indicating an area in one of the previously coded pictures that corresponds closely to the block being coded) or by spatial means (e.g. using the pixel values around the block to be coded in a specified manner). Secondly the prediction error, i.e. the difference between the predicted block of pixels and the original block of pixels, may be coded (330). This may typically be done by transforming the difference in pixel values using a specified transform (e.g. discrete cosine transform (DCT) or a variant of it) (340), quantizing the resulting transform coefficients (350), and entropy coding the quantized coefficients (360). By varying the fidelity of the quantization process, the encoder may control the balance between the accuracy of the pixel representation (picture quality) and the size of the resulting coded video representation (file size or transmission bitrate). An example of the encoding process is illustrated in FIG.3.

[0055] The decoder may reconstruct the output video by applying prediction means (410)similar to the encoder to form a predicted representation of the pixel blocks (e.g. using the motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (e.g. inverse operation of the prediction error coding recovering the quantized prediction error signal in spatial pixel domain) (420). After applying prediction and prediction error decoding means, the decoder may sum up the prediction and prediction error signals (pixel values) to form the output video frame (440). The decoder (and encoder) may also apply additional filtering means (430) to improve the quality of the output video before passing it for display and / or storing it as a prediction reference for the forthcoming frames in the video sequence. An example of the decoding process is illustrated in FIG.4.

[0056] Versatile Video Coding (VVC) is an international video coding standard. In VVC, avideo sequence comprises a plurality of pictures. VVC is a block-based video coding standard, where an input picture is divided into coding tree units (CTUs), and each CTU may be further split into coding units (CUs). A CU (or block) is coded in either inter-coding mode or intra-coding mode. If the block is coded in inter-coding mode, the encoder searches for a temporal prediction block in reference picture(s) and signals the decoder an indication of how to find the same prediction block in reference picture(s) at the decoder end. If the block is coded in intra-codingmode, the encoder constructs a spatial prediction block from the current picture and signals thedecoder an indication of how to form the same spatial prediction block from the current picture at the decoder end.

[0057] Encoding and / or decoding may involve the use of pixel prediction, such as predictivecoding. Examples of predictive coding may include sample prediction and syntax prediction.

[0058] In sample prediction, pixel or sample values in a certain picture area or "block" arepredicted. These pixel or sample values can be predicted, for example, using one or more of motion compensation or intra prediction mechanisms.

[0059] In the present disclosure, the terms “pixel” and “sample” may be usedinterchangeably.

[0060] Motion compensation mechanisms (which may also be referred to as inter prediction,temporal prediction, or motion-compensated temporal prediction or motion-compensatedprediction (MCP)) involve finding and indicating an area in one of the previously encoded videoframes that corresponds closely to the block being coded. Inter prediction may reduce temporal redundancy.

[0061] Intra prediction, where pixel or sample values can be predicted by spatialmechanisms, involve finding and indicating a spatial region relationship. Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra predictioncan be performed in the spatial or transform domain (i.e. either sample values or transform coefficients can be predicted). Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0062] In syntax prediction, which may also be referred to as parameter prediction, syntaxelements and / or syntax element values and / or variables derived from syntax elements are predicted from syntax elements (de)coded earlier and / or variables derived earlier. Non-limiting examples of syntax prediction are provided below.

[0063] In motion vector prediction, motion vectors (e.g. for inter and / or inter-viewprediction) may be coded differentially with respect to a block-specific predicted motion vector. In many video codecs, the predicted motion vectors are created in a predefined way, for example by calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions, sometimes referred to as advanced motion vector prediction (AMVP), is to generate a list of candidate predictions from adjacent blocks and / or co- located blocks in temporal reference pictures and signaling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of a previously coded / decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and / or co-located blocks in a temporal reference picture. Differential coding of motion vectors is typically disabled across slice boundaries. The block partitioning, e.g. from CTU to CUs and down to PUs, may be predicted.

[0064] In filter parameter prediction, the filtering parameters (e.g. for sample adaptive offset)may be predicted. Prediction approaches using image information from a previously coded imagemay also be called inter prediction methods, or temporal prediction and motion compensation.Prediction approaches using image information within the same image may also be called intraprediction methods.

[0065] In many video codecs, including H.264 / AVC and HEVC, motion information isindicated by motion vectors associated with each motion compensated image block. Each of thesemotion vectors represents the displacement of the image block in the picture to be coded (in the encoder) or decoded (at the decoder) and the prediction source block in one of the previously codedor decoded images (or pictures). In H.264 / AVC and HEVC, as in many other video compressionstandards, a picture is divided into a mesh of rectangles, for each of which a similar block in one of the reference pictures is indicated for inter prediction. The location of the prediction block is coded as a motion vector that indicates the position of the prediction block relative to the block being coded.

[0066] In the developing Versatile Video Codec (VVC), there are at least the following newcoding tools:

[0067] - Intra prediction: 67 intra mode with wide angles mode extension; block size andmode dependent 4 tap interpolation filter; position dependent intra prediction combination (PDPC); cross component linear model intra prediction (CCLM); multi-reference line intraprediction; intra sub-partitions; weighted intra prediction with matrix multiplication; etc.

[0068] - Inter-picture prediction: block motion copy with spatial, temporal, history-based,and pairwise average merging candidates; affine motion inter prediction; sub-block based temporal motion vector prediction; adaptive motion vector resolution; 8x8 block-based motion compression for temporal motion prediction; high precision (1 / 16 pel) motion vector storage and motion compensation with 8-tap; interpolation filter for luma component and 4-tap interpolation filter for chroma component; triangular partitions; combined intra and inter prediction; merge with MVD (MMVD); symmetrical MVD coding; bi-directional optical flow; decoder side motion vectorrefinement; bi-prediction with CU-level weight; etc.

[0069] - Transform, quantization and coefficients coding: multiple primary transformselection with DCT2, DST7 and DCT8; secondary transform for low frequency zone; sub-block transform for inter predicted residual; dependent quantization with max QP increased from 51 to63; transform coefficient coding with sign data hiding; transform skip residual coding; etc.

[0070] - Entropy Coding: arithmetic coding engine with adaptive double windowsprobability update; etc.

[0071] - In loop filter: in-loop reshaping; deblocking filter with strong longer filter; sampleadaptive offset; adaptive loop filter; etc.

[0072] - Screen content coding: current picture referencing with reference region restriction;etc.

[0073] - 360-degree video coding: horizontal wrap-around motion compensation; etc.

[0074] - High-level syntax and parallel processing: reference picture management with directreference picture list signaling; tile groups with rectangular shape tile groups; etc.

[0075] Features as described herein may generally relate to partitioning in VVC. In VVC,each picture is divided into coding tree units (CTUs) similar to HEVC. A picture may also bedivided into slices, tiles, bricks, and / or sub-pictures. A CTU may be split into smaller coding units(CU) using a quaternary tree structure. Each CU may be divided using quad-tree and nested multi-type tree including ternary and binary split. There are specific rules to infer partitioning in pictureboundaries. The redundant split patterns are disallowed in nested multi-type partitioning.

[0076] Features as described herein may generally relate to cross-component linear modelprediction (CCLM). To reduce the cross-component redundancy, a cross-component linear model (CCLM) prediction mode is used in the VVC, for which the chroma samples are predicted based on the reconstructed luma samples of the same CU by using a linear model as follows: ^^^^^(^, ^) = ^ ∙ ^^^^^ (^, ^) + ^where ^^^^^(^, ^) represents the predicted chroma samples in a CU and ^^^^^ (^, ^) represents thedown-sampled reconstructed luma samples of the same CU.

[0077] The CCLM parameters (α and β) are derived with at most four neighboring chromasamples and their corresponding down-sampled luma samples. Suppose the current chroma block dimensions are W×H, then W’ and H’ are set as: W’ = W, H’ = H when LM mode is applied; W’ = W + H when LM-A mode is applied;H’ = H + W when LM-L mode is applied;

[0078] The above neighboring positions are denoted as S[ 0, −1 ]…S[ W’ − 1, −1 ] and theleft neighboring positions are denoted as S[ −1, 0 ]…S[ −1, H’ − 1 ]. Then the four samples are selected as: S[W’ / 4, −1 ], S[ 3 * W’ / 4, −1 ], S[ −1, H’ / 4 ], S[ −1, 3 * H’ / 4 ] when LM mode is applied and both above and left neighboring samples are available S[ W’ / 8, −1 ], S[ 3 * W’ / 8, −1 ], S[ 5 * W’ / 8, −1 ], S[ 7 * W’ / 8, −1 ] when LM-A mode is applied or only the above neighboring samples are available S[ −1, H’ / 8 ], S[ −1, 3 * H’ / 8 ], S[ −1, 5 * H’ / 8 ], S[ −1, 7 * H’ / 8 ] when LM-L mode is applied or only the left neighboring samples are available

[0079] The four neighboring luma samples at the selected positions are down-sampled andcompared four times to find two smaller values: x0A and x1A, and two larger values: x0B andx1B. Their corresponding chroma sample values are denoted as y0A, y1A, y0B and y1B. ThenXa, Xb, Ya and Yb are derived as:^^ = (^^^ + ^^^ + 1) ≫ 1

[0080] Finally, the linear model parameters ^ and ^ are obtained according to the followingequations: ^− ^^ = ^ ^^^ − ^^^ = ^^ − ^ ∙ ^^

[0081] Referring now to FIG. 5, illustrated is an example of the location of the left and abovesamples and the sample of the current block involved in the CCLM mode. At 510, the gray circles indicate the left and above samples, with respect to a NxN block 520, in the chroma domain. At 530, the gray circles indicate the left and above samples, with respect to a 2Nx2N block 540, in the luma domain.

[0082] The division operation to calculate parameter α is implemented with a look-up table.To reduce the memory required for storing the table, the diff value (i.e. difference between maximum and minimum values) and the parameter α are expressed by an exponential notation. For example, diff is approximated with a 4-bit significant part and an exponent. Consequently, the table for 1 / diff is reduced into 16 elements for 16 values of the significand as follows: DivTable [ ] = { 0, 7, 6, 5, 5, 4, 4, 3, 3, 2, 2, 1, 1, 1, 1, 0 }

[0083] This would have a benefit of both reducing the complexity of the calculation as wellas the memory size required for storing the needed tables.

[0084] The above template and left template can be used to calculate the linear modelcoefficients together, and they can also can be used alternatively in the other 2 LM modes, called LM_A, and LM_L modes.

[0085] In LM_A mode, only the above template is used to calculate the linear modelcoefficients. To get more samples, the above template is extended to (W+H). In LM_L mode,only left template is used to calculate the linear model coefficients. To get more samples, the lefttemplate is extended to (H+W). For a non-square block, the above template is extended to W+W,and the left template is extended to H+H.

[0086] To match the chroma sample locations for 4:2:0 video sequences, two types ofdownsampling filter are applied to luma samples to achieve 2 to 1 downsampling ratio in bothhorizontal and vertical directions. The selection of downsampling filter may be specified by asequence parameter set (SPS) level flag. The two downsampling filters are as follows, whichcorrespond to “type-0” and “type-2” content, respectively. ^^^^^ (^, ^) = [^^^^(2^ − 1,2^ − 1) + 2 ∙ ^^^^(2^ − 1,2^ − 1) + ^^^^(2^ + 1,2^ − 1)+ ^^^^(2^ − 1,2^) + 2 ∙ ^^^^(2^, 2^) + ^^^^(2^ + 1,2^) + 4] ≫ 3^^^^^ (^, ^) = [^^^^(2^, 2^ − 1) + ^^^^(2^ − 1,2^) + 4 ∙ ^^^^(2^, 2^)+ ^^^^(2^ + 1,2^) + ^^^^(2^, 2^ + 1) + 4] ≫ 3

[0087] Note that only one luma line (e.g.. general line buffer in intra prediction) is used tomake the down-sampled luma samples when the upper reference line is at the CTU boundary.

[0088] This parameter computation is performed as part of the decoding process and not justas an encoder search operation. As a result, no syntax is used to convey the α and β values to the decoder.

[0089] For chroma intra mode coding, a total of 8 intra modes are allowed for chroma intramode coding. Those modes include five traditional intra modes and three cross-component linear model modes (CCLM, LM_A, and LM_L). Chroma mode signaling and derivation process are shown in TABLE 1. Chroma mode coding directly depends on the intra prediction mode of the corresponding luma block. Since separate block partitioning structure for luma and chroma components is enabled in I slices, one chroma block may correspond to multiple luma blocks.Therefore, for chroma direct mode (DM), the intra prediction mode of the corresponding luma block covering the center position of the current chroma block is directly inherited. Chroma Corresponding luma intra prediction mode prediction mode0 50 18 1 X(0<= X<= 66) 066 0 0 0 01 50 66 50 50 502 18 18 66 18 183 1 1 1 66 14 0 50 18 1 X5 81 81 81 81 816 82 82 82 82 827 83 83 83 83 83TABLE 1

[0090] TABLE 1 illustrates derivation of the chroma prediction mode from luma mode whencclm is enabled.

[0091] A single binarization table may be used regardless of the value ofsps_cclm_enabled_flag as shown in TABLE 2, which illustrates a unified binarization table for chroma prediction mode. Value of intra_chroma_pred_mode Bin string4 000 01001 01012 01103 01115 106 1107 111TABLE 2

[0092] In TABLE 2, the first bin indicates whether regular mode (0) or LM mode (1) is used.If it is LM mode, then the next bin indicates whether it is LM_CHROMA (0) or not. If it is notLM_CHROMA, next 1 bin indicates whether it is LM_L (0) or LM_A (1). For this case, when sps_cclm_enabled_flag is 0, the first bin of the binarization table for the corresponding intra_chroma_pred_mode can be discarded prior to the entropy coding. Or, in other words, the first bin is inferred to be 0 and hence not coded. This single binarization table is used for both sps_cclm_enabled_flag equal to 0 and 1 cases. The first two bins in TABLE 2 are context coded with its own context model, and the rest of the bins are bypass coded.

[0093] In addition, in order to reduce luma-chroma latency in a dual tree, when the 64x64luma coding tree node is partitioned with Not Split (and ISP is not used for the 64x64 CU) or quadtree (QT), the chroma CUs in a 32x32 / 32x16 chroma coding tree node are allowed to use CCLM in the following way: if the 32x32 chroma node is not split or partitioned QT split, allchroma CUs in the 32x32 node may use CCLM; if the 32x32 chroma node is partitioned withHorizontal BT, and the 32x16 child node does not split or uses Vertical BT split, all chroma CUsin the 32x16 chroma node may use CCLM. In all the other luma and chroma coding tree splitconditions, CCLM is not allowed for chroma CU.

[0094] Features as described herein may generally relate to the multi-model linear model(MMLM). The CCLM included in VVC is extended by adding three multi-model LM (MMLM)modes. In each MMLM mode, the reconstructed neighboring samples are classified into twoclasses using a threshold which is the average of the luma reconstructed neighboring samples. Thelinear model of each class is derived using the least-mean-square (LMS) method. For the CCLMmode, the LMS method is also used to derive the linear model. Referring now to FIGs. 6A-B,illustrated are two luma-to-chroma models obtained for a luma (Y) threshold of 17. FIG. 6Aillustrates the sample domain, while FIG.6B illustrates the spatial domain. The luma samples may be classified into two classes used in the derivation of two sets of α and β. Each luma-to-chromamodel has its own linear model parameters α and β. As can be seen from FIG. 6B, each luma-to-chroma model corresponds to a spatial segmentation of the content (i.e., they correspond todifferent objects or textures in the scene).

[0095] Features as described herein may generally relate to a convolutional cross-componentmodel (CCCM). An improved version of cross-component prediction, known as CCCM, uses a 2D filter kernel to derive the luma-to-chroma model. The filter coefficients are derived decoder- side using reconstructed set of input data and chroma samples. For the filter coefficient derivation, co-located reference sample areas (consisting of reconstructed luma and chroma samples) are defined for both luma and chroma, as shown in FIG. 7, where the typically used 4:2:0 chroma down-sampling has been applied. The reference sample area for a given block, which may be usedfor the derivation of a CCCM filter, may be, for example, six lines above and left, as shown inFIG. 7, yet any number of reference lines (that can be realized by both the encoder and decoder) may be used. Generally, reference samples may contain any chroma (710) and luma (720) samples that have been reconstructed by both the encoder and decoder. Once the reference samples are determined, the filter coefficients may be derived, for example, using different types of linear regression tools such as ordinary least-squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or least absolute shrinkage and selection operator.

[0096] The dimensions of the filter kernel may be for example 1^3 (1D vertical), 3^1 (1Dhorizontal), 3^3, 7^7, or any other dimensions, and may be shaped (e.g. by selecting only a subset of all possible kernel locations) as a cross or a diamond (for example as shown in FIG.8) or as any given shape. When referring to the samples within the filter kernel, the following notation is used: north (above), east (right), south (below), west (left) and center, as illustrated in FIG. 8 using the letters N, E, S, W, C. 810 illustrates a 3-tap vertical filter kernel configuration. 820 illustrates a 3-tap horizontal filter kernel configuration. 830 illustrates a 5-tap cross filter kernel configuration. 840 illustrates a 25-tap diamond filter kernel configuration.

[0097] The overall method of reconstructing chroma samples using convolution between adecoder-side obtained filter kernel and a set of input data may be referred to as a convolutional cross-component model (CCCM). The following steps may be applied to perform a CCCMoperation: (1) Define co-located reference areas over the luma and chroma components. (2) Down-sample the luma samples to match the chroma grid (optional). (3) Scan the luma and chromasamples of the reference area and collect available statistics (such as auto-correlation matrix andcross-correlation vector) based on the filter shape. (4) Solve the filter coefficients by minimizingsquared-error (or any other metric) based on the available statistics (such as the auto-correlationmatrix and cross-correlation vector). (5) Calculate a predicted chroma block by convolving thedown-sampled luma samples with the filter kernel.

[0098] Let us define the (possibly down-sampled) luma samples as a 2D array ^(^, ^)indexed using horizontal ^-coordinate and vertical ^-coordinate. Let us also define the co-located chroma samples as a 2D array ^(^, ^) and the filter kernel (i.e., coefficients) as 3^3 array ^(^, ^).On a sample level we define the convolution between ^ and ^ as:

[0099] When using other data terms, such as the non-linear square-root term, the appendedconvolution becomes:where ^^ are filter coefficients that reside outside of the 2D filter kernel yet have been obtained asa part of the system of linear equations that were used to solve the 2D filter coefficients in Step 4 above. Similarly, we can add the bias term to the convolution with:

[00100] Features as described herein may generally relate to multiple reference line (MRL)intra prediction. Multiple reference line (MRL) intra prediction uses more reference lines for intraprediction. In FIG. 9, an example of 4 reference lines is depicted, where the samples of segmentsA (910) and F (940) are not fetched from reconstructed neighboring samples but padded with theclosest samples from segments B (920) and E (930), respectively. HEVC intra-picture predictionuses the nearest reference line (i.e., reference line 0). In MRL, 2 additional lines (reference line 1and reference line 3) are used.

[00101] The index of selected reference line (mrl_idx) is signaled and used to generate intrapredictor. For reference line idx, which is greater than 0, additional reference line modes are onlyincluded in the MPM list and the mpm index is only signaled without remaining mode. Thereference line index is signaled before intra prediction modes, and the planar mode is excluded from intra prediction modes in case a nonzero reference line index is signaled.

[00102] MRL is disabled for the first line of blocks inside a CTU to prevent using extendedreference samples outside the current CTU line. Also, PDPC is disabled when an additional line(s)is used. For MRL mode, the derivation of DC value in DC intra prediction mode for non-zeroreference line indices are aligned with that of reference line index 0. MRL requires the storage of3 neighboring luma reference lines with a CTU to generate predictions. The cross-componentlinear model (CCLM) tool also requires 3 neighboring luma reference lines for its down-samplingfilters. The definition of MLR to use the same 3 lines is aligned as CCLM to reduce the storagerequirements for decoders.

[00103] Features as described herein may generally relate to intra sub-partitions (ISP). Theintra sub-partitions (ISP) divides luma intra-predicted blocks vertically or horizontally into 2 or 4sub-partitions, depending on the block size. For example, the minimum block size for ISP is 4x8(or 8x4). If the block size is greater than 4x8 (or 8x4), then the corresponding block is divided by4 sub-partitions.

[00104] It has been noted that the ^ × 128 (with ^ ≤ 64) and 128 × ^ (with ^ ≤ 64) ISPblocks could generate a potential issue with the 64 × 64 VDPU. For example, an ^ × 128 CU inthe single tree case has an ^ × 128 luma transport block (TB) and two corresponding^ ^ × 64chroma TBs. If the CU uses ISP, then the luma TB will be divided into four ^ × 32 TBs (e.g. onlythe horizontal split is possible), each of them smaller than a 64 × 64 block. However, in the currentdesign of ISP chroma blocks are not divided. Therefore, both chroma components will have a sizegreater than a 32 × 32 block. Analogously, a similar situation could be created with a 128 × ^ CUusing ISP. Hence, these two cases are an issue for the 64 × 64 decoder pipeline. For this reason,the CU sizes that can use ISP is restricted to a maximum of 64 × 64. All sub-partitions fulfill thecondition of having at least 16 samples.

[00105] Features as described herein may generally relate to matrix weighted intra prediction(MIP). The MIP method is a newly added intra prediction technique in VVC. For predicting thesamples of a rectangular block of width ^ and height ^, MIP takes one line of H reconstructedneighboring boundary samples left of the block and one line of ^ reconstructed neighboringboundary samples above the block as input. If the reconstructed samples are unavailable, they aregenerated as in the conventional intra prediction. The generation of the prediction signal is basedon the following three steps: averaging (1010), matrix vector multiplication (1020), and linearinterpolation (1030), for example as shown in FIG. 10.

[0106] Features as described herein may generally relate to inter prediction in VVC. Inmerge prediction, for a current CU, both encoder and decoder construct a same list of merge candidates. The merge candidates hold the motion information of the past inter coded CUs around the current CU both spatially and temporally. The encoder may select a merge candidate (i.e. motion information) from the merge candidate list for the current CU, and may signal to the decoder which merge candidate in the merge candidate list is to be used for the current CU. Amerge list may include the following candidates: spatial motion vector prediction (MVP) fromspatial neighbor CUs; temporal MVP from collocated CUs; history-based MVP from a first in first out (FIFO) table; pairwise average MVP (using the candidates already in the list); and / or zero motion vectors (MV).

[00107] Merged mode width motion vector difference (MMVD) is used to signal motionvector deltas (MVD) and a resolution index after signaling the merge candidate. In symmetricMVD, motion information of list-1 is derived from motion information of list-0 in the bi-predictioncase. In affine prediction, several motion vectors are indicated / signaled for different corners of a block, which are used to derive the motion vectors of sub-block. In affine merge, affine motion information of a block is generated based on the normal or affine motion information of the neighboring blocks. In sub-block-based temporal motion vector prediction, motion vectors of sub- blocks of the current block are predicted from proper subblocks in the reference frame, which are indicated by the motion vector of a spatial neighboring block (if available). In adaptive motion vector resolution (AMVR), precision of MVD is signaled for each CU. In bi-prediction with CU- level weight, an index indicates the weight values for a weighted average of two prediction block. Bi-directional optical flow (BDOF) refines the motion vectors in the bi-prediction case. BDOF generates two prediction blocks using the signaled motion vectors. Then, a motion refinement iscalculated to minimize the error between two prediction blocks using their gradient values. Thefinal prediction blocks are refined using the motion refinement and gradient values.

[0108] Features as described herein may generally relate to bi-prediction with CU-levelweight (BCW) and weighted prediction (WP). In HEVC, the bi-prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bi-prediction mode is extended beyond simple averaging to allow weighted averaging of the two prediction signals, for example:

[00109] Five weights are allowed in the weighted averaging bi-prediction, ^ ∈ {−2, 3, 4, 5,10}. For each bi-predicted CU, the weight w is determined in one of two ways: 1) for a non- merge CU, the weight index is signaled after the motion vector difference; 2) for a merge CU, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width times CU height is greater than or equal to 256). For low-delay pictures, all 5 weights are used. For non-low-delay pictures,only 3 weights (w ∈ {3,4,5}) are used.

[0110] At the encoder, fast search algorithms are applied to find the weight index withoutsignificantly increasing the encoder complexity. These algorithms are summarized as follows. When combined with AMVR, unequal weights are only conditionally checked for 1-pel and 4-pelmotion vector precisions if the current picture is a low-delay picture. When combined with affine,affine motion estimation (ME) will be performed for unequal weights if and only if the affine modeis selected as the current best mode. When the two reference pictures in bi-prediction are the same, unequal weights are only conditionally checked. Unequal weights are not searched when certain conditions are met, depending on the POC distance between current picture and its reference pictures, the coding QP, and the temporal level.

[00111] The BCW weight index is coded using one context coded bin followed by bypasscoded bins. The first context coded bin indicates if equal weight is used; and if unequal weight isused, additional bins are signaled using bypass coding to indicate which unequal weight is used.

[0112] Weighted prediction (WP) is a coding tool supported by the H.264 / AVC and HEVCstandards to efficiently code video content with fading. Support for WP was also added into the VVC standard. WP allows weighting parameters (weight and offset) to be signaled for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the weight(s) and offset(s) of the corresponding reference picture(s) are applied. WP and BCW are designed for different types of video content. In order to avoid interactions between WP and BCW, which will complicate VVC decoder design, if a CU uses WP, then the BCW weight index is not signaled, and w is inferred to be 4 (i.e. equal weight is applied). For amerge CU, the weight index is inferred from neighboring blocks based on the merge candidateindex. This can be applied to both normal merge mode and inherited affine merge mode. Forconstructed affine merge mode, the affine motion information is constructed based on the motioninformation of up to 3 blocks. The BCW index for a CU using the constructed affine merge modeis simply set equal to the BCW index of the first control point MV.

[00113] In VVC, CIIP and BCW cannot be jointly applied for a CU. When a CU is codedwith CIIP mode, the BCW index of the current CU is set to 2 (e.g. equal weight).

[0114] Features as described herein may generally relate to combined inter and intraprediction (CIIP). In VVC, when a CU is coded in merge mode, if the CU contains at least 64 luma samples (that is, CU width times CU height is equal to or larger than 64), and if both CU width and CU height are less than 128 luma samples, an additional flag is signaled to indicate if the combined inter / intra prediction (CIIP) mode is applied to the current CU. As its name indicates, the CIIP prediction combines an inter prediction signal with an intra prediction signal. The inter prediction signal in the CIIP mode ^^^^^^is derived using the same inter prediction process applied to regular merge mode; and the intra prediction signal ^^^^^^is derived following the regular intra prediction process with the planar mode. Then, the intra and inter prediction signals are combined using weighted averaging, where the weight value is calculated dependingon the coding modes of the top and left neighboring blocks (depicted in FIG. 11) as follows: if thetop neighbor is available and intra coded, then set isIntraTop to 1, otherwise set isIntraTop to 0; if the left neighbor is available and intra coded, then set isIntraLeft to 1, otherwise set isIntraLeft to 0; if (isIntraLeft + isIntraTop) is equal to 2, then wt is set to 3; otherwise, if (isIntraLeft + isIntraTop) is equal to 1, then wt is set to 2; otherwise, set wt to 1. The CIIP prediction is formed as follows: ^^^^^ = ^(4 − ^^)

[00115] Referring now to FIG. 11, illustrated is an example where top (1110) and left (1120)neighboring blocks are used in CIIP weight derivation.

[00116] Features as described herein may generally relate to local illumination compensation(LIC). LIC is an inter prediction technique to model local illumination variation between thecurrent block and its prediction block as a function of that between the current block template andthe reference block template. The parameters of the function can be denoted by a scale α and anoffset β, which forms a linear equation, that is, α*p[x]+β to compensate illumination changes,where p[x] is a reference sample pointed to by MV at a location x on a reference picture. Since αand β can be derived based on the current block template and the reference block template, nosignaling overhead is required for them, except that an LIC flag is signaled for AMVP mode toindicate the use of LIC.

[00117] The local illumination compensation proposed in JVET-00066 is used in ECM foruni-prediction inter CUs with the following modifications. Intra neighbor samples can be used in LIC parameter derivation. LIC is disabled for blocks with less than 32 luma samples. For bothnon-subblock and affine modes, LIC parameter derivation is performed based on the templateblock samples corresponding to the current CU, instead of partial template block samples corresponding to first top-left 16x16 unit. Samples of the reference block template are generatedby using MC with the block MV without rounding it to integer-pel precision.

[0118] Features as described herein may generally relate to handling of out-of-boundarysamples (OOB). In bi-directional motion compensation the out of boundary (OOB) prediction samples are discarded and only the non-OOB predictors, when available, are used to generate the final predictor. Specifically, let ^^^_^i,j and ^^^_^i,j denote the position of one prediction samplein one current block; ^^_^^^^^^,^and ^^_^^,^ (x = 0,1) denote the MV of the current block; and^^^LeftBdry, ^^^RightBdr, ^^^TopBdry and ^^^BottomBdry are the positions of four boundaries of thepicture. One prediction sample is regarded as OOB when at least one of the following conditionsis satisfied: (^^^_^i,j+ ^^_^^^^,^ ) > (^^^RightBdry+half_pixel),(^^^_^i,j + ^^_^^^,^^) > (^^^LeftBdry-half_pixel), (^^^_yi,j + ^^_^^^,^^) > (^^^BottomBdry+half_pixel),where half_pixel is equal to 8, which represents the half-pel sample distance in the 1 / 16-pel sampleprecision.

[0119] After examining the OOB condition for each sample, the final prediction samples ofone bi-directional block are generated as follows:

[00120] If ^^^,^^is OOB and ^^^,^^is non-OOB: ^ ^^^^^= ^^^ ^,^ ^,^

[0121] else^non-OOB and ^^,^^is OOB: ^ ^^^^^ ^= ^^^ ,^ ^,^

[0122] else

[00123] The OOB checking process is also applicable when BCW is enabled.

[0124] Finally, note this sample-adaptive bi-prediction process only applies to predictionunits for which at least a reference block is first detected as partially or entirely out-of-bounds.Thus, a block-level OOB criteria is first checked. If both prediction blocks are non-OOB, then theusual bi-prediction takes place.

[00125] Features as described herein may generally relate to in-loop filters. There are in totalthree in-loop filters in VVC. Besides deblocking filter and SAO (the two loop filters in HEVC),adaptive loop filter (ALF) may be applied. The ALF comprises of luma ALF, chroma ALF andcross-component ALF (CC-ALF). The ALF filtering process may be designed such that lumaALF, chroma ALF and CC-ALF can be executed in parallel. The order of the filtering process inthe VVC is the deblocking filter, SAO, and then ALF. The SAO in VVC is the same as that inHEVC.

[0126] In VVC, an adaptive loop filter (ALF) with block-based filter adaption is applied. Forthe luma component, one among 25 filters is selected for each 4×4 block, based on the direction and activity of local gradients. In other words, a filter may be selected based on the output of a Laplacian classifier, which may be a 4x4 block. Filter adaptation may be performed for the filter of each class.

[00127] Two diamond filter shapes (as shown in FIG.12) may be used for the ALF processing.The 7×7 diamond shape (1220) may be applied for the luma components, and the 5×5 diamondshape (1210) may be applied for the chroma components.

[0128] For the luma component, each 4 × 4 block may be categorized or classified into oneout of 25 classes. The classification index C may be derived based on its directionality ^ and aquantized value of activity ^^, for example as follows: ^= 5^ + ^^

[0129] To calculate ^ and ^^, gradients of the horizontal, vertical and two diagonal directionmay first be calculated using 1-D (one dimensional) Laplacian:

[0130] Where indices ^ and ^ refer to the coordinates of the upper left sample within the 4 ×4 block and ^(^, ^) indicates a reconstructed sample at coordinate (^, ^).

[00131] To reduce the complexity of block classification, the subsampled 1-D Laplaciancalculation is applied. As shown in FIGs. 13A-D, the same subsampled positions may be used forgradient calculation of all directions. The letters indicate the subsampled positions, while the gray blocks indicate the classification based on the subsampled positions. FIG. 13A illustrates subsampled positions for the vertical gradient. FIG.13B illustrates subsampled positions for the horizontal gradient. FIG.13C illustrates subsampled positions for a diagonal gradient. FIG.13D illustrates subsampled positions for another diagonal gradient.

[00132] After block classification, ^ maximum and minimum values of the gradients ofhorizontal and vertical directions may be set as:

[00133] The maximum and minimum values of the gradient of two diagonal directions maybe set as:

[00134] To derive the value of the directionality ^, these values are compared against eachother and with two thresholds ^1 and ^2:

[00135] Step 1: If both

[00136] Step 2: If ^^^,^^^ / ^^^,^^^> ^^^^^,^^^ / ^^^^^,^^^, continue from Step 3; otherwise continue from Step 4.

[00137] Stepset to 2; otherwise ^ is set to 1.

[0138] Stepotherwise ^ is set to 3.

[0139] The activity value ^ is calculated as:

[00140] ^ is further quantized to the range of 0 to 4, inclusively, and the quantized value isdenoted as ^^.

[00141] For chroma components in a picture, no classification method is applied.

[0142] Before filtering each 4×4 luma block (e.g. Laplacian classification), geometrictransformations such as rotation or diagonal and vertical flipping may be applied to the filter coefficients ^(^, ^) and to the corresponding filter clipping values ^(^, ^), depending on gradient values calculated for that block. This is equivalent to applying these transformations to the samples in the filter support region. The idea is to make different blocks to which ALF is applied more similar by aligning their directionality.

[00143] Three geometric transformations, including diagonal, vertical flip and rotation areintroduced: Diagonal: ^D(^, ^) = ^(^, ^), ^D(^, ^) = ^(^, ^) Vertical flip: ^V(^, ^) = ^(^, ^ − ^ − 1), ^V(^, ^) = ^(^, ^ − ^ − 1)Rotation: ^R(^, ^) = ^(^ − ^ − 1, ^), ^R(^, ^) = ^(^ − ^ − 1, ^)where ^ is the size of the filter and 0 ≤ ^, ^ ≤ ^ − 1 are coefficients coordinates, such that location(0,0) is at the upper left corner and location (^ − 1, ^ − 1) is at the lower right corner. Thetransformations are applied to the filter coefficients f (k, l) and to the clipping values ^(^, ^)depending on gradient values calculated for that block. The relationship between thetransformation and the four gradients of the four directions are summarized in the following table.

[0144] TABLE 3 illustrates a mapping of the gradient calculated for one block and thetransformations: Gradient values Transformationgd2 < gd1 and gh < gv No transformationgd2 < gd1 and gv < gh Diagonalgd1 < gd2 and gh < gv Vertical flipgd1 < gd2 and gv < gh RotationTABLE 3

[00145] At decoder side, when ALF is enabled for a CTB, each sample ^(^, ^) within the CUis filtered, resulting in sample value ^′(^, ^) as shown below:≫7)where ^(^, ^) denotes the decoded filter coefficients, ^(^, ^) is the clipping function and ^(^, ^)denotes the decoded clipping parameters. The variables k and l vary between −^ ^ andwhere L denotes the filter length.

[00146] In the equation above for ALF filtering, if the number of coefficients (f(k,l)) and thenumber of clipping values (c(k,l)) are N, in an example embodiment the number of coefficientsand clipping values may be extended to N+M. In other words, an additional M coefficients andclipping values may be added to the equation. The training and application of those coefficientsand clipping values may be kept the same as in ALF. In an example embodiment, a new filtershape for the trained new coefficients and clipping values may be chosen in a similar shape asshown in FIG. 12. In an example embodiment, a larger filter shape may be selected or determinedfor performing ALF filtering based, at least partially, on the Laplacian filter or edge values.

[0147] The clipping function ^(^, ^) = min (^, max(−^, ^)) corresponds to the function ^^^^3(−^, ^, ^). The clipping operation introduces non-linearity to make ALF more efficient byreducing the impact of neighbor sample values that are too different from the current sample value.

[0148] CC-ALF uses luma sample values to refine each chroma component by applying anadaptive, linear filter to the luma channel and then using the output of this filtering operation for chroma refinement. FIG. 14A provides a system level diagram of the CC-ALF process with respect to the SAO (1410, 1420, 1430), luma ALF (1440) and chroma ALF (1450) processes.

[00149] Filtering in CC-ALF is accomplished by applying a linear, diamond shaped filter (e.g.1480 as illustrated in FIG. 14B) to the luma channel (1460). One filter is used for each chromachannel (1470), and the operation is expressed as:where (^, ^) is the chroma component i location being refined, (^Y, ^Y) is the luma location basedon (^, ^), ^i is filter support area in luma component, and ^i(^0, ^0) represents the filter coefficients.

[00150] As shown in FIGs. 14A-B, the luma filter support is the region collocated with thecurrent chroma sample after accounting for the spatial scaling factor between the luma and chroma planes.

[00151] In the VVC reference software, CC-ALF filter coefficients are computed byminimizing the mean square error of each chroma channel with respect to the original chromacontent. To achieve this, the VVC test model (VTM) algorithm uses a coefficient derivationprocess similar to the one used for chroma ALF. Specifically, a correlation matrix is derived, and the coefficients are computed using a Cholesky decomposition solver in an attempt to minimize a mean square error metric. In designing the filters, a maximum of 8 CC-ALF filters can be designed and transmitted per picture. The resulting filters are then indicated for each of the two chroma channels on a CTU basis.

[0152] Adaptive in-loop filters (such as ALF in VVC) may perform luma and chromafiltering in parallel fashion to lower the mean square error (MSE) between the reconstruction and the original samples. More specifically, in VVC the ALF may first filter both luma and chroma inparallel, and subsequently may also apply cross-component filtering (CC-ALF) to further improvethe chroma (see, e.g., FIG. 14A). The luma ALF (1510) and chroma ALF (1520) may use luma reconstruction and chroma buffers as an input, respectively (see, e.g., FIG.15).

[00153] The input set diversity to luma ALF may be important to derive statistics and trainingfilter coefficients of ALF filters. In VVC, only luma reconstruction and chroma buffers may beinput to luma ALF and chroma ALF, respectively.

[0154] In an example embodiment, Laplacian filtered / enhanced luma and chromareconstruction buffers and / or Laplacian edge buffers (obtained from luma and chroma reconstruction buffers by deriving Laplacian edges) may be included as input to luma ALF and chroma ALF, respectively.

[00155] In an example embodiment, a Laplacian enhanced reconstruction buffer and / orLaplacian edges obtained from a reconstruction buffer may give an additional source of information, which may be helpful to derive statistics for training filter coefficients.

[00156] Referring now to FIG. 16, illustrated is an example of luma (1620) and chroma ALF(1640) with additional Laplacian filtering input (1610, 1630) according to an example embodiment of the present disclosure. In an example embodiment, the ALF pipeline may include Laplacian enhanced buffers.

[00157] In an example embodiment, a Laplacian filter may be applied to luma and chromareconstruction buffers. In an example embodiment, information from a buffer of Laplacianfiltering information may be used as additional input to luma and chroma ALFs.

[00158] Referring now to FIG. 17, illustrated is an example of luma (1720) and chroma ALF(1740) with additional Laplacian edge input (1710, 1730) according to an example embodiment ofthe present disclosure. In an example embodiment, the ALF pipeline may include Laplacian edgeinformation buffers.

[0159] In an example embodiment, a Laplacian filter may be applied to luma and chromareconstruction buffers. In an example embodiment, information from a buffer of Laplacian edge input may be used as additional input to luma and chroma ALFs.

[00160] In an example embodiment, Laplacian enhancements / edges may be derived, and edgestrengths may be adjusted, based on a slice temporal layer identifier and / or a quantizationparameter. A technical effect of example embodiments of the present disclosure, for examplecontrolling edge strengths, may be to avoid over enhancement, or introduction of artificialedges / artifacts.

[00161] In an example embodiment, coding unit (CU), transform unit (TU), and / or predictionunit (PU) boundaries may be tackled / treated / considered separately when deriving Laplacian edgesand applying Laplacian filtering. A technical effect of example embodiments of the presentdisclosure may be to avoid artificial edge enhancements.

[0162] In an example embodiment, Laplacian information may be given / provided to lumaALF and chroma ALF simultaneously. Alternatively, Laplacian information may begiven / provided to luma ALF and chroma ALF in sequence (e.g. luma ALF may be performed atleast partially before chroma ALF is performed, or vice versa). Alternatively, Laplacian information may be given / provided to only one of the ALF processes (e.g. only to luma ALF or only to chroma ALF).

[00163] In an example embodiment, encoder side ALF filter parameters derivation / trainingand signaling may be kept the same as in VVC. In an example embodiment, the number of filtercoefficients and clipping values may be increased. For example, one or more additional coefficients for ALF may be used based, at least partially, on Laplacian information. In an exampleembodiment, decoder side ALF filter parameter reading and filtering may be kept the same as inVVC.

[00164] FIG. 18 illustrates the potential steps of an example method 1800. The examplemethod 1800 may include: obtaining one or more sample value differences associated with samplereconstruction, 1810; performing filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample comprises at least one of: at least one luma sample, or at least one chroma sample, 1820; and providing the at least one filtered sample, 1830. The example method 1800 may be performed, for example, with a user equipment, an image encoder, a video encoder, an image decoder, a video decoder, a codec, etc.

[00165] In accordance with one example embodiment, an apparatus may comprise: at leastone processor; and at least one memory storing instructions that, when executed by the at least oneprocessor, cause the apparatus at least to: obtain one or more sample value differences associatedwith sample reconstruction; perform filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample may compriseat least one of: at least one luma sample, or at least one chroma sample; and provide the at leastone filtered sample.

[00166] The at least one filtered sample may be provided for further filtering.

[0167] The one or more sample value differences may comprise at least one of: one or moregradient values, one or more Laplacian values, one or more Laplacian filtering values, or one or more Laplacian edge derivation values.

[0168] The one or more gradient values may comprise at least one of: one or more horizontalgradient values, one or more vertical gradient values, one or more diagonal gradient values, one or more directions of a gradient, or one or more activities of the gradient.

[00169] The at least one sample may comprise at least one sample filtered with at least onesample adaptive offset filter.

[00170] The at least one filter may comprise at least one of: at least one adaptive loop filter,or at least one cross-component adaptive loop filter.

[00171] The one or more sample value differences may be obtained from at least one buffer.

[0172] The example apparatus may be further configured to: adjust at least one edge strengthassociated with the one or more sample value differences based, at least partially, on at least one of: at least one slice temporal layer identifier, or at least one quantization parameter.

[00173] The example apparatus may be further configured to: adjust at least one of the one ormore sample value differences based, at least partially, on at least one of: at least one coding unit boundary, at least one transform unit boundary, or at least one prediction unit boundary.

[00174] Performing filtering of the at least one sample may comprise the example apparatusbeing further configured to: determine one or more first correlation parameters based on a plurality of luma samples; determine one or more second correlation parameters based on a plurality of chroma samples, wherein the plurality of luma samples and the plurality of chroma samples may be at least partially co-located; and determine one or more coefficients for the at least one filter based, at least partially, on the one or more first correlation parameters and the one or more second correlation parameters.

[00175] Performing filtering of the at least one sample may comprise the example apparatusbeing further configured to: determine at least one coefficient for the at least one filter based, at least partially, on the one or more sample value differences.

[0176] Performing filtering of the at least one sample may comprise the example apparatusbeing further configured to: increase a size of the at least one filter based, at least partially, on the one or more sample value differences.

[00177] Performing filtering of the at least one sample may comprise the example apparatusbeing further configured to: add at least one coefficient to the at least one filter based, at least partially, on the one or more sample value differences.

[00178] The example apparatus may be further configured to: provide at least one filteredpicture based, at least partially, on the at least one filtered sample.

[00179] The example apparatus may comprise at least one of: an image encoder, a videoencoder, an image decoder, a video decoder, or a codec.

[00180] In accordance with one aspect, an example method may be provided comprising:obtaining, with a user equipment, one or more sample value differences associated with sample reconstruction; performing filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample may comprise at least one of: at least one luma sample, or at least one chroma sample; and providing the at least one filtered sample.

[00181] The at least one filtered sample may be provided for further filtering.

[0182] The one or more sample value differences may comprise at least one of: one or moregradient values, one or more Laplacian values, one or more Laplacian filtering values, or one or more Laplacian edge derivation values.

[00183] The one or more gradient values may comprise at least one of: one or more horizontalgradient values, one or more vertical gradient values, one or more diagonal gradient values, one or more directions of a gradient, or one or more activities of the gradient.

[0184] The at least one sample may comprise at least one sample filtered with at least onesample adaptive offset filter.

[00185] The at least one filter may comprise at least one of: at least one adaptive loop filter,or at least one cross-component adaptive loop filter.

[00186] The one or more sample value differences may be obtained from at least one buffer.

[0187] The example method may further comprise: adjusting at least one edge strengthassociated with the one or more sample value differences based, at least partially, on at least one of: at least one slice temporal layer identifier, or at least one quantization parameter.

[00188] The example method may further comprise: adjusting at least one of the one or moresample value differences based, at least partially, on at least one of: at least one coding unit boundary, at least one transform unit boundary, or at least one prediction unit boundary.

[00189] The performing of filtering of the at least one sample may comprise: determining oneor more first correlation parameters based on a plurality of luma samples; determining one or more second correlation parameters based on a plurality of chroma samples, wherein the plurality of luma samples and the plurality of chroma samples may be at least partially co-located; and determining one or more coefficients for the at least one filter based, at least partially, on the one or more first correlation parameters and the one or more second correlation parameters.

[00190] The performing of filtering of the at least one sample may comprise: determining atleast one coefficient for the at least one filter based, at least partially, on the one or more sample value differences.

[00191] The performing of filtering of the at least one sample may comprise: increasing a sizeof the at least one filter based, at least partially, on the one or more sample value differences.

[0192] The performing of filtering of the at least one sample may comprise: adding at leastone coefficient to the at least one filter based, at least partially, on the one or more sample value differences.

[00193] The example method may further comprise: providing at least one filtered picturebased, at least partially, on the at least one filtered sample.

[00194] The user equipment may comprise at least one of: an image encoder, a video encoder,an image decoder, a video decoder, or a codec.

[00195] In accordance with one example embodiment, an apparatus may comprise: circuitryconfigured to perform: obtaining, with a user equipment, one or more sample value differencesassociated with sample reconstruction; circuitry configured to perform: performing filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample,wherein the at least one sample may comprise at least one of: at least one luma sample, or at leastone chroma sample; and circuitry configured to perform: providing the at least one filtered sample.

[00196] In accordance with one example embodiment, an apparatus may comprise: processingcircuitry; memory circuitry including computer program code, the memory circuitry and thecomputer program code configured to, with the processing circuitry, enable the apparatus to: obtainone or more sample value differences associated with sample reconstruction; perform filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample may comprise at least one of: at least one luma sample, or at least one chroma sample; and provide the at least one filtered sample.

[00197] As used in this application, the term “circuitry” or “means” may refer to one or moreor all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry) and (b) combinations of hardware circuits and software, such as (asapplicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.” This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[00198] In accordance with one example embodiment, an apparatus may comprise means for:obtaining one or more sample value differences associated with sample reconstruction; performing filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample may comprise at least one of: at least one luma sample, or at least one chroma sample; and providing the at least one filtered sample.

[00199] The at least one filtered sample may be provided for further filtering.

[0200] The one or more sample value differences may comprise at least one of: one or moregradient values, one or more Laplacian values, one or more Laplacian filtering values, or one or more Laplacian edge derivation values.

[00201] The one or more gradient values may comprise at least one of: one or more horizontalgradient values, one or more vertical gradient values, one or more diagonal gradient values, one or more directions of a gradient, or one or more activities of the gradient.

[0202] The at least one sample may comprise at least one sample filtered with at least onesample adaptive offset filter.

[00203] The at least one filter may comprise at least one of: at least one adaptive loop filter,or at least one cross-component adaptive loop filter.

[00204] The one or more sample value differences may be obtained from at least one buffer.

[0205] The means may be further configured for: adjusting at least one edge strengthassociated with the one or more sample value differences based, at least partially, on at least one of: at least one slice temporal layer identifier, or at least one quantization parameter.

[00206] The means may be further configured for: adjusting at least one of the one or moresample value differences based, at least partially, on at least one of: at least one coding unit boundary, at least one transform unit boundary, or at least one prediction unit boundary.

[00207] The means configured for performing filtering of the at least one sample maycomprise means configured for: determining one or more first correlation parameters based on a plurality of luma samples; determining one or more second correlation parameters based on a plurality of chroma samples, wherein the plurality of luma samples and the plurality of chroma samples may be at least partially co-located; and determining one or more coefficients for the at least one filter based, at least partially, on the one or more first correlation parameters and the one or more second correlation parameters.

[00208] The means configured for performing filtering of the at least one sample maycomprise means configured for: determining at least one coefficient for the at least one filter based, at least partially, on the one or more sample value differences.

[00209] The means configured for performing filtering of the at least one sample maycomprise means configured for: increasing a size of the at least one filter based, at least partially, on the one or more sample value differences.

[0210] The means configured for performing filtering of the at least one sample maycomprise means configured for: adding at least one coefficient to the at least one filter based, at least partially, on the one or more sample value differences.

[00211] The means may be further configured for: providing at least one filtered picture based,at least partially, on the at least one filtered sample.

[00212] The example apparatus may comprise at least one of: an image encoder, a videoencoder, an image decoder, a video decoder, or a codec.

[00213] A processor, memory, and / or example algorithms (which may be encoded asinstructions, program, or code) may be provided as example means for providing or causing performance of operation.

[00214] In accordance with one example embodiment, a (non-transitory) computer-readablemedium comprising instructions stored thereon which, when executed with at least one processor,cause the at least one processor to: cause obtaining, with a user equipment, of one or more samplevalue differences associated with sample reconstruction; filter of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample may comprise at least one of: at least one luma sample, or at least one chroma sample; and cause providing of the at least one filtered sample.

[00215] In accordance with one example embodiment, a (non-transitory) computer-readablemedium comprising program instructions stored thereon for performing at least the following: causing obtaining, with a user equipment, of one or more sample value differences associated with sample reconstruction; filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample may comprise at least oneof: at least one luma sample, or at least one chroma sample; and causing providing of the at least one filtered sample.

[00216] The at least one filtered sample may be provided for further filtering.

[0217] The one or more sample value differences may comprise at least one of: one or moregradient values, one or more Laplacian values, one or more Laplacian filtering values, or one or more Laplacian edge derivation values.

[00218] The one or more gradient values may comprise at least one of: one or more horizontalgradient values, one or more vertical gradient values, one or more diagonal gradient values, one or more directions of a gradient, or one or more activities of the gradient.

[00219] The at least one sample may comprise at least one sample filtered with at least onesample adaptive offset filter.

[00220] The at least one filter may comprise at least one of: at least one adaptive loop filter,or at least one cross-component adaptive loop filter.

[00221] The one or more sample value differences may be obtained from at least one buffer.

[0222] The example computer-readable medium may be further configured to: adjusting atleast one edge strength associated with the one or more sample value differences based, at least partially, on at least one of: at least one slice temporal layer identifier, or at least one quantization parameter.

[00223] The example computer-readable medium may be further configured to: adjusting atleast one of the one or more sample value differences based, at least partially, on at least one of: at least one coding unit boundary, at least one transform unit boundary, or at least one prediction unit boundary.

[0224] The program instructions for performing filtering of the at least one sample maycomprise program instructions for performing: determining one or more first correlation parameters based on a plurality of luma samples; determining one or more second correlation parameters based on a plurality of chroma samples, wherein the plurality of luma samples and the plurality of chroma samples may be at least partially co-located; and determining one or more coefficients for the at least one filter based, at least partially, on the one or more first correlation parameters and the one or more second correlation parameters.

[00225] The program instructions for performing filtering of the at least one sample maycomprise program instructions for performing: determining at least one coefficient for the at least one filter based, at least partially, on the one or more sample value differences.

[00226] The program instructions for performing filtering of the at least one sample maycomprise program instructions for performing: increasing a size of the at least one filter based, at least partially, on the one or more sample value differences.

[00227] The program instructions for performing filtering of the at least one sample maycomprise program instructions for performing: adding at least one coefficient to the at least one filter based, at least partially, on the one or more sample value differences.

[00228] The example computer-readable medium may be further configured to: providing atleast one filtered picture based, at least partially, on the at least one filtered sample.

[00229] The user equipment may comprise at least one of: an image encoder, a video encoder,an image decoder, a video decoder, or a codec.

[00230] In accordance with another example embodiment, a (non-transitory) program storagedevice readable by a machine may be provided, tangibly embodying instructions executable by themachine for performing operations, the operations comprising: causing obtaining, with a user equipment, of one or more sample value differences associated with sample reconstruction; filtering of at least one sample with at least one filter based, at least partially, on the one or moresample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample may comprise at least one of: at least one luma sample, or at least one chroma sample; and causing providing of the at least one filtered sample.

[00231] In accordance with another example embodiment, a (non-transitory) computer-readable medium comprising instructions that, when executed by an apparatus, cause the apparatusto perform at least the following: causing obtaining, with a user equipment, of one or more samplevalue differences associated with sample reconstruction; filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample may comprise at least one of: at least one luma sample, or at least one chroma sample; and causing providing of the at least one filtered sample.

[00232] A computer implemented system comprising: at least one processor and at least one(non-transitory) memory storing instructions that, when executed by the at least one processor,cause the system at least to perform: causing obtaining, with a user equipment, of one or moresample value differences associated with sample reconstruction; filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample may comprise at least one of: at least one luma sample, or at least one chroma sample; and causing providing of the at least one filtered sample.

[00233] A signal with embedded data, the signal being encoded in accordance with anencoding process which comprises: causing obtaining, with a user equipment, of one or moresample value differences associated with sample reconstruction; filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample may comprise at least one of: at least one luma sample, or at least one chroma sample; and causing providing of the at least one filtered sample.

[0234] A computer implemented system comprising: means for causing obtaining, with auser equipment, of one or more sample value differences associated with sample reconstruction; means for filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample may comprise at least one of: at least one luma sample, or at least one chroma sample; and means for causing providing of the at least one filtered sample.

[00235] The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e.tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[00236] It should be understood that the foregoing description is only illustrative. Variousalternatives and modifications can be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modification and variances which fall within the scope of the appended claims.

Claims

1. CLAIMS What is claimed is:

1. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed with the at least one processor, cause the apparatus at least to: obtain one or more sample value differences associated with sample reconstruction; perform filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample comprises at least one of the following: at least one luma sample, or at least one chroma sample; and provide the at least one filtered sample.

2. The apparatus of claim 1, wherein the one or more sample value differences comprise at least one value obtained via adaptive clipping.

3. The apparatus of claim 2, wherein the adaptive clipping is adapted based, at least partially, on at least one of the following: a slice resolution associated with the at least one sample, a frame resolution associated with the at least one sample, a picture resolution associated with the at least one sample, a slice temporal layer identifier associated with the at least one sample, a quantization parameter associated with the at least one sample, a sample location associated with the at least one sample, a slice type associated with the at least one sample, a picture type associated with the at least one sample, or a cost function associated with the sample reconstruction.

4. The apparatus of any of claims 2 to 3, wherein the at least one filtered sample isprovided for further filtering.

5. The apparatus of any of claims 2 to 4, wherein the one or more sample value differences comprise at least one of the following: one or more gradient values, one or more Laplacian values, one or more Laplacian filtering values, one or more Laplacian edge derivation values, one or more Gaussian filtering output values, one or more offline-trained fixed filtering output values, one or more clipped gradient values, one or more clipped Laplacian values, one or more clipped Laplacianfiltering values, one or more clipped Laplacian edge derivation values, one or more clipped Gaussian filtering output values, or one or more clipped offline-trained fixed filtering output values.

6. The apparatus of any of claims 2 to 5, wherein the at least one sample comprises at least one sample filtered with at least one sample adaptive offset filter.

7. The apparatus of any of claims 2 to 6, wherein the at least one filter comprises at least one of the following: at least one adaptive loop filter, or at least one cross-component adaptive loop filter.

8. The apparatus of any of claims 2 to 7, wherein the one or more sample value differences are obtained from at least one buffer.

9. The apparatus of any of claims 2 to 8, wherein obtaining the one or more sample value differences associated with sample reconstruction comprises the instructions, when executed with the at least one processor, cause the apparatus to: obtain a sample value difference via pre-filtering of the at least one sample; compare the sample value difference with a clipping value; in response to a determination that the sample value difference is greater than the clipping value, use the clipping value as one of the one or more sample value differences; and in response to a determination that the sample value difference is less than or equal to the clipping value, use the sample value difference as one of the one or more sample value differences.

10. The apparatus of claim 9, wherein the instructions, when executed with the at least one processor, cause the apparatus to: adaptively determine the clipping value based on at least one of the following: a slice resolution associated with the at least one sample, a frame resolution associated with the at least one sample, a picture resolution associated with the at least one sample, a slice temporal layer identifier associated with the at least one sample, a quantization parameter associated with the at least one sample, a sample location associated with the at least one sample,a slice type associated with the at least one sample, a picture type associated with the at least one sample, or a cost function associated with the sample reconstruction.

11. The apparatus of any of claims 2 to 10, wherein performing filtering of the at leastone sample comprises the instructions, when executed with the at least one processor, cause theapparatus to: determine one or more first correlation parameters based on a plurality of luma samples; determine one or more second correlation parameters based on a plurality of chroma samples, wherein the plurality of luma samples and the plurality of chroma samples are at leastpartially co-located; and determine one or more coefficients for the at least one filter based, at leastpartially, on the one or more first correlation parameters and the one or more second correlation parameters.

12. The apparatus of any of claims 2 to 11, wherein performing filtering of the at leastone sample comprises the instructions, when executed with the at least one processor, cause theapparatus to: determine at least one coefficient for the at least one filter based, at least partially, on the one or more sample value differences.

13. The apparatus of any of claims 2 to 12, wherein performing filtering of the at leastone sample comprises the instructions, when executed with the at least one processor, cause theapparatus to: increase a size of the at least one filter based, at least partially, on the one or more sample value differences.

14. The apparatus of any of claims 2 to 13, wherein performing filtering of the at leastone sample comprises the instructions, when executed with the at least one processor, cause theapparatus to: add at least one coefficient to the at least one filter based, at least partially, on the one or more sample value differences.

15. The apparatus of any of claims 2 to 14, wherein the instructions, when executed with the at least one processor, cause the apparatus to: provide at least one filtered picture based, at least partially, on the at least one filtered sample.

16. The apparatus of any of claims 2 to 15, wherein the apparatus comprises at least one of the following: an image encoder, a video encoder, an image decoder, a video decoder, or a codec.

17. The apparatus of claim 1, wherein the instructions, when executed with the at least one processor, cause the apparatus to: adjust at least one edge strength associated with the one or more sample value differences based, at least partially, on at least one of the following: at leastone slice temporal layer identifier, or at least one quantization parameter.

18. The apparatus of claim 1, wherein the instructions, when executed with the at least one processor, cause the apparatus to: adjust at least one of the one or more sample value differences based, at least partially, on at least one of the following: at least one coding unitboundary, at least one transform unit boundary, or at least one prediction unit boundary.

19. The apparatus of claim 1, wherein the one or more sample value differences comprise at least one of the following: one or more gradient values, one or more Laplacian values, one or more Laplacian filtering values, or one or more Laplacian edge derivation values.

20. The apparatus of claim 19, wherein the one or more gradient values comprise at least one of the following: one or more horizontal gradient values, one or more vertical gradient values, one or more diagonal gradient values, one or more directions of a gradient, or one or more activities of the gradient.

21. A method comprising: obtaining, with a user equipment, one or more sample value differences associated with sample reconstruction; performing filtering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample comprises at least one of the following: at least one luma sample, or at least one chroma sample; and providing the at least one filtered sample.

22. The method of claim 21, wherein the one or more sample value differences comprise at least one value obtained via adaptive clipping.

23. The method of claim 22, wherein the adaptive clipping is adapted based, at least partially, on at least one of the following: a slice resolution associated with the at least one sample, a frame resolution associated with the at least one sample, a picture resolution associated with the at least one sample, a slice temporal layer identifier associated with the at least one sample, a quantization parameter associated with the at least one sample, a sample location associated withthe at least one sample, a slice type associated with the at least one sample, a picture type associatedwith the at least one sample, or a cost function associated with the sample reconstruction.

24. The method of claim 22 or 23, wherein the at least one filtered sample is provided forfurther filtering.

25. The method of any of claims 22 to 24, wherein the one or more sample value differences comprise at least one of the following: one or more gradient values, one or moreLaplacian values, one or more Laplacian filtering values, one or more Laplacian edge derivationvalues, one or more Gaussian filtering output values, one or more offline-trained fixed filtering output values, one or more clipped gradient values, one or more clipped Laplacian values, one or more clipped Laplacian filtering values, one or more clipped Laplacian edge derivation values, one or more clipped Gaussian filtering output values, or one or more clipped offline-trained fixed filtering output values.

26. The method of any of claims 22 to 25, wherein the at least one sample comprises at least one sample filtered with at least one sample adaptive offset filter.

27. The method of any of claims 22 to 26, wherein the at least one filter comprises at least one of the following: at least one adaptive loop filter, or at least one cross-component adaptive loop filter.

28. The method of any of claims 22 to 27, wherein the one or more sample value differences are obtained from at least one buffer.

29. The method of any of claims 22 to 28, wherein the obtaining of the one or more sample value differences associated with sample reconstruction comprises: obtaining a sample value difference via pre-filtering of the at least one sample; comparing the sample value difference with a clipping value; in response to a determination that the sample value difference is greater than the clipping value, using the clipping value as one of the one or more sample valuedifferences; and in response to a determination that the sample value difference is less than orequal to the clipping value, using the sample value difference as one of the one or more sample value differences.

30. The method of claim 29, further comprising: adaptively determining the clipping value based on at least one of the following: a slice resolution associated with the at least one sample, a frame resolution associated with the at least one sample, a picture resolution associated with the at least one sample, a slice temporal layer identifier associated with the at least one sample, a quantization parameter associated with the at least one sample, a sample location associated with the at least one sample, a slice type associated with the at least one sample, a picture type associated with the at least one sample, or a cost function associated with the sample reconstruction.

31. The method of any of claims 22 to 30, wherein the performing of filtering of the at least one sample comprises: determining one or more first correlation parameters based on a plurality of luma samples; determining one or more second correlation parameters based on a plurality of chroma samples, wherein the plurality of luma samples and the plurality of chroma samples are at least partially co-located; and determining one or more coefficients for the at least one filter based, at least partially, on the one or more first correlation parameters and the one or more second correlation parameters.

32. The method of any of claims 22 to 31, wherein the performing of filtering of the at least one sample comprises: determining at least one coefficient for the at least one filter based, at least partially, on the one or more sample value differences.

33. The method of any of claims 22 to 32, wherein the performing of filtering of the at least one sample comprises: increasing a size of the at least one filter based, at least partially, on the one or more sample value differences.

34. The method of any of claims 22 to 33, wherein the performing of filtering of the at least one sample comprises: adding at least one coefficient to the at least one filter based, at least partially, on the one or more sample value differences.

35. The method of any of claims 22 to 34, further comprising: providing at least one filtered picture based, at least partially, on the at least one filtered sample.

36. The method of any of claims 22 to 35, wherein the user equipment comprises at least one of the following: an image encoder, a video encoder, an image decoder, a video decoder, or a codec.

37. The method of claim 21 further comprising: adjusting at least one edge strength associated with the one or more sample value differences based, at least partially, on at least oneof the following: at least one slice temporal layer identifier, or at least one quantization parameter.

38. The method of claim 21 further comprising: adjusting at least one of the one or more sample value differences based, at least partially, on at least one of the following: at least onecoding unit boundary, at least one transform unit boundary, or at least one prediction unitboundary.

39. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed with the at least one processor, cause the apparatus at least to: obtain one or more sample value differences associated with sample reconstruction; performfiltering of at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction, and adaptive clipping, to obtain at least one filtered sample, wherein the at least one sample comprises at least one of the following: at least one luma sample, or at least one chroma sample; and provide the at least one clipped filtered sample.

40. A method comprising: obtaining, with a user equipment, one or more sample value differences associated with sample reconstruction; performing filtering of at least one sample withat least one filter based, at least partially, on the one or more sample value differences associatedwith the sample reconstruction, and adaptive clipping, to obtain at least one filtered sample, wherein the at least one sample comprises at least one of the following: at least one luma sample, or at least one chroma sample; and providing the at least one clipped filtered sample.

41. A computer-readable medium comprising program instructions stored thereon for performing at least the following: causing obtaining, with a user equipment, of one or more samplevalue differences associated with sample reconstruction; filtering at least one sample with at leastone filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction to obtain at least one filtered sample, wherein the at least one sample comprises at least one of the following: at least one luma sample, or at least one chroma sample; and causing providing of the at least one filtered sample.

42. A computer-readable medium comprising program instructions stored thereon for performing at least the following: causing obtaining, with a user equipment, of one or more sample value differences associated with sample reconstruction; filtering at least one sample with at least one filter based, at least partially, on the one or more sample value differences associated with the sample reconstruction, and adaptive clipping, to obtain at least one filtered sample, wherein the at least one sample comprises at least one of the following: at least one luma sample, or at least one chroma sample; and causing providing of the at least one clipped filtered sample.

Images