Asymmetric in-loop filters with virtual boundaries

Abstract

According to an exemplary embodiment of the present invention, there is at least a method and an apparatus for determining a virtual boundary separating an image or a part of an image into a first area and a second area, and determining to perform filtering of at least one pixel of the first area, where the coding information of the second area is derived from the first area or is set to at least one value, if the coding information of the second area needs to be used to perform filtering of at least one pixel of the first area, or determining not to perform filtering of at least one pixel of the first area, if the coding information of the second area needs to be used to perform filtering of at least one pixel of the first area.

Description

[Technical Field]

[0001] Example and non-limiting embodiments relate generally to multimedia transport and encoding and decoding of information, and more particularly to asymmetric in-loop filters with virtual boundaries. [Background technology]

[0002] In multimedia systems, it is known to perform data compression and decoding. Summary of the Invention

[0003] The foregoing aspects and other features are explained in the following description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0004] [Figure 1] 1 is a diagram illustrating a schematic diagram of an electronic device employing an embodiment of the examples described herein. [Figure 2] FIG. 1 illustrates a schematic diagram of a user equipment suitable for employing example embodiments described herein. [Figure 3] FIG. 2 further illustrates, in schematic form, electronic devices employing example embodiments described herein connected using wireless and wired network connections. [Figure 4] FIG. 1 shows a schematic block diagram of an encoder used for data compression at a common level. [Figure 5] FIG. 10 illustrates that updated areas are not allowed to use coded information from non-updated areas. [Figure 6] FIG. 10 illustrates that non-updated areas are permitted to use coded information from updated areas. [Figure 7]FIG. 10 illustrates that deblocking may not be applied to pixels pi,i=0,1,2, or that pixels qi,i=0,1,2 in the non-updated area may be filled in and still applied. [Figure 8] FIG. 1 illustrates four edge classes. [Figure 9] FIG. 1 illustrates four edge categories. [Figure 10A] FIG. 10 illustrates that the SAO edge offset may not be applied to pixel p0, or pixel q0 may be filled in and still be applied. [Figure 10B] FIG. 10 illustrates that the SAO edge offset may not be applied to pixel q0, or pixel p0 may be filled in and still be applied. [Figure 11] FIG. 10 illustrates that offsets from BIF-luma, SAO, and CCSAO are added to the deblocking output. [Figure 12A] FIG. 10 illustrates that a BIF may not be applied to pixel p0,0, or the associated pixels including qi,0,i=0,1 may be filled in and still be applied. [Figure 12B] 10 illustrates that a BIF may not be applied to pixel q0,0, or the associated pixels, including pi,0,i=0,1, may be filled in and still be applied. [Figure 13] A diagram showing the CCSAO decoding workflow. [Figure 14] FIG. 10 illustrates that, with respect to a collocated chroma sample, a collocated luma sample can be selected from nine candidate locations. [Figure 15A] FIG. 10 illustrates that CCSAO may not be applied to pixel p0, or pixel q0 may be filled in and still be applied. [Figure 15B] FIG. 10 illustrates that CCSAO may not be applied to pixel q0, or pixel p0 may be filled in and still be applied. [Figure 16A] FIG. 10 illustrates that ALF may not be applied to pixel p0,0, or the associated pixels including qi,0,i=0,1,2 may be filled in and still be applied. [Figure 16B] 10 illustrates that ALF may not be applied to pixel q0,0, or the associated pixels including pi,0,i=0,1,2 may be filled in and still be applied. [Figure 17] A basic diagram of CCALF in VVC. [Figure 18] FIG. 10 is a diagram showing a 25-tap filter of CCALF in ECM. [Figure 19A] FIG. 10 illustrates that CCALF may not be applied to at least one of the collocated chroma pixels, or luma pixels qi,j, i=0,1,2,3 and j=0,1 may be filled in and still be applied. [Figure 19B] 10 shows that CCALF may not be applied to at least one of the collocated chroma pixels, or the luma pixels pi,j,i=0,1,2,3 and j=0,1 may be filled in and still be applied. [Figure 20] FIG. 1 illustrates an example apparatus configured to implement an asymmetric in-loop filter with a virtual boundary, according to embodiments described herein. [Figure 21] FIG. 1 illustrates an exemplary method for implementing an asymmetric in-loop filter with a virtual boundary, according to embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0005] Described herein is a practical approach for implementing asymmetric in-loop filters with virtual boundaries. The models described herein may be used to perform any task, such as data compression, data decompression, video compression, video decompression, image or video classification, object classification, object detection, object tracking, speech recognition, language translation, music transcription, etc.

[0006] In the following, suitable devices and possible mechanisms for implementing aspects of the asymmetric in-loop filter with virtual boundaries will be described in detail. In this regard, reference is first made to FIGS. 1 and 2, where FIG. 1 shows an exemplary block diagram of a device 50. The device may be an Internet of Things (IoT) device configured to perform various functions, such as collecting information via one or more sensors, receiving or transmitting information, and analyzing information collected or received by the device. The device may include a neural network weight update encoding system that may incorporate a codec. FIG. 2 shows the layout of the device according to an exemplary embodiment. Next, the elements of FIGS. 1 and 2 will be described.

[0007] The electronic device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag, or other low-power device. Alternatively, the electronic device may be a non-mobile computer or part of a computer. However, it will be understood that example embodiments described herein may be implemented in any electronic device or apparatus capable of processing data.

[0008] The device 50 may include a housing 30 for housing and protecting the device. The device 50 may further include a display 32 in the form of a liquid crystal display. In other embodiments of the examples described herein, the display may be any suitable display technology suitable for displaying images or video. The device 50 may further include a keypad 34 (or touch area 34). In other embodiments of the examples described herein, any suitable data or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.

[0009] The device may include a microphone 36 or any suitable audio input, which may be a digital or analog signal input. The device 50 may further include an audio output device, which in example embodiments described herein may be earphones 38, a speaker, or any one of an analog or digital audio output connection. The device 50 may also include a battery (or in other example embodiments described herein, the device may be powered by any suitable mobile energy device, such as a solar cell, a fuel cell, or a clockwork generator). The device may further include a camera 42 capable of recording or capturing images and / or video. The device 50 may further include an infrared port for short-range line-of-sight communication to other devices. In other embodiments, the device 50 may further include any suitable short-range communication solution, such as, for example, a Bluetooth wireless connection or a USB / Firewire wired connection.

[0010] The device 50 may include a controller 56, processor, or processor circuitry for controlling the device 50. The controller 56 may be connected to a memory 58, which in the example embodiment described herein may store both data in the form of image data and audio data, and / or may also store instructions for execution by the controller 56. The controller 56 may further be connected to a codec circuit 54 suitable for performing encoding / compression of neural network weight updates and / or decoding of audio and / or video data, or for assisting in the encoding and / or decoding performed by the controller.

[0011] The device 50 may further comprise a card reader 48 and a smart card 46, e.g., a UICC and a UICC reader, suitable for providing user information and authentication information for authentication and authorization of the user on the network.

[0012] The device 50 may include a radio interface circuit 52 connected to the controller and suitable for generating wireless communication signals for communication with, for example, a cellular communication network, a wireless communication system, or a wireless local area network. The device 50 may further include an antenna 44 connected to the radio interface circuit 52 for transmitting radio frequency signals generated by the radio interface circuit 52 to other devices, such as network nodes, and / or for receiving radio frequency signals from other devices.

[0013] The device 50 may include a camera capable of recording or detecting individual frames, which are then passed to a codec 54 or controller for processing. The device may receive video image data or machine learning data from another device for processing before transmission and / or storage. The device 50 may also receive images for encoding / decoding wirelessly or via a wired connection. The structural elements of the device 50 described above represent examples of means for performing the corresponding functions.

[0014] 3, an example of a system in which embodiment of the examples described herein may be utilized is shown. System 10 comprises a plurality of communication devices capable of communicating over one or more networks. System 10 may comprise any combination of wired or wireless networks, including, but not limited to, wireless cellular telephone networks (such as GSM, UMTS, CDMA, LTE, 4G, and 5G networks), wireless local area networks (WLANs) as defined by any of the IEEE 802.x standards, Bluetooth personal area networks, Ethernet local area networks, token ring local area networks, wide area networks, and the Internet.

[0015] System 10 may include both wired and wireless communication devices and / or apparatus 50 suitable for implementing example embodiments described herein.

[0016] For example, the system shown in Figure 3 shows a representation of a cellular network 11 and the Internet 28, which are accessible to the various devices shown in Figure 3 using communication links 2 (wired or wireless). Connections to the Internet 28 may include, but are not limited to, long-range wireless connections, short-range wireless connections, and various wired connections, including, but not limited to, telephone lines, cable lines, power lines, and similar communication paths.

[0017] Exemplary communication devices shown in system 10 may include, but are not limited to, electronic devices or apparatus 50, a combination personal digital assistant (PDA) and cell phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, and a notebook computer 22. The apparatus 50 may be stationary or mobile when carried by a moving person. The apparatus 50 may also be located on a vehicle or head mounted display (HMD) 17, including, but not limited to, a car, truck, taxi, bus, train, boat, airplane, bicycle, motorcycle, or any similar suitable vehicle.

[0018] Embodiments may also be implemented in set-top boxes, i.e., digital television receivers, which may or may not have display or wireless capabilities, in tablets or (laptop) personal computers (PCs) that include hardware and / or software for processing neural network data, in various operating systems, and in chipsets, processors, DSPs, and / or embedded systems that provide hardware / software-based coding.

[0019] Some or further devices may send and receive calls and messages and communicate with a service provider via a wireless connection 25 to a base station 24. The base station 24 may be connected to a network server 26 that enables communication between the cellular network 11 and the Internet 28. The system may include additional and different types of communication devices.

[0020] Communication devices may communicate using a variety of transmission technologies, including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time division multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11, 3GPP® Narrowband IoT, and any similar wireless communication technology. Communication devices involved in implementing various embodiments of the examples described herein may communicate using a variety of mediums, including, but not limited to, radio, infrared, laser, cabled connections, and any suitable connection.

[0021] In telecommunications and data networks, a channel may refer to either a physical channel or a logical channel. A physical channel may refer to a physical transmission medium such as a wire, while a logical channel may refer to a logical connection on a multiplexed medium that can carry multiple logical channels. A channel may be used to convey an information signal, e.g., a bitstream, from one or more senders (or transmitters) to one or more receivers.

[0022] Embodiments may also be implemented in so-called IoT devices. The Internet of Things (IoT) may be defined, for example, as the interconnection of embedded computing devices that are uniquely identifiable within the existing Internet infrastructure. A collection of different technologies can enable many areas of embedded systems, such as wireless sensor networks, control systems, and home / building automation, to be included in the Internet of Things (IoT). To utilize the Internet, IoT devices are provided with an IP address as a unique identifier. IoT devices may be equipped with wireless transmitters, such as WLAN or Bluetooth transmitters or RFID tags. Alternatively, IoT devices can access an IP-based network via a wired network, such as an Ethernet-based network or a power-line connection (PLC).

[0023] One application in which the asymmetric in-loop filter with virtual boundaries and model-level update skip in compressed incremental learning are important is the use case of a neural network-based codec, such as a neural network-based video codec. The video codec may use one or more neural networks. In the first case, the video codec may be a conventional video codec, such as a versatile video codec (VVC / H.266), that has been modified to include one or more neural networks. Examples of these neural networks are: 1. A neural network filter used as one of the in-loop filters of the VVC. 2. A neural network filter to replace one or more of the VVC's in-loop filters 3. Neural network filters used as post-processing filters 4. Neural Network Used to Perform Intra-Frame Prediction 5. Neural Network Used to Perform Interframe Prediction

[0024] In the second case, typically called an end-to-end learned video codec, the video codec may include a neural network that transforms input data into a more compressible representation. The new representation may be quantized, losslessly compressed, then losslessly decompressed and dequantized, after which another neural network may transform its input into reconstructed or decoded data.

[0025] In both of the above two cases, one or more neural networks may exist on the decoder side, and we consider the example of one neural network filter. The encoder may fine-tune the neural network filter by using ground truth data (uncompressed data) available on the encoder side. The fine-tuning may be performed to improve the neural network filter when applied to current input data, such as one or more video frames. The fine-tuning may include performing one or more optimization iterations on some or all learnable weights of the neural network filter. The optimization iterations may include calculating the gradient of a loss function with respect to some or all learnable weights of the neural network filter, for example, by using a backpropagation algorithm, and then updating some or all of the learnable weights by using an optimizer, such as a stochastic gradient descent optimizer. The loss function may include one or more loss terms. One exemplary loss term may be mean squared error (MSE). Other distortion metrics may be used as the loss term. The loss function may be calculated by providing one or more data to the inputs of a neural network filter, obtaining one or more corresponding outputs from the neural network filter, and calculating a loss term using one or more outputs from the neural network filter and one or more ground truth data. The difference between the fine-tuned neural network weights and the neural network weights before fine-tuning is called a weight update. This weight update needs to be encoded and provided to the decoder side along with the encoded video data and used at the decoder side to update the neural network filter. The updated neural network filter is then used as part of the video decoding process or as part of a video post-processing process. It is desirable to encode the weight update to require a small number of bits.Therefore, the embodiments described herein also consider this use case of neural network-based codecs as a possible application of weight update compression.

[0026] To further illustrate the use case of neural network-based codecs, the MPEG-2 transport stream (TS), specified in ISO / IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a format for carrying audio, video, and other media, as well as program or other metadata, in a multiplexed stream. Packet identifiers (PIDs) are used to identify elementary streams (also known as packetized elementary streams) within a TS. Thus, logical channels within an MPEG-2 TS may be considered to correspond to specific PID values.

[0027] Available media file format standards include the ISO Base Media File Format (ISO / IEC 14496-12, sometimes abbreviated as ISOBMFF) and the File Format for NAL Unit Structured Video (ISO / IEC 14496-15), which is derived from ISOBMFF.

[0028] A video codec consists of an encoder that converts input video into a compressed representation suitable for storage / transmission, and a decoder that can restore the compressed video representation to a displayable form. The video encoder and / or video decoder may also be separate from each other, i.e., they may not need to form a codec. Typically, an encoder discards some information in the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate).

[0029] Typical hybrid video encoders, e.g., many implementations of ITU-T H.263 and H.264, encode video information in two stages. First, pixel values ​​within a specific image area (or "block") are predicted, e.g., by motion compensation (locating and indicating an area in one of the previously coded video frames that closely corresponds to the block being coded) or by spatial means (using pixel values ​​surrounding the block being coded in a specified manner). Next, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the differences in pixel values ​​using a specified transform (e.g., the Discrete Cosine Transform (DCT) or a variant thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (image quality) and the size of the resulting coded video representation (file size or transmission bit rate).

[0030] In temporal prediction, the source of prediction is a previously decoded image (also known as a reference image). In intra block copy (IBC, also known as intra block copy prediction and current image reference), prediction is applied similarly to temporal prediction, but the reference image is the current image, and only previously decoded samples may be referenced in the prediction process. Inter-layer prediction or inter-view prediction may be applied similarly to temporal prediction, but the reference image is an image decoded from another scalable layer or another view, respectively. In some cases, inter-prediction may refer only to temporal prediction, while in other cases, inter-prediction may collectively refer to any of intra-block copy, inter-layer prediction, and inter-view prediction, provided that they are performed in the same or similar process as temporal prediction. Inter-prediction or temporal prediction may also be referred to as motion compensation or motion-compensated prediction.

[0031] Inter-prediction, sometimes called temporal prediction, motion compensation, or motion-compensated prediction, reduces temporal redundancy. In inter-prediction, the source of the prediction is a previously decoded image. Intra-prediction exploits the fact that adjacent pixels in the same image are likely to be correlated. Intra-prediction can be performed in the spatial domain or the transform domain, i.e., either sample values ​​or transform coefficients can be predicted. Intra-prediction is usually used in intra-coding, where inter-prediction is not applied.

[0032] One result of the encoding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be more efficiently entropy coded if they are first predicted from spatially or temporally neighboring parameters. For example, motion vectors may be predicted from spatially neighboring motion vectors, and only the difference relative to the motion vector predictor may be coded. Prediction of coding parameters and intra-prediction are sometimes collectively referred to as in-picture prediction.

[0033] FIG. 4 shows a block diagram of the general structure of a video encoder. While FIG. 4 shows an encoder for two layers, it will be understood that the presented encoder can be similarly extended to encode three or more layers. FIG. 4 shows a video encoder comprising a first encoder section 500 for a base layer and a second encoder section 502 for an enhancement layer. Each of the first encoder section 500 and the second encoder section 502 may comprise similar elements for encoding an incoming image. The encoder sections 500, 502 may comprise a pixel predictor 302, 402, a prediction error encoder 303, 403, and a prediction error decoder 304, 404. FIG. 4 also illustrates an embodiment of the pixel predictor 302, 402 as an inter predictor 306, 406 (P inter ), intra predictors 308, 408 (P intra), mode selector 310, 410, filter 316, 416(F), and reference frame memory 318, 418 (RFM). The pixel predictor 302 of the first encoder section 500 predicts the base layer picture (I) of the video stream to be encoded by both the inter predictor 306 (which determines the difference between the picture and the motion compensated reference frame 318) and the intra predictor 308 (which determines the prediction of the picture block based only on the already processed parts of the current frame or picture). 0,n ) 300. The outputs of both the inter and intra predictors are passed to a mode selector 310. The intra predictor 308 may have two or more intra prediction modes. Thus, each mode may perform intra prediction and provide a predicted signal to the mode selector 310. The mode selector 310 also receives a copy of the base layer image 300. In response, the pixel predictor 402 of the second encoder section 502 selects an enhancement layer image (I) of the video stream to be encoded by both the inter predictor 406 (which determines the difference between the image and a motion-compensated reference frame 418) and the intra predictor 408 (which determines a prediction of an image block based only on already processed portions of the current frame or image). 1,n ) 400. The outputs of both the inter and intra predictors are passed to a mode selector 410. The intra predictor 408 may have two or more intra prediction modes. Thus, each mode may perform intra prediction and provide a predicted signal to the mode selector 410. The mode selector 410 also receives a copy of the enhancement layer image 400.

[0034] Depending on the encoding mode selected for encoding the current block, the output of the inter predictor 306, 406, or the output of one of the optional intra predictor modes, or the output of the surface encoder in the mode selector, is passed to the output of the mode selector 310, 410. The output of the mode selector is passed to a first summing device 321, 421. The first summing device subtracts the output of the pixel predictor 302, 402 from the base layer image 300 / enhancement layer image 400 to generate a first prediction error signal 320, 420 (D n ) may be generated.

[0035] The pixel predictor 302, 402 receives a predicted representation (P' n ) and the output 338, 438 of the prediction error decoder 304, 404 (D' n ) and the combination of the pre-reconstructed images 314, 414 (I' n ) may be passed to the intra predictor 308, 408 and the filter 316, 416. The filter 316, 416 that receives the preliminary representation filters the preliminary representation to produce a final reconstructed image 340, 440 (R' n ) The reference frame memory 318 may be connected to the inter predictor 306, which uses the future base layer image 300 as a reference image to be compared with in the inter prediction operation. Provided that the base layer is selected and indicated as a source of the enhancement layer inter-layer sample prediction and / or inter-layer motion information prediction according to some embodiments, the reference frame memory 318 may also be connected to the inter predictor 406, which uses the future enhancement layer image 400 as a reference image to be compared with in the inter prediction operation. Furthermore, the reference frame memory 418 may be connected to the inter predictor 406, which uses the future enhancement layer image 400 as a reference image to be compared with in the inter prediction operation.

[0036] Provided that the base layer is selected and indicated as the source for predicting the filtering parameters of the enhancement layer in some embodiments, the filtering parameters from the filter 316 of the first encoder section 500 may be provided to the second encoder section 502.

[0037] The prediction error encoder 303, 403 includes a transform unit 342, 442 (T) and a quantizer 344, 444 (Q). The transform unit 342, 442 transforms the first prediction error signal 320, 420 into a transform domain. The transform is, for example, a DCT transform. The quantizer 344, 444 quantizes the transform domain signal, for example, DCT coefficients, to form quantized coefficients.

[0038] The prediction error decoder 304, 404 receives the output from the prediction error encoder 303, 403 and performs an inverse process to the prediction error encoder 303, 403 to produce a decoded prediction error signal 338, 438 which, when combined with the predictive representation of the image block 312, 412 in a second summing device 339, 439, produces a preliminary reconstructed image 314, 414. The prediction error decoder 304, 404 also includes an inverse quantizer 346, 446 (Q -1 ), and an inverse transform unit 348, 448 (T -1 ), and the output of the inverse transform unit 348, 448 comprises a reconstructed block. The prediction error decoder may further comprise a block filter capable of filtering the reconstructed block according to the decoded information and filter parameters.

[0039] The entropy encoder 330, 430(E) may receive the output of the prediction error encoder 303, 403 and perform appropriate entropy encoding / variable length encoding on the signal, providing error detection and correction functions. The output of the entropy encoder 330, 430 may be inserted into the bitstream, for example, by a multiplexer 508(M).

[0040] The concept of virtual boundaries is introduced in VVC. In terms of coding dependencies, an image may be divided into different regions by virtual boundaries. For example, 360°, where virtual boundaries are used to define the boundaries of different planes of a 360° image in CMP format, and GDR, where virtual boundaries separate updated and non-updated areas of a GDR image / image during recovery (see U.S. Provisional Patent Application No. 63 / 296,590, "New Gradual Decoding Refresh for ECM," filed by the applicant of the present disclosure). In VVC, virtual boundaries are specified in the SPS and / or image header.

[0041] VVC has three in-loop filters: deblocking, SAO, and ALF. ECM improves the in-loop filters with new features, including bilateral (JVET-F0034, JVET-V0094), BIF for saturation (JVET-X0067), CCSAO (JVET-V0153, JVET-Y0106), CCALF (JVET-X0045), and alternative band classifier for ALF (JVET-X0070).

[0042] In-loop filtering of a current pixel often requires the use of coding information from neighboring pixels, so filtering on one side of a virtual boundary may involve the use of coding information on the other side of the virtual boundary.

[0043] In some applications, in-loop filtering may not be allowed to cross a virtual boundary. For example, in GDR, a GDR image / image being restored may be divided into an updated area and a non-updated area by a virtual boundary. Referring to FIG. 5 , to prevent leakage, the updated area 510 cannot use any information from the non-updated area 530 because there is no guarantee that the non-updated area 530 will be correctly decoded by the decoder. Incorrectly decoded coding information may contaminate the updated area 510, potentially causing leakage or inconsistency between the encoder and decoder at the image at the restoration point and subsequent images. Therefore, for a GDR image / image being restored, in-loop filtering cannot cross the virtual boundary 520, as indicated by arrow 540, from the updated area 510 to the non-updated area 530.

[0044] On the other hand, it may be perfectly acceptable for in-loop filtering to cross the virtual boundary. For example, as shown in Figure 6, in the same GDR example, the non-updated area 630 can use information from the updated area 610. Thus, for the GDR image / image being restored, in-loop filtering can cross the virtual boundary 620 from the non-updated area 630 to the updated area 610, as indicated by arrow 640.

[0045] In the current design of VVC and ECM, the in-loop filtering cannot cross the virtual boundary.

[0046] U.S. Provisional Patent Application No. 63 / 362,243, "In-Loop Filtering at Virtual Boundaries," filed by the applicant of the present disclosure, proposed several possible options for in-loop filtering at virtual boundaries, including asymmetric in-loop filtering at virtual boundaries, where in-loop filtering cannot cross the virtual boundary from one side to the other side of the virtual boundary, but can cross from the other side to one side.

[0047] Specifically, in-loop filtering on one side of the virtual boundary cannot use information on the other side of the virtual boundary, but in-loop filtering on the other side of the virtual boundary can use information on one side. If in-loop filtering of a pixel on one side of the virtual boundary requires the use of any information on the other side (e.g., pixel, coding mode, QP, etc.), in-loop filtering is either not performed for this pixel, or is still performed for this pixel by filling in the information on the other side.

[0048] In asymmetric in-loop filtering with a virtual boundary, the in-loop filtering on one side cannot use information from the other side, but the in-loop filtering on the other side is allowed to use information from one side.

[0049] In-loop filtering of pixels on one side may not be performed successfully if the in-loop filtering of pixels requires the use of coding information on the other side.

[0050] In general, in-loop filtering of pixels on the other side can be performed successfully because the in-loop filtering of pixels is allowed to use coding information from both one side and the other side. However, the other side may choose not to use coding information from one side, in which case in-loop filtering of pixels on the other side may not be performed successfully if the in-loop filtering of pixels requires the use of coding information from one side.

[0051] Since the coding information of one side is available on the other side, an offset based on the in-loop filtering of one side may be added to the output of the in-loop filtering of the other side.

[0052] A virtual border is a line used to separate an image or part of an image into two areas: a first area and a second area.

[0053] The virtual border can be vertical or horizontal. In VVC and ECM, the virtual border syntax is included in the SPS and / or image header. In one embodiment, including asymmetric operation on the virtual border, a first area is not allowed to use any information from a second area, but the second area can use information from the first area.

[0054] In one embodiment, in the GDR image / image being restored, the first area is a clean (updated) area and the second area is a dirty (not updated) area. The clean (updated) area cannot use any information from the dirty (not updated) area, but the dirty (not updated) area can use information from the clean (updated) area. In-loop filtering of a pixel may involve using coding information of neighboring pixels.

[0055] If the in-loop filtering of the pixels in the first area requires the use of coding information of the second area (e.g., pixel, coding mode, reference image, MV, QP, etc.), the in-loop filtering of the pixels may not be performed successfully. The actual in-loop filtering for the pixels may take one of two possible options: option 1, in which no in-loop filtering is performed for the pixels in the first area, or option 2, in which in-loop filtering for the pixels in the first area is still performed, but the coding information of the second area is derived from the first area or set to a default value when needed.

[0056] One embodiment related to option 2 is that if in-loop filtering of pixels in a first area requires the use of pixels in a second area, the pixels in the second area are filled in from pixels in the first area.

[0057] Another embodiment related to option 2 is that if in-loop filtering of pixels in a first area requires the use of pixels in a second area, the pixels in the second area are replaced with pixels extrapolated from the first area.

[0058] Let normal in-loop filtering of a pixel be the ideal in-loop filtering of the pixel that uses all the necessary information, and actual in-loop filtering of the pixel be the practical in-loop filtering of the pixel, whether or not it uses all the necessary information.

[0059] Actual in-loop filtering of a pixel with either option 1 or 2 will produce an output that may differ from normal in-loop filtering of the pixel, which can use the coding information of both the first and second areas.

[0060] Since in-loop filtering for pixels in the second area is allowed to use the coding information of both the first area and the second area, in-loop filtering for pixels in the second area can usually be performed successfully.

[0061] p i,j are the pixels in the first area,

number

number

number

number

[0062] In the virtual boundary, pixel p in the first area i,j The actual in-loop filtering of pixel p i,j If the in-loop filtering of requires the use of coding information of the second area, it may not be equal to the normal in-loop filtering of this pixel, i.e.

[0063]

number

[0064] On the other hand, in the virtual boundary, pixel q in the second area i,jThe actual in-loop filtering of pixel q i,j Since the actual in-loop filtering of can use the coding information of both the first and second areas, it is usually equivalent to the normal in-loop filtering of this pixel, i.e.,

[0065]

number

[0066] To compensate for the unbalanced in-loop filtering at the virtual boundary, the in-loop filtering of the second area may compensate for the difference between the normal in-loop filtering of the first area and the actual in-loop filtering. Note that it is feasible to use the first area to offset the second area because the second area can use the coding information of the first area.

[0067] Specifically, pixel p in the first area i,j The normal in-loop filtering and the actual in-loop filtering of are different, i.e.,

number

number

number

number

number

[0068] In one embodiment, the second area may choose not to use the coding information of the first area. In that case, if in-loop filtering of pixels in the second area requires the use of coding information of the first area, the in-loop filtering of the pixels may not be performed successfully. As with the first area, the actual in-loop filtering for the pixels may choose one of two possible options: option 1, in which in-loop filtering for pixels in the second area is not performed, or option 2, in which in-loop filtering for pixels in the second area is still performed, but the coding information of the first area is derived from the second area or set to a default value when needed.

[0069] One embodiment related to option 2 above is that pixels in the first area are filled from pixels in the second area if in-loop filtering of pixels in the second area requires the use of pixels in the first area.

[0070] Another embodiment related to option 2 above is that when in-loop filtering of pixels in the second area requires the use of pixels in the first area, the pixels in the first area are replaced with pixels extrapolated from the second area.

[0071] Pixel p in the first area i,j The difference between the target in-loop filtering of q and the actual in-loop filtering of q is expressed as m,n This may be used to offset the output of the in-loop filtering of

[0072]

number

number

number

number

number

[0073] In one embodiment, if the first area and the second area select the same option for in-loop filtering of pixels around the virtual boundary, i.e., either not perform in-loop filtering or perform in-loop filtering using embedding, the in-loop filtering of the first area and the second area may be considered balanced. No compensation may be required on either side of the virtual boundary.

[0074] One embodiment relates to a deblocking filter in VVC and ECM: Deblocking filtering is applied to block boundaries (horizontally or vertically), including the pixels on either side of the block boundary.

[0075] Assume that a virtual boundary separates an image or portion of an image into a first area and a second area, where the first area is not allowed to use the coded information in the second area, but the second area is allowed to use the coded information in the first area.

[0076] When block boundaries are aligned with virtual boundaries, deblocking filtering for pixels in the first area that are at most n pixel locations away from the virtual boundary (e.g., in current designs of VVC and ECM, 1 for a weak saturation filter, 2 for a weak luma filter, 3 for a strong luma and saturation filter, and 3, 5, 7 for a bilinear (long) luma filter) requires the use of coding information (e.g., pixel, coding mode, QP, etc.) in the second area.

[0077] Since the first area is not allowed to use the coding information in the second area, deblocking filtering is disabled for those pixels in the first area that are at most n pixel positions away from the virtual boundary. Figure 7 shows an example where an updated area (first area) 7010 of a GDR image / image being restored is not allowed to use the coding information of a non-updated area (second area) 7030. Deblocking (e.g., a strong filter) 7040 is performed on pixels p in the updated area 7010 that are adjacent to the virtual boundary 7020. i ,is disabled for i=0,1,2.

[0078] Alternatively, deblocking filtering 7040 is still applied to those pixels in the first area that are at most n pixel positions away from the virtual boundary 7020, but when needed, the coding information in the second area is derived from the first area or set to a default value. For example, in Figure 7, deblocking (e.g., a strong filter) is applied to pixels p in the updated area 7010 that are adjacent to the virtual boundary 7020. i ,i=0,1,2 (overall 7040) still applies, but q in the unupdated area 7030 i, i=0, 1, 2 are derived from the updated area 7010. For example, q i ,i=0,1,2 is p0 or p i , i=0,1,2, the mean or median of

[0079] Deblocking for pixels on the second area can be performed normally, permitted to use the coding information of both the first area 7010 and the second area 7030.

[0080] Pixel p in the first area 7010 i If the actual deblocking filtering of q differs from the normal deblocking filtering, this difference is calculated as follows: i may be offset from

number

number

number

number

number

number

[0081] One possible embodiment may be as follows. For i=0,...,sq-1,

number

[0082] A simple implementation could be as follows.

number

[0083] corresponding pixel p i and q i are the mirrored pixels in the first area 7010 and the second area 7030 before deblocking relative to the block boundary or virtual boundary 7020, as shown in FIG.

[0084] If the second area 7030 chooses not to use the coding information of the first area 7010, deblocking filtering is not applied to pixels in the second area 7030 that are at most n (e.g., in the current designs of VVC and ECM, 1 for a weak saturation filter, 2 for a weak luma filter, 3 for a strong luma and saturation filter, and 3, 5, 7 for a bilinear (long) luma filter) pixel positions away from the virtual boundary. Figure 7 can show an example where the non-updated area (second area) 7030 of the GDR image / image being restored chooses not to use the coding information of the updated area (first area) 7010. Deblocking (e.g., a strong filter) is applied to pixels q in the non-updated area 7030 that are adjacent to the virtual boundary 7020. i ,is disabled for i=0,1,2.

[0085] Alternatively, deblocking filtering is still applied to those pixels 7050 in the second area 7030 that are at most n pixel positions away from the virtual boundary 7020, but the coding information in the first area 7010 is derived from the second area 7030 or set to a default value. For example, in FIG. 7 , deblocking (e.g., a strong filter) is applied to pixels q in the unupdated area 7030 that are adjacent to the virtual boundary 7020. i , i=0,1,2 (overall 7050), but p in the updated area 7010 i , i=0, 1, 2 are derived from the non-updated area 7030. For example, p i ,i=0,1,2 is q0 or q i , i=0,1,2, the mean or median of

[0086] One embodiment relates to an SAO edge offset filter. In VVC, the SAO has two parts: band offset and edge offset. Each CTU can choose to use either band offset or edge offset. The choice of band offset or edge offset for each CTU is signaled. If edge offset is used in a CTU, a set of parameters (edge ​​class as shown in FIG. 8 and offsets for four edge categories as shown in FIG. 9) is signaled.

[0087] Referring to Figure 8, examples of four edge classes are shown. In example 810, pixels a and b are horizontally adjacent to pixel c. In example 820, pixels a and b are vertically adjacent to pixel c. In example 830, pixels a and b are adjacent to pixel c along a gradient from upper left to lower right. In example 840, pixels a and b are adjacent to pixel c along a gradient from lower left to upper right.

[0088] Referring to FIG. 9, four example edge categories are shown. In category 1, 910, the value of pixel c is less than the values ​​of pixels a and b. In category 2, 920, the values ​​of pixels c and b may be similar, while the value of pixel a may be greater than the values ​​of pixels c and b. Alternatively, the values ​​of pixels a and c may be similar, while the value of pixel b may be greater than the values ​​of pixels a and c. In category 3, 930, the values ​​of pixels a and c may be similar, while the value of pixel b may be less than the values ​​of pixels a and c. Alternatively, the values ​​of pixels c and b may be similar, while the value of pixel a may be less than the values ​​of pixels c and b. In category 4, 940, the value of pixel c may be greater than the values ​​of pixels a and b.

[0089] As can be seen in Figures 8 and 9, classifying the edge of a pixel involves the use of neighboring pixels.

[0090] Assume that a virtual boundary separates an image or portion of an image into a first area and a second area, where the first area is not allowed to use the coded information in the second area, but the second area is allowed to use the coded information in the first area.

[0091] SAO edge offsets for pixels in a first area immediately adjacent to the virtual boundary may require the use of coded information (e.g., pixels) in a second area, as shown in FIG. 8.

[0092] Because the first area is not permitted to use the coding information of the second area, the SAO edge offset is not applied to those pixels in the first area that are immediately adjacent to the virtual boundary. Figure 10A shows an example in which an updated area (first area) 1010 of a GDR image / image being restored is not permitted to use the coding information of an unupdated area (second area) 1030. The SAO edge offset with a diagonal class direction 1040 is disabled for pixel p0 in the updated area 1010 that is immediately adjacent to the virtual boundary 1020.

[0093] Alternatively, the SAO edge offset (e.g., 1040) is still applied to pixels in the first area 1010 immediately adjacent to the virtual boundary 1020, but when needed, the coding information (e.g., pixels) in the second area 1030 is derived from the first area 1010 or set to a default value. For example, in FIG. 10A , the SAO edge offset is still applied to pixel p0 in the updated area 1010 immediately adjacent to the virtual boundary 1020, but the associated pixel q0 on the non-updated area 1030 is filled in from the updated area 1010 (or set to a default value, e.g., 2 BD-1 where BD is the bit depth).

[0094] SAO edge offsets for pixels in the second area 1030 adjacent to the virtual boundary 1020 can be performed normally, permitted to use the coding information of both the first area 1010 and the second area 1030.

[0095] If the actual SAO edge offset filtering of pixel p0 in the first area 1010 differs from the normal SAO edge offset filtering, this difference may be offset from the corresponding pixel q0 as follows:

number

number

number

number

number

[0096] The corresponding pixels p0 and q0 are mirror pixels to the junction of the virtual boundary and SAO edge offset class direction line along the selected SAO edge class direction line 1040, as shown in FIG. 10A.

[0097] If the second area chooses not to use the coding information of the first area, the SAO edge offset is not applied to pixels in the second area adjacent to the virtual boundary. Figure 10B shows an example in which an unupdated area (second area) 1070 of the GDR image / image being restored chooses not to use the coding information of the updated area (first area) 1060. The SAO edge offset is not applied to pixel q0 in the unupdated area 1070 adjacent to the virtual boundary 1080.

[0098] Alternatively, the SAO edge offset is still applied to those pixels in the second area 1070 adjacent to the virtual boundary 1080, but the coding information in the first area 1060 is derived from the second area 1070 or set to a default value when needed. For example, in FIG. 10B, the SAO edge offset is still applied to pixel q0 in the un-updated area 1070 adjacent to the virtual boundary 1080, but the associated pixel p0 in the updated area 1060 is filled in from the un-updated area 1070. An edge class direction line 1090 is shown in FIG. 10B.

[0099] One embodiment relates to bilateral filters (BIFs) for luma and chroma. ECM enhances the in-loop filters of VVC by adding new filter functions, among them the bilateral filter. As shown in FIG. 11, the BIF 1130 runs in parallel with the SAO 1120 and CCSAO processes 1140. The BIF (1130), SAO (1120), and CCSAO (1140) use the same samples generated by the deblocking filter (1110) as input and generate three offsets per sample in parallel. These three offsets are then added to the input sample (using operation 1150) to obtain a sum, which is then clipped to form the final output sample value (1160) before proceeding to the ALF. The BIF chroma provides an on / off control mechanism at the CTU level and slice level.

[0100] The bilateral filter is a 5x5 diamond for both luma and chroma, as shown in Figure 12A, and the bilateral filter is applied to pixels adjacent to the virtual boundary.

[0101] Assume that a virtual boundary separates an image or portion of an image into a first area and a second area, where the first area is not allowed to use the coded information in the second area, but the second area is allowed to use the coded information in the first area.

[0102] BIF filtering for pixels in a first area that are at most n (e.g., 2 in the current design of the BIF) pixel positions away from the virtual boundary requires the use of coded information (e.g., pixels) in a second area.

[0103] Since the first area is not allowed to use the coding information of the second area, BIF filtering may be disabled for those pixels in the first area that are at most n (e.g., 2 in the current design of BIF) pixel positions away from the virtual boundary. Figure 12A shows an example where an updated area (first area) 1210 of a GDR image / image being restored is not allowed to use the coding information of a non-updated area (second area) 1230. BIF filtering is disabled for pixels p in the updated area 1210 that are adjacent to the virtual boundary 1220. 0,0 is not executed against

[0104] Alternatively, BIF filtering is still performed on those pixels 1240 in the first area that are at most n (e.g., 2 in the current design of the BIF) pixel locations away from the virtual boundary, but when needed, the coding information on the second area is derived from the first area or set to a default value. For example, in Figure 12A, BIF filtering is performed on pixel p in the updated area 1210 that is adjacent to the virtual boundary 1220. 0,0 q in Area 1230 that still applies to but has not been updated i,0 The relevant pixels, including i=0,1, are filled from the updated area (or a default value, e.g., 2 BD-1 where BD is the bit depth).

[0105] BIF filtering for pixels 1250 on the second area 1230 can be performed normally, permitted to use the coding information of both the first area 1210 and the second area 1230 .

[0106] Pixel p in the first area 1210 i,j If the actual BIF filtering of q differs from the normal deblocking filtering, then this difference is applied to the corresponding pixel q as follows: m,n may be offset from

number

number

number

number

number

[0107] corresponding pixel p i,j and q i,j are the mirrored pixels in the first area 1210 and the second area 1230 before the BIF relative to the virtual boundary 1220, as shown in FIG. 12A.

[0108] If the second area chooses not to use the coding information of the first area, BIF filtering is not applied to pixels in the second area that are at most n (e.g., 2 in the current design of the BIF) pixel locations away from the virtual boundary. Figure 12B shows an example where an unupdated area (second area 1280) of the GDR image / image being restored chooses not to use the coding information of the updated area (first area) 1260. The BIF applies the coding information of pixels q in the unupdated area 1280 that are adjacent to the virtual boundary 1270. 0,0 is disabled for

[0109] Alternatively, BIF filtering is still applied to those pixels 1295 in the second area 1280 that are at most n pixel positions away from the virtual boundary 1270, but when needed, the coding information in the first area 1260 is derived from the second area 1280 or set to a default value. For example, in Figure 12B, the BIF is applied to q pixels 1295 in the unupdated area 1280 that are adjacent to the virtual boundary 1270. 0,0 still applies to the pixels containing p (overall 1295), but i,0 ,i=0,1, the relevant pixels (1290 in total) are filled from the non-updated area 1280.

[0110] One embodiment relates to a CCSAO filter. Cross-component sample adaptive offset (CCSAO) is used to improve reconstructed samples. Similar to SAO, CCSAO classifies reconstructed samples into different categories, derives one offset per category, and adds this offset to the reconstructed samples in that category. However, unlike SAO (1340, 1350, 1360), which uses a single luma / chroma component (1310, 1320, 1330) of the current sample as input, CCSAO (1370, 1380, 1390) utilizes all three components (1310, 1320, 1330) to classify the current sample into different categories, as shown in FIG. 13 . To facilitate parallel processing, output samples from the deblocking filter are used as input for CCSAO.

[0111] The output of CCSAO Y 1370 is combined (e.g., added or subtracted) with the output of SAO Y 1340 using operation 1391 to generate Y 1394. The output of CCSAO U 1380 is combined (e.g., added or subtracted) with the output of SAO U 1350 using operation 1392 to generate U 1395. The output of CCSAO V 1390 is combined (e.g., added or subtracted) with the output of SAO V 1350 using operation 1393 to generate V 1396.

[0112] In CCSAO, either a band offset (BO) classifier or an edge offset (EO) classifier is used to improve the quality of the reconstructed samples. CCSAO may be applied to both luma and chroma components.

[0113] In CCSAO BO, for a given luma / chroma sample, three candidate samples are selected to classify the given sample into different categories: one collocated Y sample, one collocated U sample, and one collocated V sample. Then, the sample values ​​of these three selected samples are sorted into three different bands, and a combined index represents the category of the given sample. An offset is signaled and added to the reconstructed sample that falls into that category.

[0114] As shown in FIG. 14, the collocated luma sample 1410 can be selected from nine candidate locations (1405), while the collocated chroma sample locations (1420, 1430) are fixed.

[0115] Assume that a virtual boundary separates an image or portion of an image into a first area and a second area, where the first area is not allowed to use the coded information in the second area, but the second area is allowed to use the coded information in the first area.

[0116] A CCSAO for pixels in a first area immediately adjacent to the virtual boundary may require the use of coded information (eg, pixels) in a second area.

[0117] Because the first area is not permitted to use the coding information of the second area, CCSAO is not applied to those pixels in the first area that are immediately adjacent to the virtual boundary. Figure 15A shows an example in which an updated area (first area) 1510 of a GDR image / image being restored is not permitted to use the coding information of an unupdated area (second area) 1530. CCSAO is skipped for pixel p0 in the updated area 1510 that is immediately adjacent to the virtual boundary 1520. Figure 15A shows collocated saturation 1540.

[0118] Alternatively, the CCSAO is still applied to those pixels in the first area immediately adjacent to the virtual boundary, but the coding information in the second area is derived from the first area or set to a default value when needed. For example, in FIG. 15A, the CCSAO is still applied to pixel p0 in the updated area 1510 adjacent to the virtual boundary 1520, but the associated pixel q0 in the non-updated area 1530 is filled in from the updated area 1510 (or set to a default value, e.g., 2 BD-1 where BD is the bit depth).

[0119] The CCSAO for the pixels on the second area 1530 is allowed to use the coding information of the first area 1510 and can be performed normally.

[0120] If the actual CCSAO filtering of pixel p0 in the first area 1510 differs from the normal SAO edge offset filtering, this difference may be offset from the corresponding pixel q0 as follows:

number

number

number

number

number

[0121] The corresponding pixels p0 and q0 are mirror pixels in the first area 1510 and the second area 1530 in front of the CCSAO BO relative to the virtual boundary, as shown in FIG. 15A.

[0122] If the second area chooses not to use the coding information of the first area, CCSAO BO is not applied to pixels in the second area adjacent to the virtual boundary. Figure 15B shows an example in which an unupdated area (second area) 1580 of a GDR image / image being restored chooses not to use the coding information of the updated area (first area) 1560. CCSAO BO is not applied to pixel q0 in the unupdated area 1580 adjacent to the virtual boundary 1570.

[0123] Alternatively, CCSAO BO is still applied to those pixels in a second area 1580 adjacent to the virtual boundary, but the coding information in the first area 1560 is derived from the second area 1580 or set to a default value when needed. For example, in FIG. 15B , CCSAO BO is still applied to pixel q0 in an unupdated area 1580 adjacent to the virtual boundary 1570, but the associated pixel p0 in the updated area 1560 is filled in from the unupdated area 1580. FIG. 15B shows collocated saturation 1590.

[0124] One embodiment relates to ALF filters. In VVC, ALF filters are diamonds with a size of 7x7 for luma and 5x5 for chroma. ECM expands the size of the ALF for luma and chroma to 9x9, 7x7, and 5x5. Figure 16A shows an example of a 9x9 diamond ALF filter for pixels adjacent to a virtual boundary 1620. In addition, ECM adds an alternative band classifier for classification in ALF (ABC-ALF), which is a 13x13 diamond filter for classifying each 2x2 luma block of ALF.

[0125] Assume that a virtual boundary separates an image or portion of an image into a first area and a second area, where the first area is not allowed to use the coded information in the second area, but the second area is allowed to use the coded information in the first area.

[0126] ALF filtering for pixels in a first area that are up to n (e.g., 3 for luma and 2 for chroma ALF in the current design of VVC, 2, 3, 4 for luma and chroma ALF, and 6 for ABC-ALF in the current design of ECM) pixel locations away from the virtual boundary requires the use of coding information (e.g., pixels) in a second area.

[0127] Since the first area is not allowed to use the coding information of the second area, ALF filtering may be disabled for those pixels in the first area that are up to n positions away from the virtual boundary. Figure 16A shows an example where an updated area (first area) 1610 of a GDR image / image being restored is not allowed to use the coding information of a non-updated area (second area) 1630. The ALF filters pixels p in the updated area 1610 that are immediately adjacent to the virtual boundary 1620. 0,0 is not executed against

[0128] Alternatively, the ALF is still applied to pixels (1640) in the first area 1610 that are up to n positions away from the virtual boundary 1620, but when needed, the coding information on the second area 1630 is derived from the first area 1610 or set to a default value. For example, in Figure 16A, the ALF is applied to pixels p 0,0 q in area 1630 that is still executed but not updated i,0 , i=0, 1, 2, the associated pixels 1650 are filled from the updated area 1610 (or a default value, e.g., 2 BD-1 where BD is the bit depth).

[0129] ALF filtering for pixels on the second area can be performed normally, permitted to use the coding information of both the first and second areas.

[0130] Pixel p in the first area 1610 i,j If the actual ALF filtering of q differs from the normal deblocking filtering, this difference can be calculated by m,n may be offset from

number

number

number

number

number

[0131] corresponding pixel p i,j and q i,j are the mirrored pixels in the first area 1610 and the second area 1630 before the ALF relative to the virtual boundary 1620, as shown in FIG. 16A.

[0132] If the second area chooses not to use the coding information of the first area, ALF is not applied to pixels in the second area that are at most n (e.g., 3 for luma and 2 for chroma ALF in the current design of VVC, 2, 3, 4 for luma and chroma ALF, and 6 for ABC-ALF in the current design of ECM) pixel locations away from the virtual boundary. Figure 16B shows an example where an unupdated area (second area) 1680 of a GDR image / image being restored chooses not to use the coding information of the updated area (first area) 1660. The ALF is applied to pixels q in the unupdated area 1680 that are adjacent to the virtual boundary 1670. 0,0 does not apply to.

[0133] Alternatively, ALF is still applied to those pixels 1695 in the second area 1680 adjacent to the virtual boundary 1670, but when required, the coded information in the first area 1660 is derived from the second area 1680 or set to a default value. For example, in FIG. 16B, ALF is still applied to pixel q in the non-updated area 1680 adjacent to the virtual boundary 1670 0,0 but is still applied to p in the updated area 1660 i,0 , and the related pixels 1690 including i = 0, 1, 2 are filled from the non-updated area 1680.

[0134] One embodiment relates to the CCALF filter. As shown in FIG. 17, the CCALF process 1720 uses a linear filter to filter the luminance sample values and generate a chroma sample residual correction (1770). First, an 8-tap filter was designed for the CCALF process in VVC. Recently, in the CCALF process (1800) of ECM, the large 25-tap filter shown in FIG. 18 is used. For a given slice, the encoder can collect and analyze the slice statistics and signal up to 16 filters via APS.

[0135] Referring to FIG. 17, a basic example of CCALF is shown. In CTU(Y) 1710, as shown at 1730, CCALF(Cb) may be applied to a set of pixels (1720). This may be regarded as a linear filtering of the luminance sample values. In CTU(Cb) 1740, ALF chroma may be applied to some of the pixels (1750). This may be regarded as filtering of the chroma samples. The outputs of 1720 and 1750 are added (1760) (or alternatively, combined in some other way, e.g., subtracted by operation 1760) and output as CTB’(Cb) 1770.

[0136] Assume that a virtual boundary separates an image or portion of an image into a first area and a second area, where the first area is not allowed to use the coded information in the second area, but the second area is allowed to use the coded information in the first area.

[0137] CCALF filtering for pixels in a first area that are at most n (e.g., 1 for VVC or 4 for ECM) pixel positions away from the virtual boundary requires the use of coded information (e.g., pixels) in a second area.

[0138] Because the first area is not allowed to use the coding information of the second area, CCALF filtering may be disabled for those pixels in the first area that are up to n pixel positions away from the virtual boundary. Figure 19A shows an example in which an updated area 1910 (first area) of a GDR image / image being restored is not allowed to use the coding information of an unupdated area 1930 (second area). CCALF is skipped for chroma pixels 1950 in the updated area 1910 that are immediately adjacent to the virtual boundary 1920.

[0139] Alternatively, CCALF is still applied to those pixels in a first area that is up to n pixel locations away from the virtual boundary, but the coding information in a second area is derived from the first area or set to a default value when needed. For example, in Figure 19A, CCALF is still applied to saturation pixels 1950 in updated area 1910 adjacent to virtual boundary 1920, but is not applied to q pixels on non-updated area 1930. i,j ,i=0,1,2,3 and j=0,1 are filled from the updated area 1910 (or a default value, e.g., 2 BD-1 where BD is the bit depth).

[0140] CCALF for pixels on the second area can be performed normally, permitted to use information from the first area.

[0141] Pixel p in the first area 1910 i,j If the actual CCALF filtering of q differs from the normal deblocking filtering, this difference can be calculated by i,j may be offset from

number

number

number

number

number

[0142] corresponding pixel p i,j and q i,j are the mirrored pixels in the first area 1910 and the second area 1930 before the CCALF relative to the virtual boundary 1920, as shown in FIG. 19A.

[0143] If the second area chooses not to use the coding information of the first area, CCALF is not applied to pixels in the second area that are up to n (e.g., 1 for VVC or 4 for ECM) pixel locations away from the virtual boundary. Figure 19B shows an example in which an unupdated area (second area) 1980 of a GDR image / image being restored chooses not to use the coding information of the updated area (first area) 1960. CCALF is skipped for collocated chroma pixels 1990 in the unupdated area 1980 that are adjacent to the virtual boundary 1970.

[0144] Alternatively, CCALF is still applied to those pixels in a second area 1980 adjacent to the virtual boundary 1970, but the coding information in the first area 1960 is derived from the second area 1980 or set to a default value when needed. For example, in Figure 19B, CCALF is still applied to the collocated chroma pixels 1990 in the unupdated area 1980 adjacent to the virtual boundary 1970, but the coding information in the first area 1960 is derived from the second area 1980 or set to a default value when needed. i,j , the associated luminance pixels including i=0,1,2,3 and j=0,1 are filled from the non-updated area 1980 .

[0145] 20 is a block diagram 700 of a device 710 suitable for implementing exemplary embodiments. One non-limiting example of the device 710 is a wireless, typically mobile, device capable of accessing a wireless network. The device 710 includes one or more processors 720, one or more memories 725, one or more transceivers 730, and one or more network (N / W) interfaces (I / F) 761 interconnected via one or more buses 727. Each of the one or more transceivers 730 includes a receiver Rx 732 and a transmitter Tx 733. The one or more buses 727 may be an address bus, a data bus, or a control bus and may include any interconnection mechanism, such as a series of wires on a motherboard or integrated circuit, optical fiber, or other optical communication equipment.

[0146] The device 710 may communicate via a wired, wireless, or both interfaces. For wireless communication, one or more transceivers 730 are connected to one or more antennas 728. One or more memories 725 contain computer program code 723. The N / WI / F 761 communicates via one or more wired links 762.

[0147] The device 710 includes a control module 740 having one or both of portions 740-1 and / or 740-2, which may be implemented in multiple ways, including a reference 790 including a codec, either an encoder 780 or a decoder 782, or both 780 / 782. For ease of reference, the reference 790 is referred to herein as a codec. The control module 740 may be implemented as control module 740-1 in hardware, such as implemented as part of one or more processors 720. The control module 740-1 may also be implemented as an integrated circuit or by other hardware, such as a programmable gate array. In another example, the control module 740 may be implemented as control module 740-2, implemented as computer program code 723 and executed by one or more processors 720. For example, the one or more memories 725 and the computer program code 723, together with the one or more processors 720, may be configured to cause the user equipment 710 to perform one or more of the operations as described herein. The codec 790 may be similarly implemented as a codec 790-1 as part of the control module 740-1, or as a codec 790-2 as part of the control module 740-2, or both.

[0148] The computer-readable memory 725 may be of any type suitable for the local technology environment and may be implemented using any appropriate data storage technology, such as semiconductor-based memory devices, flash memory, firmware, magnetic memory devices and systems, optical memory devices and systems, fixed and removable memory, etc. The computer-readable memory 725 may be a means for performing storage functions. The one or more computer-readable memories 725 may be non-transitory, transient, volatile (e.g., random access memory (RAM)), or non-volatile (e.g., read-only memory (ROM)). The one or more computer-readable memories 725 may include a database for storing data.

[0149] The processor 720 may be of any type suitable for the local technology environment and may include, by way of non-limiting example, one or more of a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture. The processor 720 may be a means for performing functions such as controlling the device 710 and other functions as described herein.

[0150] In general, various embodiments of device 710 include cellular telephones (such as smartphones, mobile phones, cellular phones, voice over Internet Protocol (IP) (VoIP) phones, and / or wireless local loop phones), tablets, portable computers, indoor audio equipment, immersive audio equipment, e.g., vehicle or in-vehicle devices for wireless vehicle-to-everything (V2X) communications, image capture devices such as digital cameras, gaming devices, music storage and playback devices, Internet appliances (including Internet of Things (IoT) devices), e.g., IoT devices having sensors and / or actuators for automation applications, and portable units or terminals incorporating a combination of such functionality, laptops, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), Universal Serial Bus (USB) dongles, smart devices, wireless customer-premises equipment (CPE), and the like. The devices 710 may include, but are not limited to, wireless devices, Internet of Things (IoT) devices, watches or other wearables, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain contexts), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. That is, the device 710 can be any device that may have the capability for wireless or wired communication.

[0151] Thus, the device 710 comprises a processor 720 and at least one memory 725 containing computer program code 723, the at least one memory 725 and the computer program code 723, together with the at least one processor 720, configured to cause the device 710 to implement an asymmetric in-loop filter 790 with a virtual boundary based on embodiments described herein. The device 710 may include a display or I / O 770 that can be used to display content during ML / task / machine / NN processing or rendering. The display or I / O 770 may be configured to receive input from a user using a keypad, a touch screen, a touch area, a microphone, biometrics, one or more sensors, etc. The device 710 may comprise standard known components, such as amplifiers, filters, frequency converters, and (de)modulators.

[0152] The computer program code 723 may include object-oriented software and may implement the filtering described throughout this disclosure. The device 710 need not include each of the features mentioned, and may include other features. The device 710 may be an embodiment of the device shown in Figure 1, 2, 3, or 4, or may include any combination thereof.

[0153] FIG. 21 illustrates an example method 2100 for implementing an asymmetric in-loop filter with a virtual boundary, based on embodiments described herein. At 2110, the method includes determining a virtual boundary separating an image or a portion of an image into a first area and a second area. At 2120, the method includes determining, if encoding information of the second area needs to be used to perform filtering of at least one pixel of the first area, that encoding information of the second area is derived from the first area or that encoding information of the second area is set to at least one value to perform filtering of at least one pixel of the first area, or determining, if encoding information of the second area needs to be used to perform filtering of at least one pixel of the first area, not to perform filtering of at least one pixel of the first area. Method 2100 may be performed by an encoder, decoder, or codec, or any of the devices shown in FIG. 1, 2, 3, 4, or 20.

[0154] References to "computer," "processor," etc. should be understood to encompass not only computers having a variety of architectures, such as single-processor / multiprocessor and serial / parallel architectures, but also specialized circuitry, such as field-programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices, and other processing circuitry. References to computer programs, instructions, code, etc. should be understood to encompass, for example, the programmable content of a hardware device, such as instructions for a processor, or software or firmware for a programmable processor, such as configuration settings for a fixed-function device, gate array, or programmable logic device.

[0155] As used herein, the terms “circuitry,” “circuit,” and variations thereof may refer to either (a) a hardware circuit implementation, such as an implementation in analog and / or digital circuitry, and (b) (where applicable) a combination of (i) a processor or (ii) a portion / software of a processor including a digital signal processor, software, and memory that work together to cause a device to perform various functions, and (c) a circuit, such as a microprocessor or a portion of a microprocessor, that requires software or firmware for operation even when the software or firmware is not physically present. As a further example, as used herein, the term “circuit” also covers simply the implementation of a processor (or processors) or portion of a processor and its (or their) accompanying software and / or firmware. The term “circuit” also covers, for example, a baseband integrated circuit or an application processor integrated circuit for a mobile phone, or a similar integrated circuit in a server, cellular network device, or another network device, if applicable to the particular element. Circuitry and circuit may also be used to mean a function or process used to perform a method.

[0156] The following examples (1-32) are illustrated and provided herein.

[0157] Example 1 The apparatus includes at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform the following: determine a virtual boundary separating an image or a portion of an image into a first area and a second area; and determine, when encoding information of the second area needs to be used to perform filtering of at least one pixel of the first area, that the encoding information of the second area is derived from the first area or the encoding information of the second area is set to at least one value to perform filtering of at least one pixel of the first area; or determine, when encoding information of the second area needs to be used to perform filtering of at least one pixel of the first area, not to perform filtering of at least one pixel of the first area.

[0158] Example 2 2. The apparatus of example 1, wherein the filtering of the at least one pixel of the first area comprises in-loop filtering.

[0159] Example 3 3. The apparatus of example 1 or 2, wherein the first area comprises an updated area and the second area comprises an unupdated area.

[0160] Example 4 The apparatus of any of Examples 1 to 3, wherein the image comprises a progressive decoding update image or an image being restored.

[0161] Example 5 The apparatus of any of Examples 1 to 4, wherein at least one memory and computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform filling pixels in the second area from pixels in the first area in response to pixels in the second area being used to perform filtering of at least one pixel in the first area.

[0162] Example 6 The apparatus of any of Examples 1-5, wherein the at least one memory and computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform, in response to pixels in the second area being used to perform filtering of at least one pixel in the first area, replacing pixels in the second area with pixels extrapolated from the first area.

[0163] Example 7 The apparatus of any of Examples 1 to 6, wherein at least one memory and computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform: determining a first output of filtering at least one pixel of the first area when coding information of the first area and coding information of the second area are available for filtering at least one pixel of the first area; and determining a second output of filtering at least one pixel of the first area when coding information of the second area is not available for filtering at least one pixel of the first area.

[0164] Example 8 The apparatus of example 7, wherein the at least one memory and computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform the following: determine a difference between the first output and the second output; and determine an output of filtering at least one pixel in the second area using, at least in part, the difference or an approximation of the difference.

[0165] Example 9 9. The apparatus of example 8, wherein the coded information of the second area includes an output of filtering at least one pixel of the second area.

[0166] Example 10 10. The apparatus of example 8 or 9, wherein a location of at least one pixel in the second area corresponds to a location of at least one pixel in the first area.

[0167] Example 11 The apparatus of any of Examples 7-10, wherein at least one memory and computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform the following: determine a difference between the first output and the second output; determine an initial output of the filtering of at least one pixel of the second area; and at least partially subtract the difference from the initial output to determine a final output of the filtering of the at least one pixel of the second area, wherein the coding information of the second area includes the final output of the filtering of the at least one pixel of the second area.

[0168] Example 12 12. The apparatus of Example 11, wherein the at least one memory and the computer program code, together with the at least one processor, are configured to cause the apparatus to at least partially subtract the weighted contribution of the difference from the initial output to determine a final output of the filtering of the at least one pixel of the second area.

[0169] Example 13 Weighted contribution 1 / 2 i 13. The apparatus of example 12, wherein i corresponds to an index of a location of the at least one pixel in the first area or the at least one pixel in the second area.

[0170] Example 14 14. The apparatus of any of Examples 1 to 13, wherein at least one memory and computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform the following: determining a target output of the target filtering of at least one pixel of the first area; determining an actual output of the filtering of at least one pixel of the first area when the coding information of the first area or the coding information of the second area is not available to perform the filtering of the at least one pixel of the first area; determining a difference between the target output and the actual output; determining an initial output of the filtering of the at least one pixel of the second area; and determining a final output of the filtering of the at least one pixel of the second area using the initial output at least partially offset by the difference, wherein the coding information of the second area includes the final output of the filtering of the at least one pixel of the second area.

[0171] Example 15 The apparatus of any of Examples 1 to 14, wherein at least one memory and computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform the following in response to determining a common option related to filtering at least one pixel of the first area and filtering at least one pixel of the second area: determining whether to perform filtering of at least one pixel of the first area without using coding information of the second area; and determining whether to perform filtering of at least one pixel of the second area without using coding information of the first area.

[0172] Example 16 16. The apparatus of example 15, wherein the common options include determining not to perform filtering of at least one pixel in the first area and determining not to perform filtering of at least one pixel in the second area.

[0173] Example 17 17. The apparatus of example 15 or 16, wherein common options include: determining to fill the coding information of the second area and perform filtering of at least one pixel of the first area; and determining to fill the coding information of the first area and perform filtering of at least one pixel of the second area.

[0174] Example 18 The apparatus of any of Examples 1 to 17, wherein at least one memory and computer program code, together with at least one processor, are configured to cause the apparatus to at least perform filtering of at least one pixel in the second area using the coding information of the first area and the coding information of the second area.

[0175] Example 19 19. The apparatus of any of examples 1-18, wherein the filtering of at least one pixel of the first area includes at least one of deblocking filtering, sample adaptive offset edge offset filtering, bilateral filtering for luma, bilateral filtering for chroma, inter-component sample adaptive offset filtering, adaptive loop filtering, or inter-component adaptive loop filtering.

[0176] Example 20 The apparatus of any of Examples 1 to 19, wherein at least one memory and computer program code, together with at least one processor, are configured to cause the apparatus to at least perform the following: disable filtering of at least one pixel in the first area up to a plurality of pixel positions from the virtual boundary.

[0177] Example 21 The apparatus of any of Examples 1 to 20, wherein at least one memory and computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform filtering of at least one pixel of the first area up to a plurality of pixel positions from the virtual boundary.

[0178] Example 22 22. The apparatus of any of Examples 1-21, wherein at least one memory and computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform at least one of: setting pixel values ​​of the second area to be equal to pixel values ​​of the first area adjacent to the virtual boundary; setting pixel values ​​of the second area to be equal to an average value of the pixel values ​​of the first area; or setting pixel values ​​of the second area to be equal to a median value of the pixel values ​​of the first area; and wherein the coding information of the second area includes the set pixel values ​​of the second area.

[0179] Example 23 23. The apparatus of any of Examples 1 to 22, wherein at least one memory and computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform: when the coding information of the first area needs to be used to perform filtering of at least one pixel of the second area, the coding information of the first area is derived from the second area or the coding information of the first area is set to at least one value to perform filtering of at least one pixel of the second area; or when the coding information of the first area needs to be used to perform filtering of at least one pixel of the second area, determine not to perform filtering of at least one pixel of the second area.

[0180] Example 24 The apparatus of Example 23, wherein at least one memory and computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform the following: disable filtering of at least one pixel in the second area up to a plurality of pixel positions from the virtual boundary.

[0181] Example 25 The apparatus of example 23 or 24, wherein at least one memory and computer program code, together with the at least one processor, are configured to cause the apparatus to at least perform filtering of at least one pixel of the second area up to a plurality of pixel positions from the virtual boundary.

[0182] Example 26 26. The apparatus of any of Examples 23-25, wherein at least one memory and computer program code, together with at least one processor, are configured to cause the apparatus to at least perform at least one of: setting pixel values ​​of the first area to be equal to pixel values ​​of a second area adjacent to the virtual boundary; setting pixel values ​​of the first area to be equal to an average value of the pixel values ​​of the second area; or setting pixel values ​​of the first area to be equal to a median value of the pixel values ​​of the second area; and wherein the coding information of the first area includes the set pixel values ​​of the first area.

[0183] Example 27 27. The apparatus of any of Examples 23-26, wherein the filtering of at least one pixel in the second area includes at least one of in-loop filtering, deblocking filtering, sample adaptive offset edge offset filtering, bilateral filtering for luma, bilateral filtering for chroma, inter-component sample adaptive offset filtering, adaptive loop filtering, or inter-component adaptive loop filtering.

[0184] Example 28 The apparatus of any of Examples 1 to 27, wherein at least one memory and computer program code, together with at least one processor, are configured to cause the apparatus to at least perform determining at least one value using a bit depth BD.

[0185] Example 29 At least one memory and computer program code, together with at least one processor, provide an apparatus comprising: BD-1 The apparatus of Example 28, configured to at least perform the steps of determining at least one value as:

[0186] Example 30 The method includes determining a virtual boundary separating the image or a portion of the image into a first area and a second area, and determining that, if encoding information of the second area needs to be used to perform filtering of at least one pixel of the first area, the encoding information of the second area is derived from the first area or the encoding information of the second area is set to at least one value to perform filtering of at least one pixel of the first area, or determining not to perform filtering of at least one pixel of the first area if encoding information of the second area needs to be used to perform filtering of at least one pixel of the first area.

[0187] Example 31 The apparatus includes means for determining a virtual boundary separating an image or a portion of an image into a first area and a second area; and means for determining that, when encoding information of the second area needs to be used to perform filtering of at least one pixel of the first area, the encoding information of the second area is derived from the first area or the encoding information of the second area is set to at least one value to perform filtering of at least one pixel of the first area, or means for determining that, when encoding information of the second area needs to be used to perform filtering of at least one pixel of the first area, filtering of at least one pixel of the first area is not performed.

[0188] Example 32 A machine-readable non-transitory program storage device tangibly embodying a program of instructions executable using a machine to perform operations, the operations including: determining a virtual boundary separating an image or a portion of an image into a first area and a second area; and determining, if coding information of the second area needs to be used to perform filtering of at least one pixel of the first area, that the coding information of the second area is derived from the first area or is set to at least one value to perform filtering of at least one pixel of the first area; or determining, if coding information of the second area needs to be used to perform filtering of at least one pixel of the first area, not to perform filtering of at least one pixel of the first area.

[0189] In the diagram, arrows between individual blocks represent the operational connections between the blocks and the direction of data flow at those connections.

[0190] It should be understood that the foregoing description is illustrative only. Various alternatives and modifications may be devised by those skilled in the art. For example, the features recited in the various dependent claims may be combined with each other in any suitable combination. In addition, features from the various embodiments described above may be selectively combined into new embodiments. Accordingly, this description is intended to embrace all such alternatives, modifications, and variations that fall within the scope of the appended claims.

[0191] The following acronyms and abbreviations that may appear in the present specification and / or drawings are defined as follows: Acronyms and abbreviations can be added to each other and / or to have other characters added (e.g., a hyphen (-)). 3GPP (3rd generation partnership project) 4G (fourth generation) The fourth generation of broadband cellular network technology 5G (fifth generation) 5th generation cellular network technology 802.x A family of IEEE standards for local and metropolitan area networks ABC (alternative band classifier) ALF (adaptive loop filter) APS (adaptation parameter set) ASIC Application Specific Integrated Circuit BD (bit depth) BIF Bilateral Filter BIF Chroma Bilateral filter for chroma BIF Luminance Bilateral filter for luminance BO Band Offset Cb Blue chrominance component CCALF (cross-component ALF) or CC-ALF (cross-component ALF) CCSAO (cross-component SAO) CDMA Code Division Multiple Access CMP (cube-map projection) CPE Customer Premises Equipment Cr red chrominance component CTB (coding tree block) CTU (coding tree unit) DBF (deblocking filter) DCT Discrete Cosine Transform DSP Digital Signal Processor ECM (enhanced compression model) EO Edge Offset FDMA Frequency Division Multiple Access FPGA Field Programmable Gate Array GDR (Gradual Decoding Refresh) GSM Global System for Mobile Communications H.222.0 MPEG-2 Systems, a standard for the generic coding of moving pictures and associated audio information H.26x A family of video coding standards within the ITU-T HMD Head Mounted Display IBC Intra-block copy id or ID(identifier) IEC (International Electrotechnical Commission) IEEE (Institute of Electrical and Electronics Engineers) I / F Interface IMD Integrated Messaging Device IMS Instant Messaging Service I / O (input output) IoT Internet of Things IP Internet Protocol ISO (International Organization for Standardization) ISOBMFF (ISO base media file format) ITU (International Telecommunication Union) ITU-T (ITU Telecommunication Standardization Sector) JTC(joint technical committee) JVNET(joint video experts team) Joint video experts team LEE Laptop Embedded Device LME Laptop Built-in Devices LTE (long term evolution) ML (machine learning) MMS Multimedia Messaging Service MPEG (Moving Picture Experts Group) MPEG-2 H.222 / H.262 as defined by the ITU MSE mean square error MV (multiple views) NAL (network abstraction layer) NN (neural network) N / W Network PC personal computer PDA Personal Digital Assistant PID Packet Identifier PLC (power line communication) QP (quantization parameter or quarter pixel) RAM Random Access Memory RFID (radio frequency identification) RFM Reference Frame Memory ROM Read-Only Memory Rx Receiver SAO (sample adaptive offset) SMS Short Messaging Service SPS (sequence parameter set) TCP-IP Transmission Control Protocol-Internet Protocol TDMA Time Division Multiple Access TS Transport Stream TV (television) Tx transmitter U Blue projection of chrominance component UICC (universal integrated circuit card) UMTS Universal Mobile Telecommunications System USB Universal Serial Bus V Red projection of chrominance component V2X Vehicle to Everything VoIP Voice over IP VVC(versatile video coding) Versatile video coding WLAN Wireless Local Area Network Y luminance component

Claims

1. at least one processor; at least one memory containing computer program code, The at least one memory and the computer program code, together with the at least one processor, cause the device to: determining a virtual boundary separating an image or a portion of said image into a first area and a second area; 1. An apparatus configured to at least: determine, when encoding information of the second area needs to be used to perform filtering of at least one pixel of the first area, that the encoding information of the second area is derived from the first area or that the encoding information of the second area is set to at least one value to perform the filtering of the at least one pixel of the first area; or determine, when encoding information of the second area needs to be used to perform the filtering of the at least one pixel of the first area, not to perform the filtering of the at least one pixel of the first area.

2. The apparatus of claim 1 , wherein the filtering of the at least one pixel of the first area comprises in-loop filtering.

3. 3. The apparatus of claim 1, wherein the first area comprises an updated area and the second area comprises a non-updated area.

4. The apparatus of any one of claims 1 to 3, wherein the image comprises a progressive decoding update image or an image being restored.

5. The at least one memory and the computer program code, together with the at least one processor, cause the device to:

5. The apparatus of claim 1, configured to at least perform filling of the pixels in the second area from pixels in the first area in response to pixels in the second area being used to perform the filtering of the at least one pixel in the first area.

6. The at least one memory and the computer program code, together with the at least one processor, cause the device to:

6. The apparatus of claim 1, configured to at least perform replacing pixels in the second area with pixels extrapolated from the first area in response to pixels in the second area being used to perform the filtering of the at least one pixel in the first area.

7. The at least one memory and the computer program code, together with the at least one processor, cause the device to: determining a first output of the filtering of the at least one pixel of the first area when coding information of the first area and the coding information of the second area are available for the filtering of the at least one pixel of the first area; determining a second output of the filtering of the at least one pixel of the first area when the coding information of the second area is not available for the filtering of the at least one pixel of the first area; The apparatus according to any one of claims 1 to 6, configured to cause at least

8. The at least one memory and the computer program code, together with the at least one processor, cause the device to: determining a difference between the first output and the second output; determining an output of filtering of at least one pixel of the second area using at least in part the difference or an approximation of the difference; and The apparatus of claim 7 , configured to perform at least the following:

9. The apparatus of claim 8 , wherein the encoded information of the second area includes the output of the filtering of the at least one pixel of the second area.

10. 10. The apparatus according to claim 8 or 9, wherein the position of the at least one pixel of the second area corresponds to the position of the at least one pixel of the first area.

11. The at least one memory and the computer program code, together with the at least one processor, cause the device to: determining a difference between the first output and the second output; determining an initial output of filtering of at least one pixel of said second area; at least partially subtracting said difference from said initial output to determine a final output of said filtering of said at least one pixel of said second area; and The device according to any of claims 7 to 10, wherein the coded information of the second area comprises the final output of the filtering of the at least one pixel of the second area.

12. The at least one memory and the computer program code, together with the at least one processor, cause the device to:

12. The apparatus of claim 11, configured to at least partially subtract the weighted contribution of the difference from the initial output to determine the final output of the filtering of the at least one pixel of the second area.

13. The weighted contribution is 1 / 2 i 13. The apparatus of claim 12, comprising: i = i = 1 i , i = 2 i , i = 3 i , i = 4 i , i = 5 i , i = 6 i , i = 7 i , i = 8 i , i = 9 i , i = 10 i , i = 11 i , i = 12 i , i = 13 i , i = 14 i , i = 15 i , i = 16 i , i = 17 i , i = 18 i , i = 19 i , i = 20 i , i = 21 i , i = 22 i , i = 23 i , i = 24 i , i = 25 i , i = 26 i , i = 27 i , i = 28 i , i = 29 i , i = 30 i , i = 31 i ,

14. The at least one memory and the computer program code, together with the at least one processor, cause the device to: determining a target output of target filtering of the at least one pixel of the first area; determining an actual output of the filtering of the at least one pixel of the first area when the coding information of the first area or the coding information of the second area is not available to perform the filtering of the at least one pixel of the first area; determining a difference between the target output and the actual output; determining an initial output of filtering of at least one pixel of said second area; determining a final output of the filtering of the at least one pixel of the second area using the initial output at least partially offset by the difference; and and The device according to any of the preceding claims, wherein the coded information of the second area comprises the final output of the filtering of the at least one pixel of the second area.

15. The at least one memory and the computer program code, together with the at least one processor, cause the device to:

15. The device according to claim 1, configured to at least perform the following in response to determining a common option related to the filtering of the at least one pixel of the first area and the filtering of the at least one pixel of the second area: determining whether to perform the filtering of the at least one pixel of the first area without using the encoding information of the second area; and determining whether to perform the filtering of the at least one pixel of the second area without using the encoding information of the first area.

16. 16. The apparatus of claim 15, wherein the common options include deciding not to perform filtering of the at least one pixel of the first area and deciding not to perform filtering of the at least one pixel of the second area.

17. 17. The apparatus according to claim 15 or 16, wherein the common options include: deciding to fill the coded information of the second area and perform filtering of the at least one pixel of the first area; and deciding to fill the coded information of the first area and perform filtering of the at least one pixel of the second area.

18. The at least one memory and the computer program code, together with the at least one processor, cause the device to:

18. The apparatus according to any of claims 1 to 17, configured to at least perform filtering of at least one pixel of the second area using the coding information of the first area and the coding information of the second area.

19. The filtering of the at least one pixel of the first area comprises: deblocking filtering, Sample adaptive offset edge offset filtering, Bilateral filtering for luminance, Bilateral filtering for saturation, Inter-component sample adaptive offset filtering, Adaptive loop filtering, or An apparatus according to any preceding claim, comprising at least one of: inter-component adaptive loop filtering.

20. The at least one memory and the computer program code, together with the at least one processor, cause the device to:

20. The device according to any of the preceding claims, configured to at least cause disabling of said filtering of said at least one pixel of said first area up to a plurality of pixel positions from said virtual boundary.

21. The at least one memory and the computer program code, together with the at least one processor, cause the device to: The device according to any of the preceding claims, configured to at least cause performing said filtering of said at least one pixel of said first area up to a plurality of pixel positions from said virtual boundary.

22. The at least one memory and the computer program code, together with the at least one processor, cause the device to: setting pixel values ​​of the second area to be equal to pixel values ​​of the first area adjacent the virtual boundary; setting pixel values ​​in the second area to be equal to the average value of pixel values ​​in the first area; or setting pixel values ​​of the second area to be equal to a median of pixel values ​​of the first area; Apparatus according to any preceding claim, wherein the coded information of the second area comprises the set pixel values ​​of the second area.

23. The at least one memory and the computer program code, together with the at least one processor, cause the device to:

23. The device according to claim 1, configured to at least: determine to perform the filtering of the at least one pixel of the second area, when coding information of the first area needs to be used to perform filtering of the at least one pixel of the second area, such that the coding information of the first area is derived from the second area or is set to at least one value; or determine not to perform the filtering of the at least one pixel of the second area, when coding information of the first area needs to be used to perform filtering of the at least one pixel of the second area.

24. The at least one memory and the computer program code, together with the at least one processor, cause the device to:

24. The apparatus of claim 23, configured to at least cause disabling filtering of at least one pixel of the second area up to a plurality of pixel positions from the virtual boundary.

25. The at least one memory and the computer program code, together with the at least one processor, cause the device to:

25. The apparatus according to claim 23 or 24, configured to at least cause performing said filtering of said at least one pixel of said second area up to a plurality of pixel positions from said virtual boundary.

26. The at least one memory and the computer program code, together with the at least one processor, cause the device to: setting pixel values ​​of the first area equal to pixel values ​​of the second area adjacent the virtual boundary; setting the pixel values ​​of the first area to be equal to the average value of the pixel values ​​of the second area; or setting pixel values ​​of the first area to be equal to a median of pixel values ​​of the second area; Apparatus according to any of claims 23 to 25, wherein the coded information of the first area comprises the set pixel values ​​of the first area.

27. The filtering of the at least one pixel of the second area comprises: In-loop filtering, deblocking filtering, Sample adaptive offset edge offset filtering, Bilateral filtering for luminance, Bilateral filtering for saturation, Inter-component sample adaptive offset filtering, Adaptive loop filtering, or An apparatus according to any of claims 23 to 26, comprising at least one of inter-component adaptive loop filtering.

28. The at least one memory and the computer program code, together with the at least one processor, cause the device to: The apparatus of any of claims 1 to 27, configured to at least cause determining said at least one value using a bit depth BD.

29. The at least one memory and the computer program code, together with the at least one processor, cause the device to: 2 BD-1 29. The apparatus of claim 28, configured to at least determine the at least one value as

30. determining a virtual boundary separating an image or a portion of said image into a first area and a second area; determining, when encoding information of the second area needs to be used to perform filtering of at least one pixel of the first area, that the encoding information of the second area is derived from the first area or that the encoding information of the second area is set to at least one value to perform the filtering of the at least one pixel of the first area, or determining, when encoding information of the second area needs to be used to perform the filtering of the at least one pixel of the first area, not to perform the filtering of the at least one pixel of the first area.

31. means for determining a virtual boundary separating an image or a portion of said image into a first area and a second area; and means for determining, when encoding information of the second area needs to be used to perform filtering of at least one pixel of the first area, that the encoding information of the second area is derived from the first area or that the encoding information of the second area is set to at least one value to perform the filtering of the at least one pixel of the first area, or means for determining, when encoding information of the second area needs to be used to perform the filtering of the at least one pixel of the first area, not to perform the filtering of the at least one pixel of the first area.

32. A non-transitory program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform operations, the operations comprising: determining a virtual boundary separating an image or a portion of said image into a first area and a second area; determining, when encoding information of the second area needs to be used to perform filtering of at least one pixel of the first area, that the encoding information of the second area is derived from the first area or that the encoding information of the second area is set to at least one value to perform the filtering of the at least one pixel of the first area, or determining, when encoding information of the second area needs to be used to perform the filtering of the at least one pixel of the first area, not to perform the filtering of the at least one pixel of the first area.

Images