Asymmetric loop filter at virtual boundaries

Abstract

In accordance with an example embodiment of the present invention, there is at least one method and an apparatus to perform the following: determining a virtual boundary separating an image or a portion of an image into a first region and a second region; and determining to perform filtering of the at least one pixel of the first region using the encoding information of the second region derived from the first region or using the encoding information of the second region set to at least one value when the encoding information of the second region is to be used for performing filtering of the at least one pixel of the first region, or when the encoding information of the second region is to be used for performing the filtering of the at least one pixel of the first region, determining that the filtering of the at least one pixel of the first region is not performed.

Description

Technical Field

[0001] The exemplary and non-limiting embodiments relate generally to multimedia transmission and information encoding and decoding, and more particularly to asymmetric loop filters at virtual boundaries. Background Art

[0002] It is known to perform data compression and decoding in multimedia systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The above aspects and other features are explained in the following description taken in conjunction with the accompanying drawings, in which:

[0004] Figure 1 An electronic device employing embodiments of the examples described herein is schematically shown.

[0005] Figure 2 A user device suitable for employing embodiments of the examples described herein is schematically shown.

[0006] Figure 3 Electronic devices employing embodiments of the examples described herein connected using wireless and wired network connections are also schematically shown.

[0007] Figure 4 A block diagram of an encoder used for general horizontal data compression is schematically shown.

[0008] Figure 5 It shows that the refresh area is not allowed to use the coding information of the non-refresh area.

[0009] Figure 6 It shows that the non-refresh area is allowed to use the encoding information of the refresh area.

[0010] Figure 7 shows that deblocking may not be applied to pixel p i , i = 0, 1, 2, or still applied to the filled pixels q in the non-refresh area i ,i=0,1,2.

[0011] Figure 8 Four edge classes are shown.

[0012] Fig. 9 Four edge categories are shown.

[0013] Fig. 10A Describing the SAO edge offset may not be applied to pixel p 0 , or still applied to the filled pixel q 0 .

[0014] Fig. 10B Describes that the SAO edge offset may not be applied to pixel q0 , or still applied to the filled pixel p 0 .

[0015] Fig.11 Offsets depicting BIF-luma, SAO, and CCSAO are added to the deblocked output.

[0016] Fig. 12A The delineation BIF may not be applied to pixel p 0,0 , or still be applied to fill including q i,0 , the associated pixels of i=0,1.

[0017] Fig. 12B Delineation BIF may not be applied to pixel q 0,0 , or still be applied to fill including p i,0 , the associated pixels of i=0,1.

[0018] Fig.13 Describe the decoding workflow of CCSAO.

[0019] Fig.14 It is shown that for co-located chrominance samples, the co-located luma samples can be selected from 9 candidate positions.

[0020] Fig.15A It shows that CCSAO may not be applied to pixel p 0 , or still applied to the filled pixel q 0 .

[0021] Fig. 15B Shows that CCSAO may not be applied to pixel q 0 , or still applied to the filled pixel p 0 .

[0022] Fig.16A shows that ALF may not be applied to pixel p 0,0 , or still be applied to fill including q i,0 , the associated pixels of i=0,1,2.

[0023] Fig. 16B Shows that ALF may not be applied to pixel q 0,0 , or still be applied to fill including p i,0 , the associated pixels of i=0,1,2.

[0024] Fig.17 It is a basic diagram of CCALF in VVC.

[0025] Fig.18 Depicts the 25-tap filter used for CCALF in ECM.

[0026] Fig.19A shows that CCALF may not be applied to at least one of the co-located chrominance pixels, or may still be applied to the filled luma pixels q i,j ,i=0,1,2,3 and j=0,1.

[0027] Fig.19B shows that CCALF may not be applied to at least one of the co-located chrominance pixels, or may still be applied to the filled luma pixel p i,j ,i=0,1,2,3 and j=0,1.

[0028] Fig. 20 An example apparatus configured to implement an asymmetric loop filter at a virtual boundary based on the examples described herein.

[0029] Fig.21 is an example method of implementing an asymmetric loop filter at a virtual boundary based on the examples described herein. DETAILED DESCRIPTION

[0030] This paper describes a practical method to implement asymmetric loop filters at virtual boundaries. The model described in this paper can be used to perform any task such as data compression, data decompression, video compression, video decompression, image or video classification, object classification, object detection, object tracking, speech recognition, language translation, music transcription, etc.

[0031] Suitable means and possible mechanisms for implementing aspects of an asymmetric loop filter at a virtual boundary are described in detail below. In this regard, reference is first made to Figure 1 and Figure 2 ,in Figure 1 An example block diagram of an apparatus 50 is shown. The apparatus may be an Internet of Things (IoT) apparatus configured to perform various functions, such as, for example, collecting information through one or more sensors, receiving or sending information, analyzing information collected or received by the apparatus, etc. The apparatus may include a neural network weight update encoding system, which may include a codec. Figure 2 A layout of an apparatus according to an example embodiment is shown. Figure 1 and Figure 2 The elements of are explained next.

[0032] The electronic device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag or other low-power device. Alternatively, the electronic device may be a non-mobile computer or part of a computer. However, it should be understood that the exemplary embodiments described herein may be implemented in any electronic device or device that can process data.

[0033] Device 50 may include a housing 30 for combining and protecting the device. Device 50 may also include a display 32 in the form of a liquid crystal display. In other embodiments of the examples described herein, the display may be any suitable display technology suitable for displaying images or videos. Device 50 may also include a keyboard 34 (or touch area 34). In other embodiments of the examples described herein, any suitable data or user interface mechanism may be adopted. For example, the user interface may be implemented as a virtual keyboard or as a data input system as a part of a touch-sensitive display.

[0034] The device may include a microphone 36 or any suitable audio input, which may be a digital or analog signal input. The device 50 may also include an audio output device, which may be any of the following in the exemplary embodiments described herein: headphones 38, speakers, or analog audio or digital audio output connections. The device 50 may also include a battery (or in other exemplary embodiments described herein, the device may be powered by any suitable mobile energy device, such as a solar cell, a fuel cell, or a clockwork generator). The device may also include a camera 42 capable of recording or capturing images and / or videos. The device 50 may also include an infrared port for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may also include any suitable short-range communication solution, such as, for example, a Bluetooth wireless connection or a USB / FireWire wired connection.

[0035] The apparatus 50 may include a controller 56, a processor or processor circuitry, for controlling the apparatus 50. The controller 56 may be connected to a memory 58, which in the exemplary embodiments described herein may store both data in the form of image and audio data and / or may also store instructions for implementation on the controller 56. The controller 56 may also be connected to a codec circuitry 54, which may be adapted to perform encoding / compression of neural network weight updates and / or decoding of audio and / or video data or to assist in encoding and / or decoding performed by the controller.

[0036] The apparatus 50 may also include a card reader 48 and a smart card 46, such as a UICC and a UICC reader, for providing user information and adapted to provide authentication information for authentication and authorization of the user at the network.

[0037] The device 50 may include a radio interface circuit system 52 connected to the controller, the radio interface circuit system 52 being suitable for generating wireless communication signals, for example for communicating with a cellular communication network, a wireless communication system or a wireless local area network. The device 50 may also include an antenna 44 connected to the radio interface circuit system 52 for transmitting radio frequency signals generated at the radio interface circuit system 52 to (multiple) other devices (such as network nodes) and / or for receiving radio frequency signals from (multiple) other devices.

[0038] The device 50 may include a camera capable of recording or detecting individual frames, which are then passed to the codec 54 or controller for processing. The device may receive video image data or machine learning data from another device for processing before transmission and / or storage. The device 50 may also receive images for encoding / decoding via a wireless or wired connection. The structural elements of the device 50 described above represent examples of components for performing corresponding functions.

[0039] about Figure 3 , embodiments of the examples described herein may be shown as examples of systems in which they may be utilized. System 10 includes a plurality of communication devices that may communicate via one or more networks. System 10 may include any combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, CDMA, LTE, 4G, 5G networks, etc.), wireless local area networks (WLANs) (such as defined by any IEEE 802.x standards), Bluetooth personal area networks, Ethernet local area networks, token ring local area networks, wide area networks, and the Internet.

[0040] System 10 may include both wired and wireless communication devices and / or apparatuses 50 suitable for implementing the exemplary embodiments described herein.

[0041] For example, Figure 3 The system shown shows a representation of a mobile phone network 11 and the Internet 28, which use a communication link 2 (wired or wireless) to communicate with each other. Figure 3 The various devices shown are accessible. Connections to the Internet 28 may include, but are not limited to, long-range wireless connections, short-range wireless connections, and various wired connections, including but not limited to telephone lines, cable lines, power lines, and similar communication paths.

[0042] The example communication devices shown in the system 10 may include, but are not limited to, an electronic device or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a laptop computer 22. The apparatus 50 may be stationary or mobile and carried by a moving individual. The apparatus 50 may also be located in a vehicle, including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle, or any similar suitable vehicle, or a head mounted display (HMD) 17.

[0043] Embodiments may also be implemented in a set-top box; i.e., a digital TV receiver that may or may not have a display or wireless capabilities, in a tablet or (laptop) personal computer (PC) having hardware and / or software to process neural network data, in various operating systems, and in chipsets, processors, DSPs and / or embedded systems that provide hardware / software based encoding.

[0044] Some or additional devices may send and receive calls and messages and communicate with service providers via a wireless connection 25 with a base station 24. The base station 24 may be connected to a network server 26 that allows communication between the mobile phone network 11 and the Internet 28. The system may include additional communication devices and various types of communication devices.

[0045] The communication device may communicate using various transmission technologies, including but not limited to code division multiple access (CDMA), global system for mobile communications (GSM), universal mobile telecommunications system (UMTS), time division multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-Internet protocol (TCP-IP), short message service (SMS), multimedia message service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11, 3GPP narrowband IoT and any similar wireless communication technology. The communication devices involved in implementing the various embodiments of the examples described herein may communicate using various media, including but not limited to radio, infrared, laser, cable connection and any suitable connection.

[0046] In telecommunications and data networks, a channel can refer to a physical channel or a logical channel. A physical channel can refer to a physical transmission medium, such as a wire, while a logical channel can refer to a logical connection on a multiplexed medium, capable of transmitting multiple logical channels. A channel can be used to transmit an information signal (e.g., a bit stream) from one or several senders (or transmitters) to one or several receivers.

[0047] Embodiments may also be implemented in so-called IoT devices. The Internet of Things (IoT) may be defined as the interconnection of uniquely identifiable embedded computing devices, for example within an existing Internet infrastructure. The convergence of various technologies has and may enable many embedded system areas, such as wireless sensor networks, control systems, home / building automation, etc., to be incorporated into the Internet of Things (IoT). In order to utilize the Internet, IoT devices are provided with an IP address as a unique identifier. IoT devices may be provided with a radio transmitter, such as a WLAN or Bluetooth transmitter or an RFID tag. Alternatively, IoT devices may have access to an IP-based network via a wired network, such as an Ethernet-based network or a power line connection (PLC).

[0048] Asymmetric In-Loop Filters at Virtual Boundaries and Model Level Update Skipping in Compressed Incremental Learning One application where skipping is important is the use case of neural network based codecs, such as neural network based video codecs. A video codec may use one or more neural networks. In the first case, the video codec may be a conventional video codec, such as the Versatile Video Codec (VVC / H.266) that has been modified to include one or more neural networks. Examples of these neural networks are:

[0049] 1. A neural network filter that will be used as one of the loop filters of VVC

[0050] 2. A neural network filter to replace one or more loop filters of the VVC (multiple) loop filters

[0051] 3. Neural network filter to be used as post-processing filter

[0052] 4. The neural network that will be used to perform intra-frame prediction

[0053] 5. The neural network that will be used to perform inter-frame prediction.

[0054] In the second case, which is often referred to as end-to-end learning of a video codec, a video codec can include a neural network that transforms input data into a more compressible representation. The new representation can be quantized, losslessly compressed, then losslessly decompressed, dequantized, and then another neural network can transform its input into the reconstructed or decoded data.

[0055] In both cases, there may be one or more neural networks on the decoder side, and consider an example of a neural network filter. The encoder can fine-tune the neural network filter by using real data (uncompressed data) available on the encoder side. Fine-tuning can be performed to improve the neural network filter when it is applied to the current input data (such as one or more video frames). Fine-tuning can include running one or more optimization iterations on some or all of the learnable weights of the neural network filter. The optimization iteration can include calculating the gradient of the loss function relative to some or all of the learnable weights of the neural network filter, for example by using a back-propagation algorithm, and then updating some or all of the learnable weights by using an optimizer (such as a stochastic gradient descent optimizer). The loss function may include one or more loss terms. An example loss term may be the mean squared error (MSE). Other distortion metrics may be used as loss terms. The loss function can be calculated by providing one or more data to the input of the neural network filter, obtaining one or more corresponding outputs from the neural network filter, and calculating the loss term by using one or more outputs from the neural network filter and one or more real data. The difference between the weights of the neural network after fine-tuning and the weights of the neural network before fine-tuning is called a weight update. The weight update needs to be encoded, provided to the decoder side together with the encoded video data, and used at the decoder side to update the neural network filter. The updated neural network filter is then used as part of a video decoding process or as part of a video post-processing process. It is desirable to encode the weight updates so that the weights require fewer bits. Therefore, the examples described herein also consider this use case of a neural network based codec as a potential application for compression of weight updates.

[0056] In further description of the neural network based codec use case, the MPEG-2 Transport Stream (TS) specified in ISO / IEC 13818-1 or ITU-T Recommendation H.222.0 is a format for carrying audio, video and other media as well as program metadata or other metadata in a multiplexed stream. Packet identifiers (PIDs) are used to identify elementary streams within a TS (also known as packetized elementary streams). Therefore, logical channels within an MPEG-2 TS can be considered to correspond to specific PID values.

[0057] Available media file format standards include ISO Base Media File Format (ISO / IEC 14496-12, which may be abbreviated as ISOBMFF) and File Format for NAL Unit Structured Video (ISO / IEC 14496-15), the latter being derived from ISOBMFF.

[0058] A video codec includes an encoder that converts the input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a viewable form. The video encoder and / or video decoder can also be separate from each other, i.e., they do not need to form a codec. Typically, an encoder discards certain information in the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate).

[0059] A typical hybrid video encoder, such as many encoder implementations of ITU-T H.263 and H.264, encodes video information in two stages. First, the pixel values ​​in a particular image area (or "block") are predicted, for example, by motion compensation means (looking up in one of the previously encoded video frames and indicating the area that closely corresponds to the block being encoded) or by spatial means (using the pixel values ​​around the block to be encoded in a specified manner). Second, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This is usually done by transforming the difference in pixel values ​​using a specified transform (e.g., discrete cosine transform (DCT) or its variant), quantizing the coefficients, and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (image quality) and the size of the resulting encoded video representation (file size or transmission bit rate).

[0060] In temporal prediction, the source of the prediction is a previously decoded picture (also known as a reference picture). In intra-block copy (IBC; also known as intra-block copy prediction and current picture reference), prediction is applied similar to temporal prediction, but the reference picture is the current picture, and only previously decoded samples can be referenced during the prediction process. Inter-layer or inter-view prediction can be similarly applied to temporal prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter-frame prediction may refer only to temporal prediction, while in other cases, inter-frame prediction may be collectively referred to as temporal prediction and any of intra-block copy, inter-layer prediction, and inter-view prediction, as long as they are performed using the same or similar process as temporal prediction. Inter-frame prediction or temporal prediction may sometimes be referred to as motion compensation or motion compensated prediction.

[0061] Inter prediction (which may also be called temporal prediction, motion compensation or motion compensated prediction) can reduce temporal redundancy. In inter prediction, the prediction source is a previously decoded image. Intra prediction exploits the fact that adjacent pixels within the same image may be correlated. Intra prediction can be performed in the spatial or transform domain, i.e., sample values ​​or transform coefficients can be predicted. Intra prediction is usually utilized in intra coding, in which no inter prediction is applied.

[0062] One result of the encoding process is a set of coded parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy coded more efficiently if they are first predicted from spatially or temporally neighboring parameters. For example, a motion vector can be predicted from spatially neighboring motion vectors, and only the difference with respect to the motion vector predictor can be coded. Prediction of coding parameters and intra-frame prediction may be collectively referred to as intra-picture prediction.

[0063] Figure 4 A block diagram showing the general structure of a video encoder. Figure 4 An encoder for two layers is shown, but it should be understood that the presented encoder can be similarly extended to encode more than two layers. Figure 4 A video encoder is shown that includes a first encoder part 500 for a base layer and a second encoder part 502 for an enhancement layer. Each of the first encoder part 500 and the second encoder part 502 may include similar elements for encoding an incoming image. The encoder parts 500, 502 may include a pixel predictor 302, 402, a prediction error encoder 303, 403 and a prediction error decoder 304, 404. Figure 4 Embodiments of the pixel predictor 302, 402 are also shown as including an inter-frame predictor 306, 406 (P inter ), intra-frame predictor 308, 408 (P intra ), mode selectors 310, 410, filters 316, 416 (F) and reference frame memories 318, 418 (RFM). The pixel predictor 302 of the first encoder section 500 receives 300 base images (I ) of a video stream to be encoded at both the inter-frame predictor 306 (which determines the difference between the image and the motion compensated reference frame 318) and the intra-frame predictor 308 (which determines the prediction for the image block based only on the processed portion of the current frame or image). 0,n ). The outputs of both the inter predictor and the intra predictor are passed to the mode selector 310. The intra predictor 308 may have more than one intra prediction mode. Thus, each mode may perform intra prediction and provide a predicted signal to the mode selector 310. The mode selector 310 also receives a copy of the base layer image 300. Correspondingly, the pixel predictor 402 of the second encoder part 502 receives 400 enhancement layer images (I 2 0 ) of the video stream to be encoded at both the inter predictor 406 (which determines the difference between the image and the motion compensated reference frame 418) and the intra predictor 408 (which determines the prediction of the image block based only on the processed part of the current frame or image). 1,n). The outputs of both the inter predictor and the intra predictor are passed to a mode selector 410. The intra predictor 408 may have more than one intra prediction mode. Thus, each mode may perform intra prediction and provide a predicted signal to the mode selector 410. The mode selector 410 also receives a copy of the enhancement layer image 400.

[0064] Depending on which coding mode is selected to encode the current block, the output of the inter-frame predictor 306, 406 or the output of one of the optional intra-frame predictor modes or the output of the surface encoder within the mode selector is passed to the output of the mode selector 310, 410. The output of the mode selector is passed to the first summing device 321, 421. The first summing device can subtract the output of the pixel predictor 302, 402 from the base layer image 300 / enhancement layer image 400 to produce a first prediction error signal 320, 420 (D n ), the first prediction error signal is input to the prediction error encoder 303, 403.

[0065] The pixel predictor 302, 402 also receives a predicted representation (P') of the image block 312, 412 from the preliminary reconstructor 339, 439. n ) and the output 338, 438 (D') of the prediction error decoder 304, 404 n ) combination. Preliminary reconstructed images 314, 414 (I ' n ) may be passed to the intra predictor 308, 408 and the filter 316, 416. The filter 316, 416 receiving the preliminary representation may filter the preliminary representation and output a final reconstructed image 340, 440 (R' n ), the final reconstructed images 340, 440 (R' n ) may be stored in the reference frame memory 318, 418. The reference frame memory 318 may be connected to the inter predictor 306 to be used as a reference image for comparison with the future base layer image 300 in the inter prediction operation. According to some embodiments, subject to the base layer being selected and indicated as a source for inter-layer sample prediction and / or inter-layer motion information prediction for the enhancement layer, the reference frame memory 318 may also be connected to the inter predictor 406 to be used as a reference image for comparison with the future enhancement layer image 400 in the inter prediction operation. In addition, the reference frame memory 418 may be connected to the inter predictor 406 to be used as a reference image for comparison with the future enhancement layer image 400 in the inter prediction operation.

[0066] According to some embodiments, the filter parameters of the filter 316 from the first encoder portion 500 may be provided to the second encoder portion 502, subject to the base layer being selected and indicated as a prediction source for the filter parameters of the enhancement layer.

[0067] The prediction error encoder 303, 403 comprises a transform unit 342, 442 (T) and a quantizer 344, 444 (Q). The transform unit 342, 442 transforms the first prediction error signal 320, 420 into a transform domain. The transform is, for example, a DCT transform. The quantizer 344, 444 quantizes the transform domain signal (e.g., DCT coefficients) to form quantized coefficients.

[0068] The prediction error decoder 304, 404 receives the output from the prediction error encoder 303, 403 and performs the inverse process of the prediction error encoder 303, 403 to produce a decoded prediction error signal 338, 438, which when combined with the predicted representation of the image block 312, 412 at the second summing device 339, 439, produces a preliminary reconstructed image 314, 414. The prediction error decoder 304, 404 can be considered to include an inverse quantizer 346, 446 (Q -1 ) and inverse transform units 348, 448 (T -1 ), inverse quantizer 346, 446 (Q -1 ) dequantizes the quantized coefficient values, such as DCT coefficients, to reconstruct the transformed signal, the inverse transform unit 348, 448 (T -1 ) performs an inverse transform on the reconstructed transformed signal, wherein the output of the inverse transform unit 348, 448 contains (multiple) reconstructed blocks. The prediction error decoder may also include a block filter that may filter (multiple) reconstructed blocks based on further decoded information and filter parameters.

[0069] The entropy encoder 330, 430 (E) receives the output of the prediction error encoder 303, 403 and may perform appropriate entropy encoding / variable length encoding on the signal to provide error detection and correction capabilities. The output of the entropy encoder 330, 430 may be inserted into the bitstream, for example, via a multiplexer 508 (M).

[0070] The concept of virtual boundaries is introduced in VVC. From the perspective of coding dependency, images can be divided into different areas by virtual boundaries. For example, 360°: virtual boundaries are used to define the boundaries of different faces of a 360° image in CMP format, and GDR (refer to U.S. Provisional Application No. 63 / 296,590, "New Gradual Decoding Refresh for ECM" filed by the present applicant), where virtual boundaries separate the refresh area and non-refresh area of ​​the GDR / recovery image. In VVC, virtual boundaries are specified in the SPS and / or image header.

[0071] There are three loop filters in VVC. They are deblocking, SAO and ALF. ECM enhances the loop filter with new features, including bilateral (JVET-F0034, JVET-V0094), BIF for chroma (JVET-X0067), CCSAO (JVET-V0153, JVET-Y0106), CCALF (JVET-X0045) and alternative band classifier for ALF (JVET-X0070).

[0072] Loop filtering of the current pixel usually requires using coded information of its neighbors. Therefore, filtering on one side of a virtual boundary may involve using coded information about the other side of the virtual boundary.

[0073] For some applications, loop filtering may not be allowed to cross virtual boundaries. For example, in GDR, the GDR / restored image can be divided into refreshed and non-refreshed areas by virtual boundaries. Figure 5 , in order to avoid leakage, the refresh area 510 cannot use any information of the non-refresh area 530, because there is no guarantee that the non-refresh area 530 is correctly decoded at the decoder. The incorrectly decoded encoded information may pollute the refresh area 510, which may cause leakage or mismatch between the encoder and the decoder at the recovery point image and the continuous image. Therefore, as shown by arrow 540, for the GDR / recovery image, the loop filtering cannot cross the virtual boundary 520 from the refresh area 510 to the non-refresh area 530.

[0074] On the other hand, sometimes it is perfectly fine to let the loop filter cross an imaginary boundary. Figure 6 As shown, in the same example of GDR, the non-refresh region 630 can use the information of the refresh region 610. Therefore, as shown by arrow 640, for the GDR / restored image, the loop filtering can cross the virtual boundary 620 from the non-refresh region 630 to the refresh region 610.

[0075] In the current design of VVC and ECM, loop filtering cannot cross virtual boundaries.

[0076] U.S. Provisional Application No. 63 / 362,243, "In-Loop Filtering at Virtual Boundaries," filed by the applicant of the present application, proposes several possible options for performing in-loop filtering at virtual boundaries. Among them is asymmetric in-loop filtering at virtual boundaries. With this asymmetric option, in-loop filtering cannot cross from one side of a virtual boundary to the other side of the virtual boundary, but can cross from the other side to the one side.

[0077] Specifically, loop filtering on one side of the virtual boundary cannot use information on the other side of the virtual boundary, but loop filtering on the other side of the virtual boundary can use information on that side. If loop filtering of a pixel on one side of the virtual boundary requires the use of any information on the other side (e.g., pixel, coding mode, QP, etc.), loop filtering is either not performed on the pixel, or is still performed on the pixel but the information on the other side is filled in.

[0078] In the case of asymmetric loop filtering at a virtual boundary, loop filtering on one side cannot use information on the other side, but loop filtering on the other side can use the information on the one side.

[0079] If loop filtering of pixels requires using encoding information on the other side, loop filtering of pixels on the one side may not be performed normally.

[0080] Generally, the loop filtering of the pixels on the other side can be performed normally because the loop filtering of the pixels can use both the encoding information of the one side and the other side. However, the other side may choose not to use the encoding information of the one side. In this case, if the loop filtering of the pixels needs to use the encoding information of the one side, the loop filtering of the pixels on the other side may not be performed normally.

[0081] Since the encoding information of one side is available to the other side, an offset based on the loop filtering of the one side may be added to the output of the loop filtering of the other side.

[0082] The virtual boundary is a line used to divide an image or a portion of an image into two areas: a first area and a second area.

[0083] The virtual boundary can be vertical or horizontal. In VVC and ECM, the virtual boundary syntax is included in the SPS and / or the image header. In one embodiment, such as in the case of asymmetric operation at the virtual boundary, the first area is not allowed to use any information of the second area, but the second area can use the information of the first area.

[0084] In one embodiment, in the GDR / restored image, the first region is a clean (refreshed) region and the second region is a dirty (unrefreshed) region. The clean (refreshed) region cannot use any information of the dirty (unrefreshed) region, but the dirty (unrefreshed) region can use information of the clean (refreshed) region. Loop filtering for a pixel may involve using the coded information of its neighbors.

[0085] If loop filtering of pixels in the first region requires the use of coding information (e.g., pixels, coding modes, reference images, MVs, QPs, etc.) of the second region, loop filtering of the pixels may not be performed normally. The actual loop filtering for the pixels may adopt one of two possible options, option 1, in which loop filtering for pixels in the first region is not performed, or option 2, in which loop filtering for pixels in the first region is still performed, but when necessary, the coding information of the second region derived from the first region or the coding information of the second region set to a predetermined value is used.

[0086] An embodiment related to option 2 is that if loop filtering of pixels in the first region requires the use of pixels in the second region, the pixels in the second region are filled from pixels in the first region.

[0087] Another embodiment related to option 2 is that if loop filtering of pixels in the first region requires the use of pixels in the second region, the pixels in the second region will be replaced by pixels extrapolated from the first region.

[0088] Let normal loop filtering of a pixel be ideal loop filtering performed on the pixel using all necessary information, and actual loop filtering of a pixel be actual loop filtering performed on the pixel with or without using all necessary information.

[0089] Regardless of option 1 or 2, the actual loop filtering generated output of the pixel may be different from normal loop filtering of the pixel, which may use the encoded information of both the first region and the second region.

[0090] In-loop filtering for pixels in the second region may generally be performed normally because in-loop filtering for pixels in the second region is allowed to use encoding information of both the first region and the second region.

[0091] Make p i,j becomes a pixel within the first region, and and Correspondingly, p i,j The normal and actual loop filter output, and make q i,j become pixels within the second region, and and Correspondingly, q i,j Normal loop filter output and actual loop filter output.

[0092] At the virtual boundary, if pixel p i,j The loop filtering needs to use the encoding information of the second area, so the pixel p in the first area i,j The actual loop filtering of may not be equal to the normal loop filtering of the pixel, i.e.

[0093] On the other hand, at the virtual boundary, the pixel q in the second area i,j The actual loop filtering of is usually equal to the normal loop filtering of the pixel, i.e. Because the pixel q i,j The actual loop filtering may use the encoding information of both the first region and the second region.

[0094] To compensate for the unbalanced loop filtering at the virtual boundary, the difference between the normal loop filtering and the actual loop filtering of the first region can be compensated by the loop filtering of the second region. Note that using the first region to offset the second region is feasible because the second region can use the encoding information of the first region.

[0095] Specifically, if the pixel p in the first region i,j The normal and actual loop filtering is different, namely The difference or its approximation can be used to offset the corresponding pixel q in the second region m,n The output of the loop filter. A possible example is as follows.

[0096]

[0097] in Yes m,n The final output of the loop filter, and w i,j is used for Q m,n The weight of the contribution.

[0098] In one embodiment, the second region may choose not to use the encoding information of the first region. In this case, if the loop filtering of the pixels in the second region needs to use the encoding information of the first region, the loop filtering of the pixels may not be performed normally. Similar to the first region, the actual loop filtering for the pixels can adopt one of two possible options, option 1, in which the loop filtering for the pixels in the second region is not performed, or option 2, in which the loop filtering for the pixels in the second region is still performed, but when necessary, the encoding information of the first region derived from the second region or set to a predetermined value of the first region is used.

[0099] An embodiment related to the above Option 2 is that if loop filtering of pixels in the second area requires the use of pixels in the first area, the pixels in the first area are filled from pixels in the second area.

[0100] Another embodiment related to option 2 above is that if loop filtering of pixels in the second region requires the use of pixels in the first region, the pixels in the first region are replaced by pixels extrapolated from the second region.

[0101] Pixel p in the first region i,j The difference between the target loop filtering and the actual loop filtering can be used to offset the corresponding pixel q in the second area. m,n The output of the loop filter. A possible example is as follows.

[0102]

[0103] in Yes m,n The final output of the loop filter is Yes m,n The output of the loop filter, Yes i,j The target loop filter output, and w i,j is used for Q m,n The weight of the contribution.

[0104] In one embodiment, if the first region and the second region select the same option for loop filtering of pixels around the virtual boundary, i.e., not performing loop filtering or performing it with padding, the loop filtering of the first region and the second region can be considered balanced. Compensation may not be required on both sides of the virtual boundary.

[0105] One embodiment relates to deblocking filters in VVC and ECM.Deblocking filtering is applied at (horizontal or vertical) block boundaries, involving pixels on both sides of the block boundary.

[0106] It is assumed that a virtual boundary separates an image or a portion of an image into a first region and a second region, and the first region is not allowed to use encoding information in the second region, but the second region can use encoding information in the first region.

[0107] If the block boundary is aligned with the virtual boundary, deblocking filtering of pixels in the first region that are at most n pixel positions away from the virtual boundary (e.g., 1 for a weak chroma filter, 2 for a weak luma filter, 3 for a strong luma and chroma filter, and 3, 5, 7 for the luma bilinear (long) filter in the current design of VVC and ECM) requires the use of encoding information (e.g., pixels, encoding mode, QP, etc.) in the second region.

[0108] Since the first region is not allowed to use the encoding information in the second region, deblocking filtering is disabled for pixels in the first region that are at most n pixel positions away from the virtual boundary. Figure 7An example is shown in which a refresh area (first area) 7010 of a GDR / restored image is not allowed to use the coded information of a non-refresh area (second area) 7030. Deblocking (e.g., strong filter) 7040 is performed on pixels p in the refresh area 7010 immediately following the virtual boundary 7020. i , i=0,1,2 are disabled.

[0109] Alternatively, deblocking filtering 7040 is still applied to pixels in the first region that are at most n pixel positions away from the virtual boundary 7020, but the encoding information in the second region is derived from the first region or is set to a predetermined value (when necessary). Figure 7 In the example, deblocking (eg, strong filtering) is still applied to the pixels immediately following the virtual boundary 7020 (usually 7040) in the refresh area 7010. i , i = 0, 1, 2, but the associated pixels 7050 (including q i , i = 0, 1, 2) are obtained from the refresh area 7010. For example, q i , i=0,1,2 can be set equal to p 0 , or p i , the average or median of i=0,1,2.

[0110] Deblocking for pixels on the second region may be performed normally, allowing the use of encoding information of both the first region 7010 and the second region 7030.

[0111] If the pixel p in the first area 7010 i The actual deblocking filter is different from the normal deblocking filter, the difference can be seen from the corresponding pixel q i Offset is

[0112]

[0113] in Yes i The final output of the deblocking filter is Yes i The output of the deblocking filter, is to use all necessary information including information of the first area 7010 and / or the second area 7030 to p i Perform normal deblocking filtering on the output, Yes i The actual deblocking filter output, w i,j is used for Q i ∈ ...

[0114] A possible implementation may be as follows,

[0115] for i=0, ..., sq-1 where sp and sq are respectively for pixel p in the first region i and pixels q in the second region i The filter length is .

[0116] A simple implementation could even be as follows,

[0117]

[0118] like Figure 7 As shown, the corresponding pixel p i and q i These are mirrored pixels in the first region 7010 and the second region 7030 relative to the block boundary or virtual boundary 7020 before deblocking.

[0119] If the second area 7030 chooses not to use the encoding information of the first area 7010, deblocking filtering is not applied to pixels in the second area 7030 that are at most n pixel positions away from the virtual boundary (for example, 1 for a weak chroma filter, 2 for a weak luma filter, 3 for a strong luma and chroma filter, and 3, 5, 7 for the luma bilinear (long) filter in the current design of VVC and ECM). Figure 7 An example in which the non-refresh area (second area) 7030 that can display the GDR / restored image chooses not to use the encoding information of the refresh area (first area) 7010. For the pixel q immediately following the virtual boundary 7020 in the non-refresh area 7030 i , i=0,1,2, deblocking (eg, strong filter) is disabled.

[0120] Alternatively, deblocking filtering is still applied to those pixels 7050 in the second area 7030 that are at most n pixel positions away from the virtual boundary 7020, but the encoding information in the first area 7010 is derived from the second area 7030 or is set to a predetermined value. For example, in FIG. 7 , deblocking (e.g., a strong filter) can still be applied to the pixels (usually 7050) in the non-refreshed area 7030 that are immediately adjacent to the virtual boundary 7020. i , i = 0, 1, 2, but the associated pixels 7040 in the refresh area 7010 (including p i , i = 0, 1, 2) are derived from the non-refresh area 7030. For example, p i , i=0,1,2 can be set equal to q 0 , or q i , the average or median of i=0,1,2.

[0121] One embodiment relates to the SAO edge offset filter. In VVC, SAO has two parts. They are band offset and edge offset. Each CTU can choose to use band offset or edge offset. The choice of band offset or edge offset for each CTU is signaled. For a CTU, if edge offset is used, the parameter set (such as Figure 8 The edge classes shown in Fig. 9 The offsets for the four edge categories shown) are signaled.

[0122] refer to Figure 8 , examples of four edge classes are shown. In example 810, pixels a and b are horizontally adjacent to pixel c. In example 820, pixels a and b are vertically adjacent to pixel c. In example 830, pixels a and b are adjacent to pixel c along a slope from top left to bottom right. In example 840, pixels a and b are adjacent to pixel c along a slope from bottom left to top right.

[0123] refer to Fig. 9 , examples of four edge categories are shown. In category 1 (910), the value of pixel c is lower than the values ​​of pixels a and b. In category 2 (920), the values ​​of pixels c and b may be similar, while the value of pixel a may be higher than the values ​​of pixels c and b. Alternatively, the values ​​of pixels a and c may be similar, while the value of pixel b may be higher than the values ​​of pixels a and c. In category 3 (930), the values ​​of pixels a and c may be similar, while the value of pixel b may be lower than the values ​​of pixels a and c. Alternatively, the values ​​of pixels c and b may be similar, while the value of pixel a may be lower than the values ​​of pixels c and b. In category 4 (940), the value of pixel c may be higher than the values ​​of pixels a and b.

[0124] from Figure 8 and Fig. 9 It can be seen that classifying the edge of a pixel involves using neighboring pixels.

[0125] It is assumed that a virtual boundary separates an image or a portion of an image into a first region and a second region, and the first region is not allowed to use encoding information in the second region, but the second region can use encoding information in the first region.

[0126] The SAO edge offset for pixels in the first region immediately adjacent to the virtual boundary may need to use a Figure 8 The encoded information (eg, pixels) in the second region is shown.

[0127] Since the first region is not allowed to use the encoding information of the second region, the SAO edge offset is not applied to those pixels in the first region that are immediately adjacent to the virtual boundary. Fig. 10AAn example is shown in which the refresh area (first area) 1010 of the GDR / restored image is not allowed to use the coded information of the non-refresh area (second area) 1030. For the pixel p in the refresh area 1010 adjacent to the virtual boundary 1020 0 , the SAO edge offset with diagonal-like direction 1040 is disabled.

[0128] Alternatively, the SAO edge offset (e.g., 1040) is still applied to the pixels in the first region 1010 that are next to the virtual boundary 1020, but when necessary, the encoded information (e.g., pixels) in the second region 1030 derived from the first region 1010 or the encoded information in the second region 1030 that is set to a predetermined value is used. For example, in FIG. 10A, the SAO edge offset is still applied to the pixels p in the refresh region 1010 that are next to the virtual boundary 1020. 0 , but the associated pixel q on the non-refresh area 1030 0 is filled from refresh area 1010 (or is set to a predetermined value, such as 2 BD-1 , where BD is the bit depth).

[0129] For pixels in the second region 1030 that are next to the virtual boundary 1020 , the SAO edge shift may be performed normally, allowing the use of encoding information of both the first region 1010 and the second region 1030 .

[0130] If the pixel p in the first area 1010 0 The actual SAO edge offset filtering is different from the normal SAO edge offset filtering, the difference can be seen from the corresponding pixel q 0 Offset is

[0131]

[0132] in Q 0 The final output of the SAO edge offset filter, Q 0 The output of the SAO edge offset filter, To use all necessary information including information of the first area 1010 and / or the second area 1030, 0 Perform normal SAO edge offset filtering on the output, and For p 0 The output of the actual SAO edge offset filter.

[0133] like Fig. 10A As shown, the corresponding pixel p 0 and q 0It is the mirror pixel along the selected SAO edge class direction line 1040 relative to the junction point of the virtual boundary and the SAO edge offset class direction line.

[0134] If the second region selects not to use the encoding information of the first region, the SAO edge offset is not applied to pixels in the second region immediately adjacent to the virtual boundary. Fig. 10B An example is shown in which the non-refresh area (second area) 1070 of the GDR / restored image chooses not to use the encoding information of the refresh area (first area) 1060. The SAO edge offset is not applied to the pixels q in the non-refresh area 1070 immediately following the virtual boundary 1080. 0 .

[0135] Alternatively, the SAO edge offset is still applied to those pixels immediately following the virtual boundary 1080 in the second region 1070, but utilizing the encoded information in the first region 1060 derived from the second region 1070 or set to a predetermined value when necessary. Fig. 10B In the example, the SAO edge offset is still applied to the pixel q in the non-refresh area 1070 immediately following the virtual boundary 1080. 0 , but the associated pixel p in the refresh area 1060 0 Filled from the never refreshed area 1070 . Fig. 10B Edge class direction lines 1090 are shown in FIG.

[0136] One embodiment involves bilateral filters (BIFs) for luma and chroma. ECM enhances the loop filter of VVC by adding new filter features. Among them is the bilateral filter. Fig.11 As shown, BIF 1130 is executed in parallel with SAO 1120 and CCSAO process 1140. BIF (1130), SAO (1120) and CCSAO (1140) use the same samples produced by the deblocking filter (1110) as input and generate three offsets for each sample in parallel. These three offsets are then added to the input sample (operation 1150) to obtain a sum, which is then clipped to form the final output sample value (1160) before proceeding to the ALF. BIF-Chroma provides an on / off control mechanism at both the CTU level and the slice level.

[0137] like Fig. 12A As shown, the bilateral filters are 5x5 diamonds for both luma and chroma, where the bilateral filters are applied to pixels immediately adjacent to the virtual boundary.

[0138] It is assumed that a virtual boundary divides an image or a portion of an image into a first region and a second region, and the first region is not allowed to use encoding information in the second region, but the second region can use encoding information in the first region.

[0139] BIF filtering of pixels in the first region that are at most n (eg, 2 in the current design of the BIF) pixel locations away from the virtual boundary requires use of coded information (eg, pixels) in the second region.

[0140] Since the first region is not allowed to use the encoding information of the second region, BIF filtering may be disabled for pixels in the first region that are at most n (eg, 2 in the current design of BIF) pixel locations away from the virtual boundary. Fig. 12A An example is shown in which the refresh area (first area) 1210 of the GDR / restored image is not allowed to use the encoding information of the non-refresh area (second area) 1230. The BIF filter does not filter the pixel p immediately following the virtual boundary 1220 in the refresh area 1210. 0,0 implement.

[0141] Alternatively, the BIF filtering is still performed on those pixels 1240 in the first region that are at most n (e.g., 2 in the current design of the BIF) pixel positions away from the virtual boundary, but using the encoding information about the second region derived from the first region or set to a predetermined value when necessary. Fig. 12A In the example, BIF filtering is still applied to the pixel p in the refresh area 1210 immediately following the virtual boundary 1220. 0,0 , but the associated pixels in the non-refresh area 1230 (including q i,0 , i = 0, 1) are filled from the refresh area (or set to a predetermined value, such as 2 BD-1 , where BD is the bit depth).

[0142] BIF filtering of the pixels 1250 on the second region 1230 may be performed normally, allowing the use of encoding information of both the first region 1210 and the second region 1230 .

[0143] If the pixel p in the first area 1210 i,j The actual BIF filtering is different from the normal deblocking filtering, the difference can be seen from the corresponding pixel q m,n Offset is

[0144]

[0145] in Yes m,n The final output of the BIF filter is Yes m,nThe output of the BIF filter, is to use all necessary information including information of the first area 1210 and / or the second area 1230 to p i,j The output of the normal BIF filtering, and Yes i,j The actual BIF filtered output.

[0146] Corresponding pixel p i,j and q i,j are the mirrored pixels in the first region 1210 and the second region 1230 relative to the virtual boundary 1220 before BIF, such as Fig. 12A shown.

[0147] If the second region chooses not to use the encoding information of the first region, then BIF filtering is not applied to pixels in the second region that are at most n pixel positions (eg, 2 in the current design of BIF) away from the virtual boundary. Fig. 12B An example is shown in which the non-refreshed area (second area 1280) of the GDR / restored image chooses not to use the encoding information of the refreshed area (first area) 1260. For the pixel q immediately following the virtual boundary 1270 in the non-refreshed area 1280, 0,0 , BIF is disabled.

[0148] Alternatively, BIF filtering is still applied to those pixels 1295 in the second region 1280 that are at most n pixel positions away from the virtual boundary 1270, but using the encoding information in the first region 1260 derived from the second region 1280 or the encoding information in the first region 1260 set to a predetermined value when necessary. Fig. 12B In the non-refresh region 1280, the BIF filter is still applied to the pixels immediately following the virtual boundary 1270 (usually 1295) (including q 0,0 ), but refreshes the associated pixels in area 1260 (typically 1290) (including p i,0 , i=0,1) are filled from the non-refresh area 1280.

[0149] One embodiment relates to a CCSAO filter. Cross-component sample adaptive offset (CCSAO) is used to refine the reconstructed samples. Similar to SAO, CCSAO classifies the reconstructed samples into different categories, derives an offset for each category, and adds the offset to the reconstructed samples in that category. However, as Fig.13As shown, unlike SAO (1340, 1350, 1360) which uses a single luma / chroma component (one of 1310, 1320, 1330) of the current sample as input, CCSAO (1370, 1380, 1390) utilizes all three components (1310, 1320, 1330) to classify the current sample into different categories. To facilitate parallel processing, the output samples from the deblocking filter are used as input to CCSAO.

[0150] The output of CCSAO Y 1370 is combined (e.g., added or subtracted) with the output of SAO Y 1340 using operation 1391 to generate Y 1394. The output of CCSAO U 1380 is combined (e.g., added or subtracted) with the output of SAO U 1350 using operation 1392 to generate U 1395. The output of CCSAO V 1390 is combined (e.g., added or subtracted) with the output of SAO V 1350 using operation 1393 to generate V 1396.

[0151] In CCSAO, a band offset (BO) classifier or an edge offset (EO) classifier is used to enhance the quality of reconstructed samples.CCSAO can be applied to both luma and chroma components.

[0152] In CCSAO BO, for a given luma / chroma sample, three candidate samples are selected to classify the given sample into different categories, namely, one co-located Y sample, one co-located U sample, and one co-located V sample. The sample values ​​of the three selected samples are then classified into three different bands, and the joint index indicates the category of the given sample. An offset is signaled and added to the reconstructed samples belonging to the category.

[0153] like Fig.14 As shown, the co-located luma sample 1410 may be selected from nine candidate positions (1405), while the co-located chroma sample positions (1420, 1430) are fixed.

[0154] It is assumed that a virtual boundary separates an image or a portion of an image into a first region and a second region, and the first region is not allowed to use encoding information in the second region, but the second region can use encoding information in the first region.

[0155] CCSAO for pixels in the first region immediately adjacent to the virtual boundary may require use of coded information (eg, pixels) in the second region.

[0156] Since the first region is not allowed to use the encoding information of the second region, CCSAO is not applied to those pixels in the first region that are adjacent to the virtual boundary. Fig.15AAn example is shown in which the refresh area (first area) 1510 of the GDR / restored image is not allowed to use the encoding information of the non-refresh area (second area) 1530. For the pixel p immediately following the virtual boundary 1520 in the refresh area 1510 0 , CCSAO is skipped. Fig.15A Shown in FIG. 1 is the co-located chrominance 1540.

[0157] Alternatively, CCSAO is still applied to those pixels in the first region immediately following the virtual boundary, but using the encoded information in the second region derived from the first region or set to a predetermined value when necessary. Fig.15A In the example, CCSAO is still applied to the pixel p in the refresh area 1510 next to the virtual boundary 1520. 0 , but the associated pixel q in the non-refresh area 1530 0 is filled from refresh area 1510 (or is set to a predetermined value, such as 2 BD-1 , where BD is the bit depth).

[0158] CCSAO for pixels on the second region 1530 may be performed normally, being allowed to use the encoding information of the first region 1510 .

[0159] If the pixel p in the first area 1510 0 The actual CCSAO filtering is different from the normal SAO edge offset filtering, the difference can be seen from the corresponding pixel q 0 Offset is

[0160]

[0161] in Q 0 The final output of CCSAO BO filtering is Q 0 The output of the CCSAO BO filter, To use all necessary information including information of the first area 1510 and / or the second area 1530, 0 Perform normal CCSAO BO filtering on the output, and For p 0 The actual CCSAO BO filtered output.

[0162] Corresponding pixel p 0 and q 0 are mirrored pixels in the first region 1510 and the second region 1530 before the CCSAO BO relative to the virtual boundary, such as Fig.15A shown.

[0163] If the second region selects not to use the encoding information of the first region, the CCSAO BO is not applied to the pixels next to the virtual boundary in the second region. Fig. 15B An example is shown in which the non-refresh area (second area) 1580 of the GDR / restored image chooses not to use the encoding information of the refresh area (first area) 1560. CCSAO BO is not applied to the pixels q immediately following the virtual boundary 1570 in the non-refresh area 1580. 0 .

[0164] Alternatively, CCSAO BO is still applied to those pixels in the second region 1580 that are next to the virtual boundary, but using the encoding information in the first region 1560 derived from the second region 1580 or the encoding information in the first region 1560 set to a predetermined value when necessary. Fig. 15B In the example, CCSAO BO is still applied to the pixel q immediately following the virtual boundary 1570 in the non-refresh area 1580. 0 , but refreshes the associated pixel p in region 1560 0 Filled from non-refresh area 1580. Fig. 15B Co-located chrominance 1590 is shown.

[0165] One embodiment is related to the ALF filter. In VVC, the ALF filter is diamond-shaped, with a size of 7x7 for luma and 5x5 for chroma. ECM extends the ALF size to 9x9, 7x7, and 5x5 for luma and chroma. Fig.16A An example of a 9x9 diamond ALF filter is shown on the pixels immediately following the virtual boundary 1620. In addition, ECM adds an alternative band classifier for classification in ALF (ABC-ALF), which is a 13x13 diamond filter used to classify each 2x2 luma block of the ALF.

[0166] It is assumed that a virtual boundary separates an image or a portion of an image into a first region and a second region, and the first region is not allowed to use encoding information in the second region, but the second region can use encoding information in the first region.

[0167] ALF filtering of pixels in the first region that are at most n pixel positions away from the virtual boundary (e.g., 3 for luma ALF and 2 for chroma ALF in the current design of VVC, 2, 3, 4 for luma and chroma ALF and 6 for ABC-ALF in the current design of ECM) requires use of encoded information (e.g., pixels) in the second region.

[0168] Since the first region is not allowed to use the encoding information of the second region, the ALF filtering may be disabled for pixels in the first region that are at most n positions away from the virtual boundary. Fig.16A An example is shown in which the refresh area (first area) 1610 of the GDR / restored image is not allowed to use the encoding information of the non-refresh area (second area) 1630. The ALF does not refresh the pixel p adjacent to the virtual boundary 1620 in the refresh area 1610. 0,0 implement.

[0169] Alternatively, the ALF is still applied to pixels (1640) at a maximum of n positions away from the virtual boundary 1620 in the first region 1610, but the encoding information about the second region 1630 derived from the first region 1610 or the encoding information about the second region 1630 set to a predetermined value is used when necessary. Fig.16A In the refresh area 1610, the ALF still refreshes the pixel p immediately following the virtual boundary 1620. 0,0 Execute, but do not refresh the associated pixels 1650 (including q i,0 , i = 0, 1, 2) are filled from refresh area 1610 (or set to a predetermined value, such as 2 BD-1 , where BD is the bit depth).

[0170] ALF filtering of pixels on the second region may be performed normally, allowing the use of encoding information of the first region and the second region.

[0171] If the pixel p in the first area 1610 i,j The actual ALF filtering is different from the normal deblocking filtering, the difference can be seen from the corresponding pixel q m,n Offset is

[0172]

[0173] in Q m,n The final output of the ALF filter is Q m,n The output of the ALF filter, To use all necessary information including information of the first area 1610 and / or the second area 1630, i,j Perform normal ALF filtering on the output. For p i,j The actual ALF filtered output.

[0174] like Fig.16A As shown, the corresponding pixel p i,j and q i,j1610 and 1630 are mirrored pixels of the first region 1610 and the second region 1630 relative to the virtual boundary 1620 before ALF.

[0175] If the second region chooses not to use the encoding information of the first region, ALF is not applied to pixels in the second region that are at most n pixel positions away from the virtual boundary (for example, in the current VVC design, there are 3 for luma ALF and 2 for chroma ALF, in the current ECM design, there are 2, 3, 4 for luma and chroma ALF, and 6 for ABC-ALF). Fig. 16B An example is shown in which the non-refresh area (second area) 1680 of the GDR / restored image chooses not to use the encoding information of the refresh area (first area) 1660. ALF is not applied to the pixels q immediately following the virtual boundary 1670 in the non-refresh area 1680. 0,0 .

[0176] Alternatively, ALF is still applied to those pixels 1695 in the second region 1680 that are next to the virtual boundary 1670, but using the encoding information in the first region 1660 derived from the second region 1680 or the encoding information in the first region 1660 set to a predetermined value when necessary. Fig. 16B In the example, ALF is still applied to the pixel q in the non-refresh area 1680 next to the virtual boundary 1670. 0,0 , but the associated pixels 1690 in refresh area 1660 (including p i,0 , i=0,1,2) are filled from the unrefreshed area 1680.

[0177] One embodiment involves a CCALF filter. Fig.17 As shown, the CCALF process 1720 uses a linear filter to filter the luma sample values ​​and generates a residual correction (1770) for the chroma samples. Initially, an 8-tap filter was designed for the CCALF process in VVC. More recently, Fig.18 As shown, a 25-tap large filter is used in the CCALF process in ECM (1800). For a given slice, the encoder can collect statistics for the slice, analyze the statistics, and can signal up to 16 filters through the APS.

[0178] refer to Fig.17, shows a basic example of CCALF. At CTU (Y) 1710, CCALF (Cb) can be applied to a set of pixels 1720, as shown at 1730. This can be viewed as a linear filtering of the luma sample values. At CTU (Cb) 1740, ALF chroma can be applied 1750 to a portion of the pixels. This can be viewed as filtering of the chroma samples. The outputs of 1720 and 1750 can be added 1760 (or combined in some other manner, such as subtracted from operation 1760), and the output is CTB' (Cb) 1770.

[0179] It is assumed that a virtual boundary separates an image or a portion of an image into a first region and a second region, and the first region is not allowed to use encoding information in the second region, but the second region can use encoding information in the first region.

[0180] For pixels in the first region that are at most n pixel positions (eg, 1 for VVC or 4 for ECM) away from the virtual boundary, CCALF filtering requires use of coded information (eg, pixels) in the second region.

[0181] Since the first region is not allowed to use the encoding information of the second region, CCALF filtering may be disabled for those pixels in the first region that are at most n pixel positions away from the virtual boundary. Fig.19A An example is shown where the refresh area 1910 (first area) of the GDR / restored image is not allowed to use the coded information of the non-refresh area 1930 (second area). For the chroma pixels 1950 immediately following the virtual boundary 1920 in the refresh area 1910, CCALF is skipped.

[0182] Alternatively, CCALF is still applied to those pixels in the first region that are at most n pixel positions away from the virtual boundary, but using the encoding information in the second region derived from the first region or set to a predetermined value when necessary. Fig.19A In the example, CCALF is still applied to the chroma pixels 1950 in the refresh area 1910 immediately following the virtual boundary 1920, but the associated luma pixels (including q i,j , i = 0, 1, 2, 3 and j = 0, 1) are filled from refresh area 1910 (or set to a predetermined value, such as 2 BD-1 , where BD is the bit depth).

[0183] In the case where the use of the information of the first region is allowed, CCALF for pixels on the second region may be performed normally.

[0184] If the pixel p in the first area 1910 i,jThe actual CCALF filtering is different from the normal deblocking filtering, then the difference can be seen from the corresponding pixel q i,j Offset is

[0185]

[0186] in Q m,n The final output of the CCALF filter is Q m,n The output of the CCALF filter, To use all necessary information including first area and / or second area information i,j Perform normal CCALF filtering on the output, and For p i,j The actual CCALF filtered output.

[0187] like Fig.19A As shown, the corresponding pixel p i,j and q i,j 1910 and 1930 are mirrored pixels of the first region 1910 and the second region 1930 relative to the virtual boundary 1920 before CCALF.

[0188] If the second region chooses not to use the encoding information of the first region, CCALF is not applied to pixels in the second region that are at most n (eg, 1 for VVC or 4 for ECM) pixel positions away from the virtual boundary. Fig.19B An example is shown where a non-refreshed area (second area) 1980 of a GDR / restored image chooses not to use the encoding information of a refreshed area (first area) 1960. For co-located chroma pixels 1990 in the non-refreshed area 1980 that are next to a virtual boundary 1970, CCALF is skipped.

[0189] Alternatively, CCALF is still applied to those pixels in the second region 1980 that are immediately adjacent to the virtual boundary 1970, but utilizing the coded information in the first region 1960 derived from the second region 1980 or set to a predetermined value when necessary. Fig.19B In the example, CCALF is still applied to the co-located chrominance pixels 1990 in the non-refreshed region 1980 immediately following the virtual boundary 1970, but the associated luma pixels (including p i,j , i=0,1,2,3 and j=0,1) are filled with the unrefreshed area 1980.

[0190] Fig. 207 is a block diagram 700 of an apparatus 710 suitable for implementing an example embodiment. A non-limiting example of an apparatus 710 is a wireless, typically mobile device that can access a wireless network. The apparatus 710 includes one or more processors 720, one or more memories 725, one or more transceivers 730, and one or more network (N / W) interfaces (I / F) 761, which are interconnected by one or more buses 727. Each of the one or more transceivers 730 includes a receiver Rx 732 and a transmitter Tx 733. The one or more buses 727 can be an address, data, or control bus, and can include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, optical fiber or other optical communication device, etc.

[0191] Device 710 can communicate via wired, wireless, or both interfaces. For wireless communication, one or more transceivers 730 are connected to one or more antennas 728. One or more memories 725 include computer program code 723. (Multiple) N / WI / F 761 communicate via one or more wired links 762.

[0192] The device 710 includes a control module 740, which includes one or both of parts 740-1 and / or 740-2, and the control module 740 includes a reference 790, which includes a codec of an encoder 780, a decoder 782, or both 780 / 782, and can be implemented in a variety of ways. For ease of reference, the reference 790 is referred to as a codec in this article. The control module 740 can be implemented in hardware as a control module 740-1, such as being implemented as part of one or more processors 720. The control module 740-1 can also be implemented as an integrated circuit or by other hardware (such as a programmable gate array). In another example, the control module 740 can be implemented as a control module 740-2, which is implemented as a computer program code 723 and executed by one or more processors 720. For example, one or more memories 725 and computer program code 723 can be configured to use the user device 710 with one or more processors 720 to perform one or more operations described herein. Codec 790 may similarly be implemented as codec 790-1 as part of control module 740-1, or codec 790-2 as part of control module 740-2, or both.

[0193] The computer-readable memory 725 may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, firmware, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The computer-readable memory 725 may be a component for performing a storage function. The computer-readable memory(s) 725 may be non-transitory, transient, volatile (e.g., random access memory (RAM)) or non-volatile (e.g., read-only memory (ROM)). The computer-readable memory(s) 725 may include a database for storing data.

[0194] The processor 720 may be any type suitable for the local technical environment, and may include, as non-limiting examples, one or more of the following: a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture. The processor 720 may be a component for performing functions such as controlling the device 710 and other functions described herein.

[0195] In general, various embodiments of the device 710 may include, but are not limited to, a cellular phone (such as a smartphone, a mobile phone, a cellular phone, a Voice over Internet Protocol (IP) (VoIP) phone, and / or a wireless local loop phone), a tablet, a portable computer, an indoor audio device, an immersive audio device, a vehicle or vehicle-mounted device (e.g., for wireless V2X (vehicle-to-everything) communication), an image capture device (such as a digital camera), a gaming device, a music storage and playback device, an Internet device (including an Internet of Things (IoT) device), an IoT device with sensors and / or actuators (e.g., for automation applications), and a portable unit or terminal incorporating a combination of these functionalities, a laptop, a laptop embedded device (LEE), a laptop mounted device (LME), a universal serial bus (USB) dongle, a smart device, a wireless client equipment (CPE), an Internet of Things (IoT) device, a watch or other wearable device, a head mounted display (HMD), a vehicle, a drone, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of an industrial and / or automation process chain), a consumer electronic device, devices operating on a commercial and / or industrial wireless network, and the like. That is, the apparatus 710 may be any device capable of wireless or wired communication.

[0196] Thus, the apparatus 710 includes a processor 720, at least one memory 725 including computer program code 723, wherein the at least one memory 725 and the computer program code 723 are configured to, together with the at least one processor 720, cause the apparatus 710 to implement an asymmetric loop filter 790 at a virtual boundary based on the examples described herein. The apparatus 710 optionally includes a display or I / O 770, which can be used to display content during ML / task / machine / NN processing or rendering. The display or I / O 770 can be configured to receive input from a user, such as using a keyboard, a touch screen, a touch area, a microphone, biometrics, one or more sensors, etc. The apparatus 710 may include standard well-known components such as amplifiers, filters, frequency converters, and (de)modulators.

[0197] The computer program code 723 may include object-oriented software and may implement the filtering described throughout this disclosure. The apparatus 710 need not include each of the features mentioned, or may include other features. The apparatus 710 may be Figure 1 , Figure 2 , Figure 3 or Figure 4 The embodiments of the devices shown in include any combination of these devices.

[0198] Fig.21 An example method 2100 for implementing an asymmetric loop filter at a virtual boundary based on the examples described herein. At 2110, the method includes determining a virtual boundary that separates an image or a portion of an image into a first region and a second region. At 2120, the method includes determining to perform filtering of at least one pixel of the first region using encoding information of the second region derived from the first region, or using encoding information of the second region set to at least one value, when encoding information of the second region is to be used to perform filtering of the at least one pixel of the first region, or determining not to perform filtering of at least one pixel of the first region when the encoding information of the second region is to be used for filtering of at least one pixel of the first region. Method 2100 may be performed by an encoder, a decoder, or a codec or Figure 1 , Figure 2 , Figure 3 , Figure 4 or Fig. 20 Executed by any of the means shown in .

[0199] References to "computers," "processors," and the like should be understood to include not only computers having different architectures (such as single / multi-processor architectures and sequential / parallel architectures), but also special-purpose circuits (such as field programmable gate arrays (FPGAs), application-specific circuits (ASICs), signal processing devices, and other processing circuit systems). References to computer programs, instructions, code, and the like should be understood to encompass software for programmable processors or firmware, such as, for example, the programmable contents of a hardware device (such as instructions for a processor) or configuration settings for a fixed-function device, gate array, or programmable logic device, and the like.

[0200] As used herein, the terms "circuitry", "circuitry" and variations thereof may refer to any of the following: (a) hardware circuit implementations, such as implementations in analog and / or digital circuitry, and (b) combinations of circuitry and software (and / or firmware), such as, as applicable: (i) a combination of (multiple) processors or (ii) portions of (multiple) processors / software, including (multiple) digital signal processors, software and (multiple) memories, which work together to enable the device to perform various functions, and (c) circuitry, such as (multiple) microprocessors or portions of (multiple) microprocessors, which require software or firmware to operate, even if the software or firmware is not physically present. As another example, as used herein, the term "circuitry" would also cover implementations of only a processor (or multiple processors) or a portion of a processor and its (or their) accompanying software and / or firmware. The term "circuitry" would also cover, for example and if applicable to the particular element, a baseband integrated circuit or an application processor integrated circuit of a mobile phone or a similar integrated circuit in a server, cellular network device or other network device. Circuitry or circuitry may also be used to represent a function or process used to perform a method.

[0201] The following examples (1-32) are described and provided herein.

[0202] Example 1. A device comprising: at least one processor; and at least one memory, comprising computer program code; wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the device to at least: determine a virtual boundary that separates an image or a portion of an image into a first region and a second region; and when encoding information of the second region is to be used to perform filtering of at least one pixel of the first region, determine to perform filtering of at least one pixel of the first region using encoding information of the second region derived from the first region, or using encoding information of the second region set to at least one value, or when encoding information of the second region is to be used to perform filtering of at least one pixel of the first region, determine not to perform filtering of at least one pixel of the first region.

[0203] Example 2. An apparatus according to Example 1, wherein the filtering of at least one pixel of the first region includes loop filtering.

[0204] Example 3. The apparatus of any of Examples 1 to 2, wherein the first region comprises a refresh region and the second region comprises a non-refresh region.

[0205] Example 4. An apparatus according to any one of Examples 1 to 3, wherein the image includes a step-by-step decoding refresh image or a recovery image.

[0206] Example 5. An apparatus according to any one of Examples 1 to 4, wherein at least one memory and computer program code are configured, together with at least one processor, so that the apparatus at least: fills pixels in the second area from pixels in the first area in response to pixels in the second area being used to perform filtering of at least one pixel in the first area.

[0207] Example 6. An apparatus according to any one of Examples 1 to 5, wherein at least one memory and computer program code are configured, together with at least one processor, so that the apparatus at least: replaces pixels in the second area with pixels extrapolated from the first area in response to pixels in the second area being used to perform filtering of at least one pixel in the first area.

[0208] Example 7. An apparatus according to any one of Examples 1 to 6, wherein at least one memory and computer program code are configured to, together with at least one processor, cause the apparatus to at least: determine a first output of filtering of at least one pixel of the first region when encoding information of the first region and encoding information of the second region can be used for the filtering of at least one pixel of the first region; and determine a second output of filtering of at least one pixel of the first region when encoding information of the second region cannot be used for filtering of at least one pixel of the first region.

[0209] Example 8. An apparatus according to Example 7, wherein at least one memory and computer program code are configured to, together with at least one processor, cause the apparatus to at least: determine a difference between the first output and the second output; and determine an output of filtering at least one pixel of the second region using, at least in part, the difference or an approximation of the difference.

[0210] Example 9. An apparatus according to Example 8, wherein the encoding information of the second region includes an output of filtering of at least one pixel of the second region.

[0211] Example 10. An apparatus according to any of Examples 8 to 9, wherein the position of at least one pixel of the second region corresponds to the position of at least one pixel of the first region.

[0212] Example 11. An apparatus according to any one of Examples 7 to 10, wherein at least one memory and computer program code are configured to, together with at least one processor, cause the apparatus to at least: determine a difference between a first output and a second output; determine an initial output of filtering at least one pixel of the second region; and determine a final output of filtering at least one pixel of the second region by at least partially subtracting the difference from the initial output; wherein the encoding information of the second region: includes the final output of filtering at least one pixel of the second region.

[0213] Example 12. An apparatus according to Example 11, wherein at least one memory and computer program code are configured to, together with at least one processor, cause the apparatus to at least: determine a filtered final output of at least one pixel of the second region by at least partially subtracting a weighted contribution of the difference from the initial output.

[0214] Example 13. The apparatus of Example 12, wherein the weighted contribution comprises 1 / 2 i , where i corresponds to the index of the position of at least one pixel of the first region or at least one pixel of the second region.

[0215] Example 14. An apparatus according to any one of Examples 1 to 13, wherein at least one memory and computer program code are configured to, together with at least one processor, cause the apparatus to at least: determine a target output for a target filtering of at least one pixel of a first region; determine an actual output for filtering of at least one pixel of the first region when encoding information of the first region or encoding information of the second region cannot be used to perform filtering of at least one pixel of the first region; determine a difference between the target output and the actual output; determine an initial output for filtering at least one pixel of the second region; and determine a final output for filtering of at least one pixel of the second region using at least in part an initial output offset having a difference; wherein the encoding information of the second region includes the final output for filtering of at least one pixel of the second region.

[0216] Example 15. An apparatus according to any one of Examples 1 to 14, wherein at least one memory and computer program code are configured to, together with at least one processor, cause the apparatus to at least: determine whether to perform filtering of at least one pixel of the first area without using encoding information of the second area, and determine whether to perform filtering of at least one pixel of the second area without using encoding information of the first area, in response to determining a common option related to filtering of at least one pixel of the first area and filtering of at least one pixel of the second area.

[0217] Example 16. The apparatus of Example 15, wherein the common options include determining not to perform filtering of at least one pixel of the first region, and determining not to perform filtering of at least one pixel of the second region.

[0218] Example 17. An apparatus according to any one of Examples 15 to 16, wherein the common options include determining to perform filtering of at least one pixel of the first region using the encoded information filling the second region, and determining to perform filtering of at least one pixel of the second region using the encoded information filling the first region.

[0219] Example 18. An apparatus according to any one of Examples 1 to 17, wherein at least one memory and computer program code are configured, together with at least one processor, so that the apparatus at least: uses encoding information of the first region and encoding information of the second region to perform filtering of at least one pixel of the second region.

[0220] Example 19. An apparatus according to any one of Examples 1 to 18, wherein the filtering of at least one pixel of the first region includes at least one of the following items: deblocking filtering; sample adaptive offset edge offset filtering; bilateral filtering for luminance; bilateral filtering for chrominance; cross-component sample adaptive offset filtering; adaptive loop filtering; or cross-component adaptive loop filtering.

[0221] Example 20. An apparatus according to any one of Examples 1 to 19, wherein at least one memory and computer program code are configured, together with at least one processor, to cause the apparatus to at least: disable filtering of at least one pixel in the first region until a plurality of pixel positions away from the virtual boundary.

[0222] Example 21. An apparatus according to any one of Examples 1 to 20, wherein at least one memory and computer program code are configured, together with at least one processor, to cause the apparatus to at least: perform filtering of at least one pixel of the first area until a plurality of numbers of pixel positions away from the virtual boundary.

[0223] Example 22. An apparatus according to any one of Examples 1 to 21, wherein at least one memory and computer program code are configured to, together with at least one processor, cause the apparatus to perform at least one of the following items: setting pixel values ​​of the second region equal to pixel values ​​of the first region immediately adjacent to the virtual boundary; setting pixel values ​​of the second region equal to the average value of the pixel values ​​of the first region; or setting pixel values ​​of the second region equal to the median value of the pixel values ​​of the first region; wherein the encoding information of the second region includes the set pixel values ​​of the second region.

[0224] Example 23. An apparatus according to any one of Examples 1 to 22, wherein at least one memory and computer program code are configured to, together with at least one processor, cause the apparatus to at least: determine to perform filtering of at least one pixel of the second area using encoding information of the first area derived from the second area, or using encoding information of the first area set to at least one value, when encoding information of the first area is to be used to perform filtering of at least one pixel of the second area, or determine not to perform filtering of at least one pixel of the second area when encoding information of the first area is to be used to perform said filtering of at least one pixel of the second area.

[0225] Example 24. An apparatus according to Example 23, wherein at least one memory and computer program code are configured to, together with at least one processor, cause the apparatus to at least: disable filtering of the at least one pixel in the second region until a plurality of pixel positions away from the virtual boundary.

[0226] Example 25. An apparatus according to any one of Examples 23 to 24, wherein at least one memory and computer program code are configured, together with at least one processor, to cause the apparatus to at least: perform the filtering of the at least one pixel of the second area until a plurality of numbers of pixel positions away from the virtual boundary.

[0227] Example 26. An apparatus according to any one of Examples 23 to 25, wherein at least one memory and computer program code are configured to, together with at least one processor, cause the apparatus to perform at least one of the following items: setting pixel values ​​of a first region equal to pixel values ​​of a second region adjacent to a virtual boundary; setting pixel values ​​of a first region equal to an average value of pixel values ​​of a second region; or setting pixel values ​​of a first region equal to a median value of pixel values ​​of a second region; wherein the encoding information of the first region includes the set pixel values ​​of the first region.

[0228] Example 27. An apparatus according to any one of Examples 23 to 26, wherein filtering of at least one pixel of the second region includes at least one of the following: loop filtering; deblocking filtering; sample adaptive offset edge offset filtering; bilateral filtering for luminance; bilateral filtering for chrominance; cross-component sample adaptive offset filtering; adaptive loop filtering; or cross-component adaptive loop filtering.

[0229] Example 28. An apparatus according to any one of Examples 1 to 27, wherein at least one memory and computer program code are configured to, together with at least one processor, cause the apparatus to at least: determine at least one value having a bit depth BD.

[0230] Example 29. The apparatus of Example 28, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least: determine at least one value as 2 BD-1 .

[0231] Example 30. A method, comprising determining a virtual boundary that separates an image or a portion of an image into a first region and a second region; and when encoding information of the second region will be used to perform filtering of at least one pixel of the first region, determining to perform filtering of at least one pixel of the first region using encoding information of the second region derived from the first region, or using encoding information of the second region set to at least one value, or when encoding information of the second region will be used to perform filtering of at least one pixel of the first region, determining not to perform filtering of at least one pixel of the first region.

[0232] Example 31. A device comprising: a component for determining a virtual boundary that separates an image or a portion of an image into a first region and a second region; and a component for: determining to perform filtering of at least one pixel of the first region using encoding information of the second region derived from the first region or using encoding information of the second region set to at least one value when encoding information of the second region will be used to perform filtering of at least one pixel of the first region, or determining not to perform filtering of at least one pixel of the first region when encoding information of the second region will be used to perform filtering of at least one pixel of the first region.

[0233] Example 32. A non-volatile program storage device readable by a machine, tangibly embodying an instruction program executable by the machine to perform operations, the operations comprising determining a virtual boundary that separates an image or a portion of an image into a first region and a second region; and when encoding information of the second region is to be used to perform filtering of at least one pixel of the first region, determining to perform filtering of at least one pixel of the first region using the encoding information of the second region derived from the first region, or using the encoding information of the second region set to at least one value, or when the encoding information of the second region is to be used to perform filtering of at least one pixel of the first region, determining not to perform filtering of at least one pixel of the first region.

[0234] In the figures, arrows between individual blocks indicate operational couplings between them and the direction of data flow over those couplings.

[0235] It should be understood that the above description is illustrative only. Various alternatives and modifications can be designed by those skilled in the art. For example, the features described in each dependent claim can be combined with each other in any suitable (multiple) combination. In addition, the features in the above different embodiments can be selectively combined into new embodiments. Therefore, this description is intended to cover all such alternatives, modifications and variations that belong to the scope of the attached claims.

[0236] The following acronyms and abbreviations that may be found in the specification and / or drawings are defined as follows. Acronyms and abbreviations may be appended to each other and / or other characters (eg, a hyphen (-)).

[0237] 3GPP Third Generation Partnership Project

[0238] 4G Fourth generation broadband cellular network technology

[0239] 5G fifth generation cellular network technology

[0240] 802.x IEEE standard series, covering local area networks and metropolitan area networks ABC Alternative Band Classifier

[0241] ALF Adaptive Loop Filter

[0242] APS Adaptive Parameter Set

[0243] ASIC Application-Specific Integrated Circuit

[0244] BD bit depth

[0245] BIF Bilateral Filter

[0246] BIF-chroma Bilateral filter for chroma

[0247] BIF-luma Bilateral filter for luminance

[0248] BO with offset

[0249] Cb blue chromaticity component

[0250] CCALF or CC-ALF Cross-component ALF

[0251] CCSAO Cross Component SAO

[0252] CDMA Code Division Multiple Access

[0253] CMP Cube Map Projection

[0254] CPE

[0255] Cr Red chromaticity component

[0256] CTB Coded Tree Block

[0257] CTU Coding Tree Unit

[0258] DBF Deblocking Filter

[0259] DCT Discrete Cosine Transform

[0260] DSP Digital Signal Processor

[0261] ECM Enhanced Compression Model

[0262] EO Edge Offset

[0263] FDMA Frequency Division Multiple Access

[0264] FPGA Field Programmable Gate Array

[0265] GDR Gradual Decode Refresh

[0266] GSM Global System for Mobile Communications

[0267] H.222.0 MPEG-2 Systems, a generic coding standard for moving pictures and associated audio information

[0268] H.26x ITU-T video coding standard series

[0269] HMD Head-mounted Display

[0270] IBC Intra-block Replication

[0271] id or ID identifier

[0272] IEC International Electrotechnical Commission

[0273] IEEE Institute of Electrical and Electronics Engineers

[0274] I / F Interface

[0275] IMD Integrated Messaging Device

[0276] IMS Instant Messaging Service

[0277] I / O Input and Output

[0278] IoT

[0279] IP Internet Protocol

[0280] ISO International Organization for Standardization

[0281] ISOBMFF ISO Base Media File Format

[0282] ITU International Telecommunication Union

[0283] ITU-T International Telecommunication Union Telecommunication Standardization Sector

[0284] JTC Joint Technical Committee

[0285] JVET Joint Video Experts Team

[0286] LEE Laptop Embedded Devices

[0287] LME Laptop Mounting Equipment

[0288] LTE Long Term Evolution

[0289] ML Machine Learning

[0290] MMS Multimedia Messaging Service

[0291] MPEG Moving Picture Experts Group

[0292] MPEG-2 such as H.222 / H.262 defined by ITU

[0293] MSE Mean Square Error

[0294] MV Multi-View

[0295] NAL Network Abstraction Layer

[0296] NN Neural Network

[0297] N / W Network

[0298] PC Personal Computer

[0299] PDA Personal Digital Assistant

[0300] PID Packet Identifier

[0301] PLC Power Line Communication

[0302] QP Quantization Parameter or Quarter Pixel

[0303] RAM Random Access Memory

[0304] RFID Radio Frequency Identification

[0305] RFM Reference Frame Memory

[0306] ROM Read Only Memory

[0307] Rx Receiver

[0308] SAO Sample Adaptive Offset

[0309] SMS text messaging service

[0310] SPS Sequence Parameter Set

[0311] TCP-IP Transmission Control Protocol-Internet Protocol

[0312] TDMA Time Division Multiple Access

[0313] TS Transport Stream

[0314] TV

[0315] Tx Transmitter

[0316] U blue projection of the chroma component

[0317] UICC Universal Integrated Circuit Card

[0318] UMTS Universal Mobile Telecommunications System

[0319] USB Universal Serial Bus

[0320] V Red projection of the chroma component

[0321] V2X

[0322] VoIP Voice over IP

[0323] VVC General Video Coding

[0324] WLAN Wireless Local Area Network

[0325] Y Luminance component

Claims

1. A device comprising: at least one processor; as well as at least one memory including computer program code; wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: determining a virtual boundary separating an image or a portion of the image into a first region and a second region; as well as When the encoding information of the second area is to be used to perform filtering of at least one pixel of the first area, determine to perform filtering of the at least one pixel of the first area using the encoding information of the second area derived from the first area, or using the encoding information of the second area set to at least one value, or when the encoding information of the second area is to be used to perform filtering of the at least one pixel of the first area, determine not to perform filtering of the at least one pixel of the first area. 2 . The apparatus of claim 1 , wherein the filtering of the at least one pixel of the first region comprises loop filtering. 3 . The apparatus according to claim 1 , wherein the first region comprises a refresh region, and the second region comprises a non-refresh region.

4. The apparatus according to any one of claims 1 to 3, wherein the image comprises a progressive decoding refresh image or a recovery image.

5. The apparatus according to any one of claims 1 to 4, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: In response to the pixels in the second region being used to perform the filtering of the at least one pixel of the first region, the pixels in the second region are filled from the pixels in the first region.

6. The apparatus according to any one of claims 1 to 5, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: In response to pixels in the second region being used to perform the filtering of the at least one pixel in the first region, pixels in the second region are replaced with pixels extrapolated from the first region.

7. The apparatus according to any one of claims 1 to 6, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: When the encoding information of the first area and the encoding information of the second area can be used for the filtering of the at least one pixel of the first area, determining a first output of the filtering of the at least one pixel of the first area; and When the encoding information of the second region cannot be used for the filtering of the at least one pixel of the first region, a second output of the filtering of the at least one pixel of the first region is determined.

8. The apparatus of claim 7, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: determining a difference between the first output and the second output; and An output of filtering at least one pixel of the second region is determined using, at least in part, the difference or an approximation of the difference. 9 . The apparatus of claim 8 , wherein the encoding information of the second region comprises the output of the filtering of the at least one pixel of the second region.

10. The apparatus according to any one of claims 8 to 9, wherein the position of the at least one pixel of the second area corresponds to the position of the at least one pixel of the first area.

11. The apparatus according to any one of claims 7 to 10, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: determining a difference between the first output and the second output; determining an initial output of filtering at least one pixel of the second region; as well as determining a final output of said filtering for said at least one pixel of said second region by at least partially subtracting said difference from said initial output; The encoding information of the second area includes: the final output of the filtering of the at least one pixel in the second area.

12. The apparatus of claim 11, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: The final output of the filtering of the at least one pixel of the second region is determined by at least partially subtracting a weighted contribution of the difference from the initial output.

13. The apparatus of claim 12, wherein the weighted contribution comprises 1 / 2 i , where i corresponds to the index of the position of the at least one pixel of the first area or the at least one pixel of the second area.

14. The apparatus according to any one of claims 1 to 13, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: determining a target output of a target filter for the at least one pixel of the first region; When the encoding information of the first region or the encoding information of the second region cannot be used to perform the filtering of the at least one pixel of the first region, determining an actual output of the filtering of the at least one pixel of the first region; determining a difference between the target output and the actual output; determining an initial output of filtering at least one pixel of the second region; as well as determining a final output of the filtering for the at least one pixel of the second region using, at least in part, the initial output offset having the difference; The encoding information of the second area includes the final output of the filtering of the at least one pixel of the second area.

15. The apparatus according to any one of claims 1 to 14, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: In response to determining a common option related to the filtering of the at least one pixel of the first area and the filtering of the at least one pixel of the second area, determining whether to perform the filtering of the at least one pixel of the first area without using the encoding information of the second area, and determining whether to perform the filtering of the at least one pixel of the second area without using the encoding information of the first area. 16 . The apparatus of claim 15 , wherein the common options include determining not to perform filtering of the at least one pixel of the first region, and determining not to perform filtering of the at least one pixel of the second region.

17. The apparatus according to any one of claims 15 to 16, wherein the public options include: Determine to use the encoding information filling the second area to perform filtering of the at least one pixel of the first area, and determine to use the encoding information filling the first area to perform filtering of the at least one pixel of the second area.

18. The apparatus according to any one of claims 1 to 17, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: Filtering of at least one pixel of the second region is performed using the encoding information of the first region and the encoding information of the second region.

19. The apparatus according to any one of claims 1 to 18, wherein the filtering of the at least one pixel of the first region comprises at least one of the following: Deblocking filtering; Sample adaptive offset edge offset filtering; Bilateral filtering for brightness; Bilateral filtering for chroma; Cross-component sampling adaptive offset filtering; Adaptive loop filtering; or Cross-component adaptive loop filtering.

20. The apparatus according to any one of claims 1 to 19, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: Disabling filtering of the at least one pixel in the first region up to a location a plurality of pixels away from the virtual boundary.

21. The apparatus according to any one of claims 1 to 20, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: Filtering of the at least one pixel of the first region is performed up to a plurality of number of pixel locations from the virtual boundary.

22. The apparatus according to any one of claims 1 to 21, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least perform at least one of the following: Setting the pixel value of the second area to be equal to the pixel value of the first area immediately adjacent to the virtual boundary; setting the pixel value of the second area to be equal to the average value of the pixel values ​​of the first area; or Setting the pixel value of the second area to be equal to the median value of the pixel values ​​of the first area; The encoding information of the second area includes the set pixel value of the second area.

23. The apparatus according to any one of claims 1 to 22, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: When the encoding information of the first area is to be used to perform the filtering of the at least one pixel of the second area, determine to perform the filtering of the at least one pixel of the second area using the encoding information of the first area derived from the second area, or using the encoding information of the first area set to at least one value, or when the encoding information of the first area is to be used to perform the filtering of the at least one pixel of the second area, determine not to perform the filtering of the at least one pixel of the second area.

24. The apparatus of claim 23, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least: Disabling filtering of the at least one pixel in the second region up to a position a plurality of pixels away from the virtual boundary.

25. The apparatus according to any one of claims 23 to 24, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: The filtering of the at least one pixel of the second region is performed up to a plurality of number of pixel positions from the virtual boundary.

26. The apparatus according to any one of claims 23 to 25, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least perform at least one of the following: Setting the pixel value of the first area equal to the pixel value of the second area immediately adjacent to the virtual boundary; setting the pixel value of the first area to be equal to the average value of the pixel values ​​of the second area; or Setting the pixel value of the first area to be equal to the median value of the pixel value of the second area; The encoding information of the first area includes the set pixel value of the first area.

27. The apparatus of any one of claims 23 to 26, wherein the filtering of the at least one pixel of the second region comprises at least one of the following: Loop filtering; Deblocking filtering; Sample adaptive offset edge offset filtering; Bilateral filtering for brightness; Bilateral filtering for chroma; Cross-component sampling adaptive offset filtering; Adaptive loop filtering; or Cross-component adaptive loop filtering.

28. The apparatus according to any one of claims 1 to 27, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least: At least one value having a bit depth BD is determined.

29. The apparatus of claim 28, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least: The at least one value is determined to be 2 BD-1 .

30. A method comprising: determining a virtual boundary separating an image or a portion of the image into a first region and a second region; as well as When the encoding information of the second area is to be used to perform filtering of at least one pixel of the first area, determine to perform filtering of the at least one pixel of the first area using the encoding information of the second area derived from the first area, or using the encoding information of the second area set to at least one value, or when the encoding information of the second area is to be used to perform filtering of the at least one pixel of the first area, determine not to perform filtering of the at least one pixel of the first area.

31. An apparatus comprising: means for determining a virtual boundary separating an image or a portion of said image into a first region and a second region; as well as A component for the following items: when the encoding information of the second area is to be used to perform filtering of at least one pixel of the first area, determining to use the encoding information of the second area derived from the first area, or using the encoding information of the second area set to at least one value to perform filtering of the at least one pixel of the first area, or when the encoding information of the second area is to be used to perform filtering of the at least one pixel of the first area, determining not to perform filtering of the at least one pixel of the first area.

32. A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform operations comprising: determining a virtual boundary separating an image or a portion of the image into a first region and a second region; as well as When the encoding information of the second area is to be used to perform filtering of at least one pixel of the first area, determine to perform filtering of the at least one pixel of the first area using the encoding information of the second area derived from the first area, or using the encoding information of the second area set to at least one value, or when the encoding information of the second area is to be used to perform filtering of the at least one pixel of the first area, determine not to perform filtering of the at least one pixel of the first area.

Images