In-loop filtering at virtual boundaries for flexible GDR

Abstract

The method includes determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary; determining at least one of a first condition in which both sides of a first segment at the first virtual boundary are within an updated area or both sides of the first segment are within an unupdated area, or a second condition in which both sides of the first segment at the first virtual boundary are not within an updated area or both sides of the first segment are not within an unupdated area; performing a first type of filtering on at least one pixel adjacent to the first segment based on the determination of the first condition; and performing a second different type of filtering on at least one pixel adjacent to the first segment based on the determination of the second condition.

Description

Technical Field

[0001] Embodiments and non-limiting examples generally relate to multimedia transport and the encoding and decoding of information, and more particularly to in-loop filters at virtual boundaries.

Background Art

[0002] In multimedia systems, it is known to perform data compression and decoding.

[0003] The foregoing aspects and other features will be described in the following description taken in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0004] [Figure 1] A diagram schematically showing an electronic device adopting an embodiment of the example described herein. [Figure 2] A diagram schematically showing a user device suitable for adopting an embodiment of the example described herein. [[ID=ID=29]] [Figure 3] A diagram further schematically showing an electronic device adopting an embodiment of the example described herein, connected using a wireless network connection and a wired network connection. [Figure 4] A diagram schematically showing a block chart of an encoder used for data compression at a general level. [Figure 5] A diagram showing that an updated area is not permitted to use the encoded information of an unupdated area. [Figure 6] A diagram showing that an unupdated area is permitted to use the encoded information of an updated area. [Figure 7]This figure shows that deblocking may not be applied to pixels pi,i=0,1,2, or that pixels qi,i=0,1,2 in the unupdated area may be filled in and still be applicable. [Figure 8] This diagram shows the four edge classes. [Figure 9] This diagram shows the four edge categories. [Figure 10A] This figure shows that the SAO edge offset may not be applied to pixel p0, or it may still be applied if pixel q0 is filled in. [Figure 10B] This figure shows that the SAO edge offset may not be applied to pixel q0, or it may still be applied if pixel p0 is filled in. [Figure 11] This figure shows that the BIF luminance (BIF-luma), SAO, and offsets from CCSAO are added to the deblocking output. [Figure 12A] This figure shows that BIF may not be applied to pixel p0,0, or that the associated pixels, including qi,0,i=0,1, may be filled in and still be applicable. [Figure 12B] This figure shows that BIF may not be applied to pixel q0,0, or that the associated pixels, including pi,0,i=0,1, may be filled in and it may still be applied. [Figure 13] This diagram shows the CCSAO decoding workflow. [Figure 14] This figure shows that, with respect to juxtaposed saturation samples, juxtaposed luminance samples can be selected from nine candidate positions. [Figure 15A] This figure shows that CCSAO may not be applied to pixel p0, or that pixel q0 may be filled in, allowing it to still be applied. [Figure 15B] This figure shows that CCSAO may not be applied to pixel q0, or that pixel p0 may be filled and it may still be applicable. [Figure 16A] This figure shows that ALF may not be applied to pixel p0,0, or that the associated pixels, including qi,0,i=0,1,2, may be filled in and still be applicable. [Figure 16B] This figure shows that ALF may not be applied to pixels q0,0, or that the associated pixels, including pi,0,i=0,1,2, may be filled in, and therefore it may still be applied. [Figure 17] This is a basic diagram of CCALF in VVC. [Figure 18] This figure shows a 25-tap filter for CCALF in ECM. [Figure 19A] This figure shows that CCALF may not be applied to at least one of the juxtaposed saturation pixels, or that luminance pixels qi,j,i=0,1,2,3 and j=0,1 may be filled and still be applicable. [Figure 19B] This figure shows that CCALF may not be applied to at least one of the juxtaposed saturation pixels, or that luminance pixels pi,j,i=0,1,2,3 and j=0,1 may be filled and still be applicable. [Figure 20] This figure shows an exemplary apparatus configured to perform asymmetric in-loop filtering at a virtual boundary, based on the embodiments described herein. [Figure 21] This figure shows an exemplary method for implementing an asymmetric in-loop filter at a virtual boundary, based on the examples described herein. [Figure 22] This figure shows the current GDR design in the VVC / ECM design, where the initial updated area starts on the left side of the GDR image and then gradually expands to the right across the associated recovering image. [Figure 23] This figure shows the initial refreshed area (RA), which is located in the center of the GDR image and then gradually expands to the left and right of the associated recovering image. [Figure 24]This figure shows the initial updated area (RA), which is located in the center of the GDR image and then gradually expands in four directions across the associated recovering images. [Figure 25A-F] This figure shows an example of GDR where multiple virtual boundaries (VB) divide the GDR image / recovery image into updated areas (RA) and unupdated areas (blank). [Figure 26] This diagram illustrates one example method. [Figure 27] This diagram illustrates one example method. [Overview of the project]

[0005] The following summary is intended to be illustrative only. This summary is not intended to limit the scope of the claims.

[0006] According to one embodiment, an exemplary method is provided which involves determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary; determining at least one of a first condition in which both sides of a first segment at the first virtual boundary are within an updated area or both sides of the first segment are within an unupdated area, or a second condition in which both sides of the first segment at the first virtual boundary are not within an updated area or both sides of the first segment are not within an unupdated area; performing a first type of filtering on at least one pixel adjacent to the first segment based on the determination of the first condition; and performing a second different type of filtering on at least one pixel adjacent to the first segment based on the determination of the second condition.

[0007] In another embodiment, an exemplary apparatus is provided comprising at least one processor and at least one non-transient memory storing instructions, the instructions, when executed by at least one processor, cause the apparatus to determine a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary; determine at least one of a first condition in which both sides of a first segment at the first virtual boundary are in an updated area or both sides of the first segment are in an unupdated area, or a second condition in which both sides of the first segment at the first virtual boundary are not in an updated area or both sides of the first segment are not in an unupdated area; perform a first type of filtering of at least one pixel adjacent to the first segment based on the determination of the first condition; and perform a second different type of filtering of at least one pixel adjacent to the first segment based on the determination of the second condition.

[0008] In another embodiment, an exemplary apparatus is provided comprising: means for determining a plurality of virtual boundaries with respect to at least a portion of an image, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary; means for determining at least one of a first condition in which both sides of a first segment at the first virtual boundary are within an updated area or both sides of the first segment are within an unupdated area, or a second condition in which both sides of the first segment at the first virtual boundary are not within an updated area or both sides of the first segment are not within an unupdated area; means for performing a first type of filtering of at least one pixel adjacent to the first segment based on the determination of the first condition; and means for performing a second different type of filtering of at least one pixel adjacent to the first segment based on the determination of the second condition.

[0009] In another embodiment, exemplary embodiments are provided that include a device-readable non-transient program storage device that tangibly embodies a program of instructions executable by the device to perform an operation, the operation being to determine a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary; to determine at least one of a first condition in which both sides of a first segment at the first virtual boundary are in an updated area or both sides of the first segment are in an unupdated area, or a second condition in which both sides of the first segment at the first virtual boundary are not in an updated area or both sides of the first segment are not in an unupdated area; to perform a first type of filtering of at least one pixel adjacent to the first segment based on the determination of the first condition; and to perform a second different type of filtering of at least one pixel adjacent to the first segment based on the determination of the second condition.

[0010] In another embodiment, an exemplary method is provided for determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary; performing filtering of pixels adjacent to the first virtual boundary as the first virtual boundary moves in a first direction; and performing filtering of pixels adjacent to the second virtual boundary as the second virtual boundary moves in a second direction.

[0011] In another embodiment, an exemplary apparatus is provided comprising at least one processor and at least one non-transient memory storing instructions, the apparatus, when executed by at least one processor, causes the apparatus to determine a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary; to perform filtering of pixels adjacent to the first virtual boundary as the first virtual boundary moves in a first direction; and to perform filtering of pixels adjacent to the second virtual boundary as the second virtual boundary moves in a second direction.

[0012] In another embodiment, an exemplary apparatus is provided comprising means for determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary; means for performing filtering of pixels adjacent to the first virtual boundary as the first virtual boundary moves in a first direction; and means for performing filtering of pixels adjacent to the second virtual boundary as the second virtual boundary moves in a second direction.

[0013] In another embodiment, an exemplary apparatus is provided comprising means for determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary; means for performing filtering of pixels adjacent to the first virtual boundary as the first virtual boundary moves in a first direction; and means for performing filtering of pixels adjacent to the second virtual boundary as the second virtual boundary moves in a second direction.

[0014] In another embodiment, exemplary embodiments are provided that include a device-readable non-transient program storage device that tangibly embodies a program of instructions executable by the device to perform operations, the operations of which include determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary; performing filtering of pixels adjacent to the first virtual boundary as the first virtual boundary moves in a first direction; and performing filtering of pixels adjacent to the second virtual boundary as the second virtual boundary moves in a second direction.

[0015] In another embodiment, an exemplary method is provided which includes determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a second virtual boundary; determining at least one of a first condition, where both sides of a first segment in the first virtual boundary are in an updated area or both sides of the first segment are in an unupdated area, or a second condition, where one side of the first segment in the first virtual boundary is in an updated area and the other side of the first segment in the first virtual boundary is in an unupdated area; performing a first type of filtering on at least one pixel of the first segment based on the determination of the first condition; and performing a second different type of filtering on at least one pixel of the first segment based on the determination of the second condition.

[0016] In another embodiment, an exemplary apparatus is provided comprising at least one processor and at least one non-transient memory storing instructions, the instructions, when executed by at least one processor, cause the apparatus to determine a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a second virtual boundary; determine at least one of a first condition, where both sides of a first segment at the first virtual boundary are in an updated area or both sides of the first segment are in an unupdated area, or a second condition, where one side of the first segment at the first virtual boundary is in an updated area and the other side of the first segment at the first virtual boundary is in an unupdated area; perform a first type of filtering on at least one pixel of the first segment based on the determination of the first condition; and perform a second different type of filtering on at least one pixel of the first segment based on the determination of the second condition.

[0017] In another embodiment, an exemplary apparatus is provided comprising: means for determining a plurality of virtual boundaries with respect to at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a second virtual boundary; means for determining at least one of a first condition, where both sides of a first segment in the first virtual boundary are within an updated area or both sides of the first segment are within an unupdated area, or a second condition, where one side of the first segment in the first virtual boundary is within an updated area and the other side of the first segment in the first virtual boundary is within an unupdated area; means for performing a first type of filtering on at least one pixel of the first segment based on the determination of the first condition; and means for performing a second different type of filtering on at least one pixel of the first segment based on the determination of the second condition.

[0018] In another embodiment, exemplary embodiments are provided that include a device-readable non-transient program storage device that tangibly embodies a program of instructions executable by the device to perform an operation, the operation comprising determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a second virtual boundary; determining at least one of a first condition, where both sides of a first segment in the first virtual boundary are in an updated area or both sides of the first segment are in an unupdated area, or a second condition, where one side of the first segment in the first virtual boundary is in an updated area and the other side of the first segment in the first virtual boundary is in an unupdated area; performing a first kind of filtering on at least one pixel of the first segment based on the determination of the first condition; and performing a second different kind of filtering on at least one pixel of the first segment based on the determination of the second condition.

[0019] In some embodiments, the subject matter of an independent claim is provided. In some further embodiments, the subject matter of a dependent claim is provided. [Modes for carrying out the invention]

[0020] This specification describes practical techniques for implementing in-loop filtering at virtual boundaries. The models described herein may be used to perform any task, such as data compression, data recovery, video compression, video recovery, image or video classification, object classification, object detection, object tracking, speech recognition, language translation, and music transcription.

[0021] The following acronyms and abbreviations, which may appear in this specification and / or in the drawings, are defined as follows: Acronyms and abbreviations may be added to each other and / or to each other with other characters (e.g., hyphens (-)). 3GPP(3rd generation partnership project) Third Generation Partnership Project 4G (fourth generation) The fourth generation of broadband cellular network technology 5G (fifth generation) Fifth-generation cellular network technology 802.x Local Area Networks and Metropolitan Area Networks A group of corresponding IEEE standards ABC(alternative band classifier) Alternative band classifier ALF (Adaptive Loop Filter) Adaptive Loop Filter APS (adaptation parameter set) Adaptive parameter set ASIC (Application-Specific Integrated Circuit) BD (bit technology) Bit depth BIF (Bilateral Filter) BIF Saturation: A bilateral filter for saturation. BIF Brightness: Bilateral filter for brightness. BO band offset Cb blue chrominance component CCALF (cross-component ALF) or CC-ALF Intercomponent ALF CCSAO (cross-component SAO) SAO between components CDMA Code Division Multiple Access CMP (cube-map projection) Cube Map Projection CPE (Customer Premises Equipment) Cr (red chrominance component) CTB (Coding Primary Block) Encoding tree block CTU (Coding Tree Unit) Encoding tree unit DBF (deblocking filter) Deblocking filter DCT (Discrete Cosine Transform) DSP (Digital Signal Processor) ECM(enhanced compression model) Extended compression model EO edge offset FDMA Frequency Division Multiple Access FPGA Field-Programmable Gate Array GDR (gradual decoding refresh) Gradual decoding update GSM Global System for Mobile Communications H.222.0 General information about the MPEG-2 system, video, and related audio information. Standards for coding H.26x is a group of video encoding standards within the ITU-T domain. HMD (Head-Mounted Display) IBC block copy ID or ID (identifier) identifier IEC (International Electrotechnical Commission) IEEE(Institute of Electrical and Electronics Engineers) Japan Institute of Electrical and Electronics Engineers I / F Interface IMD Integrated Messaging Device IMS Instant Messaging Service I / O (input output) Input / Output IoT (Internet of Things) IP Internet Protocol ISO(International Organization for Standardization) International Organization for Standardization ISOBMFF(ISO base media file format) ISO-based media file format ITU (International Telecommunication Union) ITU-T (ITU Telecommunication Standardization Sector) ITU Telecommunications Standardization Division JTC (joint technical committee) joint technical committee JVNET(joint video experts team) Joint video expert team LEE Laptop Embedded Devices LME Laptop Built-in Devices LTE (Long Term Evolution) Long-term evolution ML (machine learning) Machine Learning MMS (Multimedia Messaging Service) MPEG (moving picture experts group) Video expert group MPEG-2 H.222 / H.262 as defined by the ITU MSE mean square error MV (multiple views) Multiple Views NAL (network abstraction layer) Network Abstraction Layer NN (neural network) Neural Network Network PC (Personal Computer) PDA (Personal Digital Assistant) PID (Package Identifier) PLC (Power Line Communication) Power line communications QP (quantization parameter or quarter pixel) RAM (Random Access Memory) RFID(radio frequency identification) Wireless automatic identification RFM Reference Frame Memory ROM (Read-only memory) Rx receiver SAO (sample adaptive offset) Sample-adaptive offset SMS Short Message Service SPS (sequence parameter set) Sequence parameter set TCP-IP Transmission control protocol - Internet protocol TDMA Time-Division Multiple Access TS Transport Stream TV (television) tv set Tx transmitter Blue projection of the U chrominance component UICC(universal integrated circuit card) Universal Integrated Circuit Card UMTS Universal Mobile Telecommunications System USB Universal Serial Bus V Chrominance component red projection V2X Vehicle-to-Everything VoIP (Voice over IP) VVC (versatile video coding) Versatile video coding WLAN (Wireless Local Area Network) Y Luminance component

[0022] This specification describes a flexible GDR that allows the same coding rule to be followed as in the gradual decoding refresh (GDR) design in versatile video coding (VVC) and enhanced compression model (ECM) (VVC / ECM), namely, that the updated area of ​​the current GDR image / image being recovered cannot use any coding information from the unupdated areas of the current image and the reference image (otherwise leakage may occur). However, to prevent in-loop filtering of pixels within the updated area from using coding information from the unupdated area around the virtual boundary, the features described herein may be used as follows: 1. For a virtual boundary segment, if both sides of the segment are within an updated or unupdated area, normal (symmetric) in-loop filtering may be performed on the pixels surrounding this segment of the virtual boundary, and 2. For a virtual boundary segment, if one side is within an updated area and the other side is within an unupdated area, asymmetric in-loop filtering may be performed on the pixels surrounding this segment of the virtual boundary.

[0023] Examples of asymmetric in-loop filtering are described, for example, in U.S. Patent Application No. 17 / 731,428, filed on 28 April 2022, and U.S. Provisional Patent Application No. 63 / 388,385, filed on 12 July 2022, which are incorporated in their entirety by reference.

[0024] The following describes in detail suitable devices and possible mechanisms for implementing an in-loop filter at a virtual boundary. In this regard, references are first made to Figures 1 and 2, where Figure 1 shows an exemplary block diagram of device 50. This device may be an Internet of Things (IoT) device configured to perform various functions, such as collecting information by one or more sensors, receiving or transmitting information, and analyzing information collected or received by the device. The device may include a neural network weight update coding system that can incorporate a codec. Figure 2 shows a layout of the device according to an exemplary embodiment. The elements of Figures 1 and 2 are then described.

[0025] The electronic device 50 may be, for example, a portable terminal or user device of a wireless communication system, a sensor device, a tag, or other low-power device. Alternatively, the electronic device may be a non-mobile computer or part of a computer. However, it will be understood that embodiments of the examples described herein may be implemented in any electronic device or apparatus capable of processing data.

[0026] The device 50 may include a housing 30 for housing and protecting the device. The device 50 may further include a display 32 in the form of a liquid crystal display. In other embodiments of the embodiments described herein, the display may be any suitable display technology suitable for displaying images or videos. The device 50 may further include a keypad 34 (or touch area 34). In other embodiments of the embodiments described herein, any suitable data or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or data input system as part of a touch-sensitive display.

[0027] The device may be provided with a microphone 36 or any suitable audio input which may be a digital or analog signal input. The device 50 may further be provided with an audio output device which, in embodiments of the embodiments described herein, may be an earphone 38, a speaker, or any one of an analog audio output connection or a digital audio output connection. The device 50 may also be provided with a battery (or, in other embodiments of the embodiments described herein, the device may be powered by any suitable mobile energy device such as a solar cell, a fuel cell, or a spring-driven generator). The device may further be provided with a camera 42 which may record or capture images and / or video. The device 50 may further be provided with an infrared port for short-range line-of-sight communication to other devices. In other embodiments, the device 50 may further be provided with any suitable short-range communication solution such as a Bluetooth wireless connection or a USB / FireWire wired connection.

[0028] The device 50 may include a controller 56, a processor, or a processor circuit for controlling the device 50. The controller 56 may be connected to a memory 58, which in the embodiments of the examples described herein may store both image data and audio data, as well as instructions to be performed by the controller 56. The controller 56 may be further connected to a codec circuit 54, which is suitable for performing coding / compression of neural network weight updates and / or decoding of audio and / or video data, or for assisting coding and / or decoding performed by the controller.

[0029] The device 50 may further include a card reader 48 and a smart card 46, such as a UICC and a UICC reader, which are suitable for providing user information and for providing authentication information for user authentication and authorization on a network.

[0030] The device 50 may include a radio interface circuit 52 connected to a controller and suitable for generating wireless communication signals for communication with, for example, a cellular communication network, a wireless communication system, or a wireless local area network. The device 50 may further include an antenna 44 connected to the radio interface circuit 52 for transmitting radio frequency signals generated by the radio interface circuit 52 to other devices such as network nodes, and / or for receiving radio frequency signals from other devices.

[0031] The device 50 may include a camera capable of recording or detecting individual frames, which are then passed to a codec 54 or controller for processing. The device may receive video image data or machine learning data for processing from another device before transmission and / or storage. The device 50 may also receive images for encoding / decoding wirelessly or via a wired connection. The structural elements of the device 50 described above represent examples of means for performing the corresponding functions.

[0032] With respect to Figure 3, an example of a system in which embodiments of the embodiments described herein may be used is shown. System 10 comprises multiple communication devices that can communicate over one or more networks. System 10 may comprise any combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, CDMA, LTE, 4G, and 5G networks), wireless local area networks (WLANs) as defined by any of the IEEE 802.x standards, Bluetooth personal area networks, Ethernet local area networks, Token Ring local area networks, wide area networks, and the Internet.

[0033] System 10 may include both wired and wireless communication devices and / or apparatus 50 suitable for carrying out embodiments of the examples described herein.

[0034] For example, the system shown in Figure 3 represents a representation of a cellular network 11 and the Internet 28, which can access the various devices shown in Figure 3 using a communication link 2 (wired or wireless). The connection to the Internet 28 may include, but is not limited to, long-range wireless connections, short-range wireless connections, and various wired connections, including, but not limited to, telephone lines, cable lines, power lines, and similar communication paths.

[0035] The exemplary communication devices shown in System 10 may include, but are not limited to, electronic devices or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, and a notebook computer 22. Apparatus 50 may be stationary or mobile when carried by a moving person. Apparatus 50 may also be located in a means of transport, including, but not limited to, a car, truck, taxi, bus, train, boat, airplane, bicycle, motorcycle, or any similar suitable means of transport, or in a head-mounted display (HMD) 17.

[0036] Embodiments may be implemented in set-top boxes, i.e., digital television receivers, which may or may not have display or wireless capabilities; in tablets or (laptop) personal computers (PCs) that include hardware and / or software for processing neural network data; in various operating systems; and in chipsets, processors, DSPs, and / or embedded systems that provide hardware / software-based coding.

[0037] Some or more devices may send and receive calls and messages and communicate with service providers via a wireless connection 25 to a base station 24. The base station 24 may be connected to a network server 26 that enables communication between the cellular network 11 and the internet 28. The system may include additional communication devices and various types of communication devices.

[0038] Communication devices may communicate using a variety of transmission technologies, including but not limited to code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11, 3GPP narrowband IoT, and any similar wireless communication technologies. Communication devices involved in carrying out various embodiments of the examples described herein may communicate using a variety of media, including but not limited to radio, infrared, laser, cable connections, and any suitable connections.

[0039] In telecommunications and data networks, a channel may refer to either a physical channel or a logical channel. A physical channel may refer to a physical transmission medium such as a wire, while a logical channel may refer to a logical connection on a multiplexed medium capable of transmitting multiple logical channels. A channel may be used to transmit information signals, such as a bitstream, from one or more senders (or transmitters) to one or more receivers.

[0040] The embodiments may also be implemented in so-called IoT devices. The Internet of Things (IoT) may be defined, for example, as the interconnection of embedded computing devices that are uniquely identifiable within existing internet infrastructure. A collection of various technologies encompasses many areas of embedded systems included in the Internet of Things (IoT), such as wireless sensor networks, control systems, and home / building automation, and can enable them. To utilize the internet, IoT devices are provided with an IP address as a unique identifier. IoT devices may be equipped with wireless transmitters such as WLAN or Bluetooth transmitters or RFID tags. Alternatively, IoT devices can access IP-based networks via Ethernet-based networks or wired networks such as power-line connections (PLC).

[0041] One application where asymmetric in-loop filtering at virtual boundaries and model-level update skipping in compressed incremental learning are important is the use case of neural network-based codecs, such as neural network-based video codecs. A video codec may use one or more neural networks. In the first example, the video codec may be a conventional video codec, such as a versatile video codec (VVC / H.266) that has been modified to include one or more neural networks. Examples of these neural networks are as follows: 1. Neural network filters used as one of the in-loop filters in VVC 2. A neural network filter that replaces one or more of the VVC's in-loop filters. 3. Neural network filters used as post-processing filters 4. Neural network used to perform intraframe prediction 5. Neural network used to perform interframe prediction

[0042] In the second example, commonly referred to as an end-to-end learned video codec, the video codec may include a neural network that transforms the input data into a more compressible representation. The new representation may be quantized, losslessly compressed, then losslessly restored, dequantized, and then another neural network may transform its input into reconstructed or decoded data.

[0043] In both of the above examples, one or more neural networks may be present on the decoder side, and we consider an example of a single neural network filter. The encoder may fine-tune the neural network filter by using ground truth data (uncompressed data) available on the encoder side. Fine-tuning may be performed to improve the neural network filter when applied to the current input data, such as one or more video frames. Fine-tuning may involve performing one or more optimization iterations on some or all of the learnable weights of the neural network filter. Optimization iterations may involve, for example, calculating the gradient of the loss function with respect to some or all of the learnable weights of the neural network filter by using a backpropagation algorithm, and then updating some or all of the learnable weights by using an optimizer, such as a stochastic gradient descent optimizer. The loss function may include one or more loss terms. One exemplary loss term may be the mean squared error (MSE). Other distortion metrics may be used as loss terms. The loss function may be computed by providing one or more data points as inputs to a neural network filter, obtaining one or more corresponding outputs from the neural network filter, and computing the loss term using one or more outputs from the neural network filter and one or more ground truth data points. The difference between the weights of the finely tuned neural network and the weights of the neural network before fine-tuning is called the weight update. This weight update is encoded and provided to the decoder side along with the encoded video data, and must be used by the decoder side to update the neural network filter. The updated neural network filter is then used as part of the video decoding process or as part of the video post-processing process. It is desirable to encode the weight update so that it requires a small number of bits.Therefore, the examples described herein also consider this use case of neural network-based codecs as a possible application of weight update compression.

[0044] Further explanation of the use cases of neural network-based codecs is provided below. The MPEG-2 transport stream (TS), as defined in ISO / IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a multi-stream format for carrying audio, video, and other media, as well as program metadata or other metadata. Packet identifiers (PIDs) are used to identify the underlying stream (also known as the packetized underlying stream) within the TS. Therefore, logical channels within an MPEG-2 TS may be considered to correspond to specific PID values.

[0045] The available media file format standards include ISO-based media file formats (ISO / IEC 14496-12, sometimes abbreviated as ISOBMFF) and NAL unit structured video file formats derived from ISOBMFF (ISO / IEC 14496-15).

[0046] A video codec consists of an encoder, which converts the input video into a compressed representation suitable for storage / transmission, and a decoder, which can restore the compressed video representation into a viewable form. The video encoder and / or video decoder may also be separate from each other, i.e., they may not need to form a codec. Typically, the encoder discards some information from the original video sequence in order to represent the video in a more compact form (i.e., at a lower bitrate).

[0047] Typical hybrid video encoders, such as many implementations of ITU-T H.263 and H.264, encode video information in two stages. First, pixel values ​​within a particular image area (or "block") are predicted, for example, by motion compensation (finding and indicating an area in one of the previously encoded video frames that precisely corresponds to the block being encoded) or by spatial means (using pixel values ​​around the block being encoded in a specified way). Next, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is encoded. This is usually done by transforming the difference in pixel values ​​using a specified transform (e.g., the Discrete Cosine Transform (DCT) or a variation thereof), quantizing the coefficients, and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (image quality) and the size of the resulting encoded video representation (file size or transmission bitrate).

[0048] In time prediction, the source of the prediction is a previously decoded image (also known as a reference image). In intra-block copy (IBC, also known as intra-block copy prediction and current image reference), the prediction is applied similarly to time prediction, except that the reference image is the current image, and only previously decoded samples may be referenced in the prediction process. Inter-layer or inter-view prediction may be applied similarly to time prediction, but the reference image is an image decoded from another scalable layer or from another view, respectively. In some cases, inter-prediction may refer only to time prediction, but in other cases, inter-prediction may collectively refer to time prediction, as well as intra-block copy, inter-layer prediction, and inter-view prediction, provided that they are performed in the same or similar process as time prediction. Inter-prediction or time prediction may also be called motion-compensated or motion-compensated prediction.

[0049] Interpretation, sometimes called time prediction, motion compensation, or motion-compensated prediction, reduces temporal redundancy. In interpretation, the source of prediction is a previously decoded image. Intrapretation takes advantage of the fact that adjacent pixels within the same image are likely to be correlated. Intrapretation can be performed in the spatial domain or the transformation domain, meaning that either sample values ​​or transformation coefficients can be predicted. Intrapretation is typically used in intracoding where interpretation is not applicable.

[0050] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transformation coefficients. Many parameters can be entropically coded more efficiently if they are initially predicted from spatially or temporally adjacent parameters. For example, motion vectors may be predicted from spatially adjacent motion vectors, and only the relative difference with the motion vector predictor may be coded. The prediction and intra-prediction of coding parameters are sometimes collectively referred to as in-picture prediction.

[0051] Figure 4 shows a block diagram of a typical video encoder structure. While Figure 4 presents an encoder for two layers, it will be understood that the presented encoder can similarly be extended to encode three or more layers. Figure 4 shows a video encoder comprising a first encoder section 500 for the base layer and a second encoder section 502 for the extension layer. Each of the first encoder section 500 and the second encoder section 502 may comprise similar elements for encoding an incoming image. The encoder sections 500, 502 may comprise pixel predictors 302, 402, prediction error encoders 303, 403, and prediction error decoders 304, 404. Figure 4 also shows embodiments of the pixel predictors 302, 402 and interpreter 306, 406(P inter ), intra predictor 308, 408 (P intraThe first encoder section 500 is shown to include mode selectors 310, 410, filters 316, 416(F), and reference frame memory 318, 418 (RFM). The pixel predictor 302 of the first encoder section 500 encodes the base layer image (I) of the video stream, which is encoded by both an inter predictor 306 (which determines the difference between the image and the motion-compensated reference frame 318) and an intra predictor 308 (which determines the prediction of the image block based only on the already processed portion of the current frame or image). 0,n )300 is received. The outputs of both the inter-predictor and the intra-predictor are passed to the mode selector 310. The intra-predictor 308 may have two or more intra-prediction modes. Thus, each mode may perform an intra-prediction and provide the predicted signal to the mode selector 310. The mode selector 310 also receives a copy of the base layer image 300. Accordingly, the pixel predictor 402 of the second encoder section 502 encodes the extended layer image (I) of the video stream, which is encoded by both the inter-predictor 406 (which determines the difference between the image and the motion-compensated reference frame 418) and the intra-predictor 408 (which determines the prediction of an image block based only on the already processed portion of the current frame or image). 1,n The system receives 400. The outputs of both the inter-predictor and the intra-predictor are passed to the mode selector 410. The intra-predictor 408 may have two or more intra-prediction modes. Thus, each mode may perform an intra-prediction and provide the predicted signal to the mode selector 410. The mode selector 410 also receives a copy of the extended layer image 400.

[0052] Depending on the encoding mode selected to encode the current block, the output of the inter predictors 306, 406, or the output of one of the optional intra predictor modes, or the output of the surface encoder within the mode selector, is passed to the output of the mode selectors 310, 410. The output of the mode selector is passed to the first addition devices 321, 421. The first addition devices subtract the output of the pixel predictors 302, 402 from the base layer image 300 / enhancement layer image 400 to generate the first prediction error signals 320, 420 (D n ), which may be input to the prediction error encoders 303, 403.

[0053] The pixel predictors 302, 402 further receive, from the pre - reconstructors 339, 439, a combination of the predicted representation (P’ n ) of the image blocks 312, 412 and the output 338, 438 (D’ n ) of the prediction error decoders 304, 404. The pre - reconstructed images 314, 414 (I’ n ) may be passed to the intra predictors 308, 408 and the filters 316, 416. The filters 316, 416 that receive the preliminary representation filter the preliminary representation to output the final reconstructed images 340, 440 (R’ n ) that may be stored in the reference frame memories 318, 418. The reference frame memory 318 may be connected to the inter predictor �06 that is used as the reference image against which the future base layer image 300 is compared in the inter prediction operation. Subject to the base layer being selected and shown as the source for the inter - layer sample prediction and / or inter - layer motion information prediction of the enhancement layer in some embodiments, the reference frame memory 318 may also be connected to the inter predictor 406 that is used as the reference image against which the future enhancement layer image 400 is compared in the inter prediction operation. Further, the reference frame memory 418 may be connected to the inter predictor 406 that is used as the reference image against which the future enhancement layer image 400 is compared in the inter prediction operation.

[0054] Provided that the base layer is selected and shown as a source for predicting the filtering parameters of the extended layer according to some embodiments, the filtering parameters from the filter 316 of the first encoder section 500 may be provided to the second encoder section 502.

[0055] The prediction error encoders 303 and 403 comprise conversion units 342 and 442(T) and quantizers 344 and 444(Q). The conversion units 342 and 442 convert the first prediction error signals 320 and 420 into a conversion domain. The conversion is, for example, a DCT conversion. The quantizers 344 and 444 quantize the conversion domain signals, for example, the DCT coefficients, to form quantization coefficients.

[0056] The prediction error decoders 304 and 404 receive the output from the prediction error encoders 303 and 403 and perform the reverse process of the prediction error encoders 303 and 403 to generate the decoded prediction error signals 338 and 438. The decoded prediction error signals 338 and 438 are then combined with the prediction representations of the image blocks 312 and 412 by the second adder devices 339 and 439 to generate the preliminary reconstructed images 314 and 414. The prediction error decoders 304 and 404 also use inverse quantizers 346 and 446 (Q) to reconstruct the transformed signal by inverse quantization of the quantization coefficient values, e.g., the DCT coefficients. -1 ), and inverse conversion units 348, 448(T) that perform the inverse conversion to the reconstructed converted signal. -1 The predictive error decoder may be considered to include a block filter that can further filter the reconstructed block according to the decoded information and filter parameters.

[0057] The entropy encoders 330 and 430(E) receive the outputs of the predictive error encoders 303 and 403, perform appropriate entropy encoding / variable-length encoding on the signal, and provide error detection and correction functions. The outputs of the entropy encoders 330 and 430 may be inserted into the bitstream by, for example, a multiplexer 508(M).

[0058] The concept of virtual boundaries was introduced in VVC. From the perspective of coding dependencies, an image may be divided into different regions by virtual boundaries. For example, virtual boundaries are used to define the boundaries of different faces of a 360° image in CMP format, and in GDR (see U.S. Provisional Patent Application No. 63 / 296,590, “New Gradual Decoding Refresh for ECM,” filed by the applicant of this disclosure, which is incorporated herein by reference in its entirety) where virtual boundaries separate updated and unupdated areas of a GDR image / image being recovered. In VVC, virtual boundaries are specified in the SPS and / or image header.

[0059] VVC has three in-loop filters: deblocking, SAO, and ALF. ECM improves the in-loop filters with new features including bilateral (JVET-F0034, JVET-V0094), BIF for saturation (JVET-X0067), CCSAO (JVET-V0153, JVET-Y0106), CCALF (JVET-X0045), and an Alternative band classifier (JVET-X0070) for ALF.

[0060] Current in-loop filtering of pixels often requires the use of encoding information from adjacent pixels. Therefore, filtering on one side of a virtual boundary may involve the use of encoding information on the other side of the virtual boundary.

[0061] In some applications, in-loop filtering may not be permitted to cross virtual boundaries. For example, in GDR, the GDR image / image during recovery may be divided into updated and unupdated areas by a virtual boundary. Referring to Figure 5, since there is no guarantee that the unupdated area 530 will be correctly decoded by the decoder, the updated area 510 cannot use any information from the unupdated area 530 to prevent leakage. Incorrectly decoded encoded information may contaminate the updated area 510, potentially causing encoder and decoder leakage or mismatch in the image at the recovery point and subsequent images. Therefore, in the case of the GDR image / image during recovery, in-loop filtering cannot cross the virtual boundary 520 from the updated area 510 to the unupdated area 530, as indicated by arrow 540.

[0062] On the other hand, it is not always a problem for in-loop filtering to cross virtual boundaries. For example, as shown in Figure 6, in the same example of GDR, the unupdated area 630 can use the information from the updated area 610. Therefore, in the case of a GDR image / image being recovered, in-loop filtering can cross the virtual boundary 620 from the unupdated area 630 to the updated area 610, as indicated by arrow 640.

[0063] In current VVC and ECM designs, in-loop filtering cannot cross virtual boundaries.

[0064] U.S. Provisional Patent Application No. 63 / 362,243, “In-Loop Filtering at Virtual Boundaries,” filed by the applicant of this disclosure, which is incorporated herein by reference in its entirety, proposes several possible options for in-loop filtering at virtual boundaries. Among these is asymmetric in-loop filtering at virtual boundaries, in which in-loop filtering cannot cross the virtual boundary from one side to the other, but can cross from the other side to the first.

[0065] Specifically, in-loop filtering on one side of a virtual boundary cannot use information from the other side of the virtual boundary, but in-loop filtering on the other side of the virtual boundary can use information from the one side. If in-loop filtering of a pixel on one side of the virtual boundary requires the use of any information from the other side (e.g., pixel, encoding mode, QP, etc.), then in-loop filtering is either not performed on this pixel, or it is still performed on this pixel by filling in the information from the other side.

[0066] In asymmetric in-loop filtering at a virtual boundary, in-loop filtering on one side cannot use information from the other side, but in-loop filtering on the other side is allowed to use information from the one side.

[0067] In-loop filtering of pixels on one side may not work correctly if the in-loop filtering requires the use of encoding information from the other side.

[0068] In general, in-loop filtering of pixels on the other side can be performed successfully because it is permitted to use the encoding information of both the one side and the other side. However, the other side may choose not to use the encoding information of the one side, in which case in-loop filtering of pixels on the other side may not be performed successfully if it requires the use of the encoding information of the one side.

[0069] Since the encoded information from one side is available on the other side, an offset based on in-loop filtering on one side may be added to the output of the in-loop filtering on the other side.

[0070] A virtual boundary is a line used to separate an image or a portion of an image into two areas, namely a first area and a second area.

[0071] Virtual boundaries can be vertical or horizontal. In VVC and ECM, the syntax for virtual boundaries is included in the SPS and / or image header. In one embodiment, which includes asymmetric behavior at virtual boundaries, the first area is not permitted to use any information from the second area, but the second area can use information from the first area.

[0072] In one embodiment, within a GDR image / image being recovered, a first area is a clean (updated) area, and a second area is a dirty (unupdated) area. The clean (updated) area cannot use any information from the dirty (unupdated) area, but the dirty (unupdated) area can use information from the clean (updated) area. Pixel in-loop filtering may involve the use of encoded information from adjacent pixels.

[0073] If in-loop filtering of pixels in the first area requires the use of encoding information from the second area (e.g., pixel, encoding mode, reference image, MV, QP, etc.), then in-loop filtering of pixels may not be performed correctly. Actual in-loop filtering of pixels may choose one of two possible options: Option 1, in-loop filtering is not performed for pixels in the first area; or Option 2, in-loop filtering is still performed for pixels in the first area, but the encoding information from the second area is derived from the first area or set to a default value when necessary.

[0074] One embodiment related to Option 2 is that, if in-loop filtering of pixels in the first area requires the use of pixels in the second area, the pixels in the second area are filled from the pixels in the first area.

[0075] Another embodiment related to Option 2 is that, if in-loop filtering of pixels in the first area requires the use of pixels in the second area, the pixels in the second area are replaced with pixels extrapolated from the first area.

[0076] Normal in-loop filtering of a pixel is the ideal in-loop filtering of a pixel that uses all the necessary information, and actual in-loop filtering of a pixel is the practical in-loop filtering of a pixel, regardless of whether or not it uses all the necessary information.

[0077] The actual in-loop filtering of pixels in either option 1 or 2 may produce an output that differs from the normal in-loop filtering of pixels where the encoding information of both the first and second areas can be used.

[0078] In-loop filtering for pixels in the second area can usually be performed successfully because it is permitted to use the encoding information from both the first and second areas.

[0079] p i,j is a pixel within the first area,

number

number

number

number

[0080] In the virtual boundary, pixel p within the first area i,j The actual in-loop filtering is performed on pixel p i,j If the in-loop filtering of requires the use of encoding information from a second area, it may not be equivalent to normal in-loop filtering of this pixel, i.e.,

[0081]

number

[0082] On the other hand, in the virtual boundary, pixel q in the second area i,jThe actual in-loop filtering is performed on pixel q i,j Since the actual in-loop filtering can use the encoded information of both the first and second areas, it is usually equivalent to the normal in-loop filtering of this pixel, i.e.,

[0083]

number

[0084] To compensate for unbalanced in-loop filtering at virtual boundaries, the difference between the normal in-loop filtering and the actual in-loop filtering in the first area may be compensated by in-loop filtering in the second area. Note that it is feasible to use the first area to offset the second area, since the second area can use the encoded information of the first area.

[0085] Specifically, pixel p within the first area i,j The normal in-loop filtering and the actual in-loop filtering are different, that is,

number

number

number

number

number

[0086] In one embodiment, the second area may choose not to use the encoding information of the first area. In this case, if in-loop filtering of pixels in the second area requires the use of the encoding information of the first area, the in-loop filtering of pixels may not be performed correctly. Similar to the first area, the actual in-loop filtering of pixels may choose one of two possible options: Option 1, in-loop filtering is not performed on pixels in the second area, or Option 2, in-loop filtering is still performed on pixels in the second area, but the encoding information of the first area is derived from the second area or set to a default value when necessary.

[0087] One embodiment related to option 2 above is that, if in-loop filtering of pixels in the second area requires the use of pixels in the first area, the pixels in the first area are filled from the pixels in the second area.

[0088] Another embodiment related to option 2 above is that, when in-loop filtering of pixels in the second area requires the use of pixels in the first area, the pixels in the first area are replaced with pixels extrapolated from the second area.

[0089] Pixel p in the first area i,j The difference between the target in-loop filtering and the actual in-loop filtering is the corresponding pixel q in the second area. m,n It may be used to offset the output of in-loop filtering. Possible examples are as follows:

[0090]

number

number

number

number

number

[0091] In one embodiment, if the first and second areas choose the same option for in-loop filtering of pixels around the virtual boundary, i.e., either perform no in-loop filtering or perform in-loop filtering using embedding, then the in-loop filtering of the first and second areas may be considered balanced. Compensation may not be required on either side of the virtual boundary.

[0092] One embodiment relates to deblocking filters in VVCs and ECMs. Deblocking filtering is applied to block boundaries (horizontal or vertical) including pixels on both sides of the block boundary.

[0093] Assume that a virtual boundary separates an image or part of an image into a first area and a second area, where the first area is not permitted to use the encoded information in the second area, but the second area can use the encoded information in the first area.

[0094] When block boundaries are aligned with virtual boundaries, deblocking filtering for pixels in a first area at a maximum of n pixels away from the virtual boundary (e.g., in the current designs of VVC and ECM, 1 for a low-saturation filter, 2 for a low-luminance filter, 3 for a high-luminance and high-saturation filter, and 3, 5, 7 for a bilinear (long) luminance filter) requires the use of encoded information (e.g., pixel, encoding mode, QP, etc.) in a second area.

[0095] Because the first area is not permitted to use the encoding information in the second area, deblocking filtering is disabled for those pixels in the first area that are up to n pixels away from the virtual boundary. Figure 7 shows an example where the updated area (first area) 7010 of the GDR image / recovered image is not permitted to use the encoding information of the unupdated area (second area) 7030. Deblocking (e.g., strong filtering) 7040 is disabled for pixels p in the updated area 7010 adjacent to the virtual boundary 7020. i This is invalid for i=0,1,2.

[0096] Alternatively, the deblocking filter 7040 is still applied to those pixels in a first area up to n pixels away from the virtual boundary 7020, but the encoding information in the second area is derived from the first area or set to a default value when needed. For example, in Figure 7, deblocking (e.g., a strong filter) is applied to pixels p in the updated area 7010 adjacent to the virtual boundary 7020. i , still applies to i=0,1,2 (7040 overall), but q in area 7030 has not been updated. iThe associated pixel 7050 containing i=0,1,2 is derived from the updated area 7010. For example, q i i=0,1,2 are p0 or p i It may be set to be equal to the mean or median of i=0,1,2.

[0097] Deblocking of pixels in the second area can be successfully performed by allowing the use of encoding information from both the first area 7010 and the second area 7030.

[0098] Pixel p in area 7010 i If the actual deblocking filtering differs from normal deblocking filtering, this difference corresponds to the following pixel q i It can be offset from.

number

number

number

number

number

number

[0099] One possible embodiment may be as follows: If i=0,...,sq-1,

number

[0100] A simpler embodiment may be as follows:

number

[0101] Corresponding pixel p i and q i These are the mirrored pixels in the first area 7010 and the second area 7030 before deblocking against the block boundary or virtual boundary 7020, as shown in Figure 7.

[0102] If the second area 7030 chooses not to use the encoding information of the first area 7010, deblocking filtering will not be applied to pixels in the second area 7030 that are up to n pixels away from the virtual boundary (e.g., in the current designs of VVC and ECM, 1 for a low saturation filter, 2 for a low luminance filter, 3 for a strong luminance and saturation filter, and 3, 5, 7 for a bilinear (long) luminance filter). Figure 7 can illustrate an example where the unupdated area (second area) 7030 of the GDR image / recovered image chooses not to use the encoding information of the updated area (first area) 7010. Deblocking (e.g., strong filter) will not be applied to pixels q in the unupdated area 7030 adjacent to the virtual boundary 7020. i This is invalid for i=0,1,2.

[0103] Alternatively, deblocking filtering is still applied to those pixels 7050 in a second area 7030 that is up to n pixels away from the virtual boundary 7020, but the encoding information in the first area 7010 is derived from the second area 7030 or set to a default value. For example, in Figure 7, deblocking (e.g., strong filtering) is applied to pixels q in an unupdated area 7030 adjacent to the virtual boundary 7020. i , can still be applied to i=0,1,2 (7050 overall), but p within the updated area 7010 i The associated pixel 7040 containing i=0,1,2 is derived from the unupdated area 7030. For example, p i i=0,1,2 are q0 or q i It may be set to be equal to the mean or median of i=0,1,2.

[0104] One embodiment relates to an SAO edge offset filter. In a VVC, the SAO has two parts: a band offset and an edge offset. Each CTU can choose to use either a band offset or an edge offset. The choice of band offset or edge offset for each CTU is signaled. If an edge offset is used in a CTU, a set of parameters (offsets for the edge class as shown in Figure 8 and the four edge categories as shown in Figure 9) is signaled.

[0105] Referring to Figure 8, four examples of edge classes are shown. In Example 810, pixels a and b are adjacent to pixel c horizontally. In Example 820, pixels a and b are adjacent to pixel c vertically. In Example 830, pixels a and b are adjacent to pixel c along a slope from the top left to the bottom right. In Example 840, pixels a and b are adjacent to pixel c along a slope from the bottom left to the top right.

[0106] Referring to Figure 9, examples of four edge categories are shown. In Category 1, 910, the value of pixel c is smaller than the values ​​of pixels a and b. In Category 2, 920, the values ​​of pixels c and b may be similar, while the value of pixel a may be larger than the values ​​of pixels c and b. Alternatively, the values ​​of pixels a and c may be similar, while the value of pixel b may be larger than the values ​​of pixels a and c. In Category 3, 930, the values ​​of pixels a and c may be similar, while the value of pixel b may be smaller than the values ​​of pixels a and c. Alternatively, the values ​​of pixels c and b may be similar, while the value of pixel a may be smaller than the values ​​of pixels c and b. In Category 4, 940, the value of pixel c may be larger than the values ​​of pixels a and b.

[0107] As seen in Figures 8 and 9, classifying pixel edges involves the use of adjacent pixels.

[0108] Assume that a virtual boundary separates an image or part of an image into a first area and a second area, where the first area is not permitted to use the encoded information in the second area, but the second area can use the encoded information in the first area.

[0109] The SAO edge offset for pixels in the first area immediately adjacent to the virtual boundary may require the use of encoded information (e.g., pixels) in the second area, as shown in Figure 8.

[0110] Since the first area is not permitted to use the encoding information of the second area, the SAO edge offset is not applied to those pixels in the first area that are immediately adjacent to the virtual boundary. Figure 10A shows an example where the updated area (first area) 1010 of the GDR image / recovered image is not permitted to use the encoding information of the unupdated area (second area) 1030. The SAO edge offset with diagonal class direction 1040 is invalidated for pixel p0 in the updated area 1010 that is immediately adjacent to the virtual boundary 1020.

[0111] Alternatively, the SAO edge offset (e.g., 1040) is still applied to pixels in the first area 1010 immediately adjacent to the virtual boundary 1020, but the encoded information (e.g., pixels) in the second area 1030 is either derived from the first area 1010 or set to a default value when needed. For example, in Figure 10A, the SAO edge offset is still applied to pixel p0 in the updated area 1010 immediately adjacent to the virtual boundary 1020, but the corresponding pixel q0 on the unupdated area 1030 is filled in from the updated area 1010 (or to a default value, e.g., 2 BD-1 It is set to (where BD is the bit depth).

[0112] The SAO edge offset for pixels in the second area 1030 adjacent to the virtual boundary 1020 can be successfully performed by allowing the use of encoding information from both the first area 1010 and the second area 1030.

[0113] If the actual SAO edge offset filtering of pixel p0 in the first area 1010 differs from the normal SAO edge offset filtering, this difference may be offset from the corresponding pixel q0 as follows:

number

number

number

number

number

[0114] The corresponding pixels p0 and q0 are mirror pixels with respect to the junction of the virtual boundary along the selected SAO edge class direction line 1040 and the SAO edge offset class direction line, as shown in Figure 10A.

[0115] If the second area chooses not to use the encoding information of the first area, the SAO edge offset is not applied to pixels in the second area adjacent to the virtual boundary. Figure 10B shows an example where an unupdated area (second area) 1070 of the GDR image / recovered image chooses not to use the encoding information of the updated area (first area) 1060. The SAO edge offset is not applied to pixel q0 in the unupdated area 1070 adjacent to the virtual boundary 1080.

[0116] Alternatively, the SAO edge offset is still applied to those pixels in the second area 1070 adjacent to the virtual boundary 1080, but the encoding information in the first area 1060 is derived from the second area 1070 or set to a default value when needed. For example, in Figure 10B, the SAO edge offset is still applied to pixel q0 in the unupdated area 1070 adjacent to the virtual boundary 1080, but the corresponding pixel p0 in the updated area 1060 is filled in from the unupdated area 1070. The edge class direction line 1090 is shown in Figure 10B.

[0117] One embodiment relates to a bilateral filter (BIF) for luminance and saturation. The ECM improves the in-loop filtering of the VVC by adding new filtering capabilities, among which is the bilateral filter. As shown in Figure 11, BIF 1130 runs in parallel with SAO 1120 and CCSAO processes 1140. BIF(1130), SAO(1120), and CCSAO(1140) use the same samples generated by the deblocking filter (1110) as input and generate three offsets in parallel for each sample. These three offsets are then added to the input sample (using operation 1150) to obtain a sum, which is then clipped to form the final output sample value (1160) before proceeding to ALF. The BIF saturation provides an on / off control mechanism at the CTU level and slice level.

[0118] The bilateral filter is a 5x5 rhombus for both luminance and saturation, as shown in Figure 12A, and the bilateral filter is applied to pixels adjacent to the virtual boundary.

[0119] Assume that a virtual boundary separates an image or part of an image into a first area and a second area, where the first area is not permitted to use the encoded information in the second area, but the second area can use the encoded information in the first area.

[0120] BIF filtering for pixels in a first area that are at least n (e.g., 2 in the current design of BIF) pixels away from the virtual boundary requires the use of encoded information (e.g., pixels) in a second area.

[0121] Since the first area is not permitted to use the encoding information of the second area, BIF filtering may be disabled for those pixels in the first area that are at least n (e.g., 2 in the current design of BIF) pixels away from the virtual boundary. Figure 12A shows an example where the updated area (first area) 1210 of the GDR image / recovered image is not permitted to use the encoding information of the unupdated area (second area) 1230. BIF filtering is disabled for pixels p in the updated area 1210 adjacent to the virtual boundary 1220. 0,0 It will not be executed against it.

[0122] Alternatively, BIF filtering is still performed on those pixels 1240 in a first area that is at most n (e.g., 2 in the current design of BIF) pixels away from the virtual boundary, but the encoding information on the second area is derived from the first area or set to a default value when needed. For example, in Figure 12A, BIF filtering is performed on pixels p in the updated area 1210 adjacent to the virtual boundary 1220. 0,0 This still applies, but the unupdated q in area 1230 i,0 The associated pixels, including i=0,1, are filled from the updated area (or the default value, e.g., 2). BD-1 It is set to (where BD is the bit depth).

[0123] BIF filtering for pixel 1250 on the second area 1230 can be successfully performed by allowing the use of encoding information from both the first area 1210 and the second area 1230.

[0124] Pixel p in the first area 1210 i,j If the actual BIF filtering differs from normal deblocking filtering, this difference will result in the corresponding pixel q m,n It can be offset from.

number

number

number

number

number

[0125] Corresponding pixel p i,j and q i,j These are the mirrored pixels in the first area 1210 and the second area 1230 prior to the BIF relative to the virtual boundary 1220, as shown in Figure 12A.

[0126] If the second area chooses not to use the encoding information of the first area, BIF filtering will not apply to pixels in the second area that are up to n (e.g., 2 in the current design of BIF) pixels away from the virtual boundary. Figure 12B shows an example where an unupdated area (second area 1280) of the GDR image / recovered image chooses not to use the encoding information of the updated area (first area) 1260. The BIF filters pixels q in the unupdated area 1280 adjacent to the virtual boundary 1270. 0,0 It will be invalidated against it.

[0127] Alternatively, BIF filtering is still applied to those pixels 1295 in the second area 1280 that are up to n pixels away from the virtual boundary 1270, but the encoded information in the first area 1260 is derived from the second area 1280 or set to a default value when needed. For example, in Figure 12B, BIF is applied to q in the unupdated area 1280 adjacent to the virtual boundary 1270. 0,0 It still applies to pixels containing (1295 in total), but the updated area 1260 p i,0 Related pixels containing i=0,1 (1290 in total) are filled in from the unupdated area of ​​1280.

[0128] One embodiment relates to a CCSAO filter. A cross-component sample adaptive offset (CCSAO) is used to improve a reconstructed sample. Similar to SAO, CCSAO categorizes the reconstructed sample into different categories, deriving one offset for each category, and adding this offset to the reconstructed sample within that category. However, unlike SAO (1340, 1350, 1360), which uses a single luminance / chrominance component of the current sample (one of 1310, 1320, or 1330) as input, as shown in Figure 13, CCSAO (1370, 1380, 1390) utilizes all three components (1310, 1320, or 1330) to categorize the current sample into different categories. To facilitate parallel processing, the output sample from a deblocking filter is used as input to the CCSAO.

[0129] The output of CCSAO Y 1370 is combined (e.g., added or subtracted) with the output of SAO Y 1340 using operation 1391 to produce Y 1394. The output of CCSAO U 1380 is combined (e.g., added or subtracted) with the output of SAO U 1350 using operation 1392 to produce U 1395. The output of CCSAO V 1390 is combined (e.g., added or subtracted) with the output of SAO V 1350 using operation 1393 to produce V 1396.

[0130] In CCSAO, either a band offset (BO) classifier or an edge offset (EO) classifier is used to improve the quality of the reconstructed sample. CCSAO may be applied to both the luminance and chroma components.

[0131] In CCSAO BO, for a given luminance / chrominance sample, three candidate samples are selected—one juxtaposed Y sample, one juxtaposed U sample, and one juxtaposed V sample—to classify the given sample into different categories. The sample values ​​of these three selected samples are then classified into three different bands, and the combined index represents the category of the given sample. An offset is signal-transmitted and added to the reconstructed sample classified into that category.

[0132] As shown in Figure 14, the juxtaposed luminance sample 1410 can be selected from nine candidate positions (1405), while the juxtaposed saturation sample positions (1420, 1430) are fixed.

[0133] Assume that a virtual boundary separates an image or part of an image into a first area and a second area, where the first area is not permitted to use the encoded information in the second area, but the second area can use the encoded information in the first area.

[0134] CCSAO for pixels in the first area immediately adjacent to the virtual boundary may require the use of encoded information (e.g., pixels) in the second area.

[0135] Since the first area is not permitted to use the encoding information of the second area, CCSAO is not applied to those pixels in the first area that are immediately adjacent to the virtual boundary. Figure 15A shows an example where the updated area (first area) 1510 of the GDR image / recovered image is not permitted to use the encoding information of the unupdated area (second area) 1530. CCSAO is skipped for pixels p0 in the updated area 1510 that are immediately adjacent to the virtual boundary 1520. Figure 15A shows juxtaposed saturation 1540.

[0136] Alternatively, CCSAO is still applied to those pixels in the first area immediately adjacent to the virtual boundary, but the encoding information in the second area is either derived from the first area or set to a default value when needed. For example, in Figure 15A, CCSAO is still applied to pixel p0 in the updated area 1510 adjacent to the virtual boundary 1520, but the corresponding pixel q0 in the unupdated area 1530 is filled in from the updated area 1510 (or to a default value, e.g., 2 BD-1 It is set to (where BD is the bit depth).

[0137] CCSAO for pixels on the second area 1530 can be successfully performed by allowing the use of the encoding information of the first area 1510.

[0138] If the actual CCSAO filtering of pixel p0 in the first area 1510 differs from the normal SAO edge offset filtering, this difference may be offset from the corresponding pixel q0 as follows:

number

Number

Number

Number

Number

[0139] The corresponding pixels p0 and q0 are mirror pixels within the first area 1510 and the second area 1530 in front of the CCSAO BO with respect to the virtual boundary, as shown in FIG. 15A.

[0140] If it is selected that the second area does not use the encoded information of the first area, the CCSAO BO is not applied to the pixels within the second area adjacent to the virtual boundary. FIG. 15B shows an example where the non-updated area (the second area) 1580 of the GDR image / the image during restoration does not use the encoded information of the updated area (the first area) 1560. The CCSAO BO is not applied to the pixel q0 within the non-updated area 1580 adjacent to the virtual boundary 1570.

[0141] Alternatively, the CCSAO BO is still applied to those pixels in the second area 1580 adjacent to the virtual boundary, but the encoding information in the first area 1560 is derived from the second area 1580 or set to a default value when needed. For example, in Figure 15B, the CCSAO BO is still applied to pixel q0 in the unupdated area 1580 adjacent to the virtual boundary 1570, but the corresponding pixel p0 in the updated area 1560 is filled in from the unupdated area 1580. Figure 15B shows juxtaposed saturation 1590.

[0142] One embodiment relates to an ALF filter. In VVC, the ALF filter is a rhombus with a size of 7x7 for luminance and a size of 5x5 for saturation. ECM expands the sizes of the ALF for luminance and saturation to 9x9, 7x7, and 5x5. Figure 16A shows an example of a 9x9 rhombus ALF filter for pixels adjacent to the virtual boundary 1620. In addition, ECM adds an alternative band classifier for classification in ALF(ABC-ALF), which is a 13x13 rhombus filter for classifying each 2x2 luminance block of ALF.

[0143] Assume that a virtual boundary separates an image or part of an image into a first area and a second area, where the first area is not permitted to use the encoded information in the second area, but the second area can use the encoded information in the first area.

[0144] ALF filtering for pixels in a first area located up to n pixels away from the virtual boundary (e.g., 3 for luminance and 2 for saturation ALF in the current VVC design, 2, 3, and 4 for luminance and saturation ALF in the current ECM design, and 6 for ABC-ALF) requires the use of encoded information (e.g., pixels) in a second area.

[0145] Since the first area is not permitted to use the encoding information of the second area, ALF filtering may be disabled for those pixels in the first area that are up to n positions away from the virtual boundary. Figure 16A shows an example where the updated area (first area) 1610 of the GDR image / recovered image is not permitted to use the encoding information of the unupdated area (second area) 1630. ALF filters pixel p in the updated area 1610 that is immediately adjacent to the virtual boundary 1620. 0,0 It will not be executed against it.

[0146] Alternatively, ALF is still applied to pixels (1640) in the first area 1610 that are up to n positions away from the virtual boundary 1620, but the encoding information on the second area 1630 is derived from the first area 1610 or set to a default value when needed. For example, in Figure 16A, ALF is applied to pixels p in the updated area 1610 adjacent to the virtual boundary 1620. 0,0 It still runs against but is not updated in area 1630. i,0 The associated pixels 1650, including i=0,1,2, are filled from the updated area 1610 (or the default value, e.g., 2 BD-1 It is set to (where BD is the bit depth).

[0147] ALF filtering on pixels in the second area can be successfully performed by allowing the use of encoding information from both the first and second areas.

[0148] Pixel p in area 1610 i,j If the actual ALF filtering differs from normal deblocking filtering, this difference will result in the corresponding pixel q m,n It can be offset from.

number

number

[0149] The corresponding pixel p i,j and q i,j are the mirrored pixels within the first area 1610 and the second area 1630 in front of the ALF with respect to the virtual boundary 1620 as shown in FIG. 16A.

[0150] If the second area selects not to use the encoded information of the first area, the ALF is not applied to the pixels within the second area that are up to n (e.g., in the current design of VVC, 3 for luminance, 2 for chroma ALF, 2, 3, 4 for luminance and chroma ALF in the current design of ECM, and 6 for ABC-ALF) pixel positions away from the virtual boundary. FIG. 16B shows an example where the unupdated area (the second area) 1680 of the GDR image / recovering image selects not to use the encoded information of the updated area (the first area) 1660. The ALF is not applied to the pixel q 0,0 in the unupdated area 1680 adjacent to the virtual boundary 1670.

[0151] Alternatively, ALF is still applied to those pixels 1695 in the second area 1680 adjacent to the virtual boundary 1670, but the encoding information in the first area 1660 is derived from the second area 1680 or set to a default value when needed. For example, in Figure 16B, ALF is applied to pixels q in the unupdated area 1680 adjacent to the virtual boundary 1670. 0,0 This still applies, but the updated area 1660 p i,0 The associated 1690 pixels, including i=0,1,2, are filled in from the unupdated area 1680.

[0152] One embodiment relates to a CCALF filter. As shown in Figure 17, the CCALF process 1720 filters luminance sample values ​​using a linear filter to produce a residual correction (1770) for chrominance samples. Initially, an 8-tap filter was designed for the CCALF process in VVCs. More recently, a larger 25-tap filter, shown in Figure 18, has been used in the CCALF process (1800) of ECMs. For a given slice, the encoder can collect and analyze slice statistics and can transmit signals through up to 16 filters via APS.

[0153] Referring to Figure 17, a basic example of CCALF is shown. In CTU(Y)1710, CCALF(Cb) may be applied to a set of pixels as shown in 1730 (1720). This may be considered linear filtering of luminance sample values. In CTU(Cb)1740, ALF saturation may be applied to a portion of pixels (1750). This may be considered filtering of saturation samples. The outputs of 1720 and 1750 may be added together (1760) (or alternatively, combined in some other way, e.g., by subtraction by operation 1760) and output as CTB'(Cb)1770.

[0154] Assume that a virtual boundary separates an image or part of an image into a first area and a second area, where the first area is not permitted to use the encoded information in the second area, but the second area can use the encoded information in the first area.

[0155] CCALF filtering for pixels in a first area that are up to n pixels away from the virtual boundary (e.g., 1 for VVC or 4 for ECM) requires the use of encoded information (e.g., pixels) in a second area.

[0156] Since the first area is not permitted to use the encoding information of the second area, CCALF filtering may be disabled for those pixels in the first area that are up to n pixels away from the virtual boundary. Figure 19A shows an example where the updated area 1910 (first area) of the GDR image / recovered image is not permitted to use the encoding information of the unupdated area 1930 (second area). CCALF is skipped for saturation pixels 1950 in the updated area 1910 that are immediately adjacent to the virtual boundary 1920.

[0157] Alternatively, CCALF is still applied to those pixels in a first area up to n pixels away from the virtual boundary, but the encoding information in the second area is derived from the first area or set to a default value when needed. For example, in Figure 19A, CCALF is still applied to the chrominance pixels 1950 in the updated area 1910 adjacent to the virtual boundary 1920, but q on the unupdated area 1930. i,j The associated luminance pixels, including i=0,1,2,3 and j=0,1, are filled from the updated area 1910 (or default values, e.g., 2 BD-1 It is set to (where BD is the bit depth).

[0158] CCALF for pixels in the second area can be successfully executed if it is permitted to use the information from the first area.

[0159] Pixel p within the first area 1910 i,j If the actual CCALF filtering of pixel p is different from the normal deblocking filtering, this difference can be offset from the corresponding pixel q i,j as follows.

Number

Number

Number

Number

Number

[0160] The corresponding pixel p i,j and q i,j are the mirrored pixels within the first area 1910 and the second area 1930 in front of the CCALF with respect to the virtual boundary 1920 as shown in FIG. 19A.

[0161] If the second area chooses not to use the encoding information of the first area, CCALF is not applied to pixels in the second area that are up to n pixels (e.g., 1 for VVC or 4 for ECM) away from the virtual boundary. Figure 19B shows an example where an unupdated area (second area) 1980 of the GDR image / recovered image chooses not to use the encoding information of the updated area (first area) 1960. CCALF is skipped for juxtaposed saturation pixels 1990 in the unupdated area 1980 adjacent to the virtual boundary 1970.

[0162] Alternatively, CCALF is still applied to those pixels in the second area 1980 adjacent to the virtual boundary 1970, but the encoding information in the first area 1960 is derived from the second area 1980 or set to a default value when needed. For example, in Figure 19B, CCALF is still applied to the juxtaposed chrominance pixels 1990 in the unupdated area 1980 adjacent to the virtual boundary 1970, but the p in the updated area 1960 i,j The associated luminance pixels, including i=0,1,2,3 and j=0,1, are filled in from area 1980, which has not been updated.

[0163] Figure 20 is a block diagram 700 of a device 710 suitable for carrying out an exemplary embodiment. One non-limiting example of the device 710 is a wireless, typically mobile device that can access a wireless network. The device 710 includes one or more processors 720, one or more memories 725, one or more transceivers 730, and one or more network interfaces 761 interconnected via one or more buses 727. Each of the one or more transceivers 730 includes a receiver Rx732 and a transmitter Tx733. The one or more buses 727 may be an address bus, a data bus, or a control bus, and may include any interconnection mechanism such as a series of wires on a motherboard or integrated circuit, optical fiber, or other optical communication equipment.

[0164] The device 710 may communicate via wired, wireless, or both interfaces. In the case of wireless communication, one or more transceivers 730 are connected to one or more antennas 728. One or more memories 725 contain computer program code 723. The N / WI / F 761 communicates via one or more wired links 762.

[0165] The apparatus 710 includes a control module 740 comprising one or both of parts 740-1 and / or 740-2, wherein parts 740-1 and / or 740-2 include a reference 790 containing an encoder 780 or a decoder 782 or both 780 / 782 codecs, which may be implemented in multiple ways. For ease of reference, reference 790 is referred to herein as a codec. The control module 740 may be implemented in hardware as control module 740-1, such as being implemented as part of one or more processors 720. The control module 740-1 may also be implemented as an integrated circuit or by other hardware such as a programmable gate array. In another example, the control module 740 may be implemented as control module 740-2, which is implemented as computer program code 723 and executed by one or more processors 720. For example, one or more memories 725 and computer program code 723 may be configured together with one or more processors 720 to cause the user device 710 to perform one or more of the operations described herein. The codec 790 may similarly be implemented as codec 790-1 as part of control module 740-1, or as codec 790-2 as part of control module 740-2, or both.

[0166] The computer-readable memory 725 may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, firmware, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The computer-readable memory 725 may be a means for performing storage functions. One or more computer-readable memories 725 may be non-transient, transient, volatile (e.g., random access memory (RAM)), or non-volatile (e.g., read-only memory (ROM)). One or more computer-readable memories 725 may include a database for storing data.

[0167] The processor 720 may be of any type suitable for the local technical environment and may include, in non-limiting examples, one or more of a general-purpose computer, a dedicated computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multicore processor architecture. The processor 720 may be a means for performing functions such as controlling the device 710 and other functions as described herein.

[0168] Generally, various embodiments of the device 710 include cellular phones (such as smartphones, mobile phones, cellular phones, Voice over Internet Protocol (IP) (VoIP) phones, and / or wireless local loop phones), tablets, portable computers, indoor audio equipment, immersive audio equipment, vehicle or in-vehicle devices for wireless V2X (vehicle-to-everything) communication, image capture devices such as digital cameras, gaming devices, music storage and playback devices, internet appliances (including Internet of Things (IoT) devices), IoT devices with sensors and / or actuators for automation applications, and portable units or terminals incorporating combinations of such functions, laptops, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), Universal Serial Bus (USB) dongles, smart devices, and wireless customer-premises equipment (CPE). This includes, but is not limited to, devices such as equipment, Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain settings), consumer electronics devices, and devices operating on commercial and / or industrial wireless networks. In other words, the device 710 can be any device that may have wireless or wired communication capabilities.

[0169] Accordingly, the apparatus 710 comprises a processor 720 and at least one memory 725 containing computer program code 723, the at least one memory 725 and the computer program code 723 together with the at least one processor 720, which are configured to cause the apparatus 710 to implement an asymmetric in-loop filter 790 at a virtual boundary based on the embodiments described herein. The apparatus 710 may also include a display or I / O 770 that can be used to display content during ML / task / machine / NN processing or rendering. The display or I / O 770 may be configured to receive input from a user using a keypad, touchscreen, touch area, microphone, biometric authentication, one or more sensors, etc. The apparatus 710 may also include standard known components such as amplifiers, filters, frequency converters, and modulators / demodulators.

[0170] Computer program code 723 may include object-oriented software and may perform the filtering described throughout this disclosure. Apparatus 710 does not have to have each of the features mentioned and may have other features. Apparatus 710 may be an embodiment of the apparatus shown in Figures 1, 2, 3, or 4, or any combination thereof.

[0171] Figure 21 shows an exemplary method 2100 for performing an asymmetric in-loop filter on a virtual boundary, based on the embodiments described herein. In 2110, the method includes determining a virtual boundary that separates an image or a portion of an image into a first area and a second area. In 2120, the method includes determining that the encoding information for the second area is derived from the first area, or that the encoding information for the second area is set to at least one value, in order to perform filtering of at least one pixel in the first area, or determining that the encoding information for the second area is not used to perform filtering of at least one pixel in the first area, in order to perform filtering of at least one pixel in the first area. Method 2100 may be performed by an encoder, decoder, or codec, or by any of the devices shown in Figures 1, 2, 3, 4, or 20.

[0172] The above description illustrates an example related to a virtual boundary. The description is found, for example, in U.S. Provisional Patent Application No. 63 / 388,385, filed on 12 July 2022 by the same inventors as this specification, which is incorporated herein by reference in its entirety. The features described herein may be used in conjunction with other forms of symmetric and asymmetric inline filtering.

[0173] As mentioned above, a virtual boundary is a vertical or horizontal line used to separate an image or a portion of an image into two areas: a first area and a second area. In the Versatile Video Coding (VVC) and Enhanced Compression Model (ECM), the syntax for virtual boundaries is included in the sequence parameter set (SPS) and / or the image header. As mentioned above, the first area may refer to the updated area, and the second area may refer to the unupdated area. With Flexible Gradual Decoding Update (GDR), a GDR image / image being recovered may be divided into updated and unupdated areas by multiple virtual boundaries. Some segments of a virtual boundary may have both sides of these segments within the updated or unupdated area, while other segments may have one side within the updated area and the other side within the unupdated area.

[0174] VVC / ECM primarily employs three types of in-loop filtering: deblocking, sample adaptive offset (SAO), and adaptive loop filtering (ALF). As mentioned earlier, in-loop filtering of the current pixel often requires the use of encoding information from adjacent pixels. Therefore, filtering on one side of a virtual boundary may require the use of encoding information from the other side of the virtual boundary.

[0175] As previously mentioned, a flexible GDR can be provided that allows the same coding rule to be followed, namely, that the updated area of ​​the current GDR image / image being recovered cannot use any coding information from the unupdated area of ​​the current image and / or reference image (otherwise leakage may occur), as is the case with the progressive decoding update (GDR) design in the multipurpose video coding (VVC) and enhanced compression model (ECM) (VVC / ECM). However, to prevent in-loop filtering of pixels within the updated area from using coding information from the unupdated area around the virtual boundary, the features described herein may be used as follows: 3. With respect to a virtual boundary segment, if both sides of the segment are within an updated or unupdated area, normal (symmetric) in-loop filtering may be performed on the pixels surrounding this segment of the virtual boundary, and 4. For a virtual boundary segment, if one side of the virtual boundary lies within an updated area and the other side lies within an unupdated area, asymmetric in-loop filtering may be performed on the pixels surrounding this segment of the virtual boundary.

[0176] Universal Video Coding (VVC) is a new international video coding standard, and the Extended Compression Model (ECM), built on top of VVC, has the potential to become a new video coding standard and is currently under development with the support of the Joint Video Experts Team (JVET). Both VVC and ECM support the functionality of Gradual Decoding Update (GDR), primarily for low-latency controlled complexity (LLCC) video applications. In the current GDR design in VVC / ECM, the initial updated area starts on the left side of the GDR image and gradually expands across the associated recovering image. However, the most meaningful content information in a video sequence is not necessarily located in the left portion of the image. Features such as those described herein can be used to provide a flexible GDR where the initial updated area can be at any position in the GDR image.

[0177] In both VVC and ECM, encoded video sequences may consist of intra-encoded and inter-encoded images. Intra-encoded images typically use more bits than inter-encoded images. The transmission time of such large intra-encoded images increases the delay from encoder to decoder. Gradual Decoding Update (GDR) mitigates the delay problem associated with intra-encoded images. Instead of encoding an intra-image, GDR gradually updates the image across multiple images.

[0178] In the current GDR design in VVC / ECM design, as shown in Figure 22, the initial updated area starts on the left side of the GDR image and then gradually expands to the right across the associated recovering image. A virtual boundary specified in the image header is used to separate the updated and unupdated areas of the GDR image / recovering image. Both the encoder and decoder assume that the area to the left of the virtual boundary is the updated area and the area to the right is the unupdated area. As seen in Figure 22, the updated area (RA) starts on the left side of the GDR image and gradually expands to the right across the associated recovering image.

[0179] However, the most important content information of a video sequence is not necessarily located in the left portion of the image. In fact, most game sequences may include the protagonist in the center of the image, and many natural video sequences may include major activity and movement occurring in any part of the image. Therefore, it is proposed, along with the features described herein, to have the initial updated area cover a more meaningful area of ​​the sequence, instead of a fixed area (such as the left area in current VVC / ECM GDR designs). A type of flexible GDR is proposed in U.S. Provisional Patent Application No. 63 / 480,085, filed on 16 January 2023, which is incorporated herein in its entirety by reference, in which the initial updated area can be at any position in the GDR image and then gradually expand across the relevant recovering image.

[0180] Figures 23 and 24 show two possible examples of flexible GDR. In Figure 23, the initial updated area (RA) is located in the center of the GDR image and then gradually expands to the left and right sides of the associated recovering image. In Figure 24, the initial updated area (RA) is located in the center of the GDR image and then gradually expands in four directions across the associated recovering image.

[0181] In asymmetric in-loop filtering, filtering pixels within an updated area does not use any encoding information from the unupdated area. However, filtering pixels within an unupdated area can use the encoding information from the updated area. For details, 1. If in-loop filtering of pixels within an updated area requires the use of encoding information from an unupdated area (e.g., pixel, encoding mode, refIdx, MV, QP, etc.), then in-loop filtering of pixels may not be performed correctly. Actual in-loop filtering of pixels may be performed using one of the following two possible options: • Option 1: In-loop filtering of pixels within the updated area is not performed, or Option 2: In-loop filtering of pixels within the updated area is still performed, but if necessary, it uses the encoding information of the unupdated area, either derived from the updated area or set to the default value. 2. In-loop filtering of pixels in unupdated areas can usually be performed successfully. This is because in-loop filtering of pixels in unupdated areas is permitted to use the encoding information of both updated and unupdated areas.

[0182] Figure 25 shows several examples of GDRs in which a GDR image / image being recovered is divided into updated areas (RAs) and unupdated areas (blanks) by multiple virtual boundaries (VBs). Figure 25A shows the updated areas starting from the left. Figure 25B shows the updated areas starting from the vertical center. Figure 25C shows the updated areas starting from the top. Figure 25D shows the updated areas starting from the horizontal center. Figure 25E shows the updated areas starting from the center. Figure 25F shows the updated areas starting from the four corners.

[0183] In Figures 25A, 25B, 25C, and 25D, one side of the virtual boundary is within the updated area, and the other side is within the unupdated area. Therefore, asymmetric in-loop filtering is performed on the pixels around the virtual boundary VB.

[0184] In Figures 25E and 25F, some segments of the virtual boundary may be in areas that have not been updated on both sides, while other segments may be in areas that have been updated on one side and areas that have not been updated on the other side. Normal (symmetric) in-loop filtering may be performed on pixels around segments that are in areas that have not been updated on both sides, while asymmetric in-loop filtering may be performed on pixels around segments that are in areas that have been updated on one side and areas that have not been updated on the other side.

[0185] For example, in Figure 25E, both sides of segments s1 and s3 of the vertical virtual boundary VB1 are within the unupdated area, but the left side of segment s2 is within the unupdated area, while the right side of segment s2 is within the updated area. Therefore, normal (symmetric) in-loop filtering may be applied to segments s1 and 3, and asymmetric filtering may be applied to segment s2.

[0186] Similarly, in Figure 25F, both sides of segment s2 of the vertical virtual boundary VB1 are within the unupdated area, while the left side of segments s1 and s3 is within the updated area, and the right side of segments s1 and s3 is within the unupdated area. Therefore, normal (symmetric) in-loop filtering may be applied to segment s2, and asymmetric filtering may be applied to segments s1 and s3.

[0187] See also Figure 26, according to one exemplary embodiment, an exemplary method is provided which includes determining a plurality of virtual boundaries for at least a portion of an image, as shown by block 2602, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary; determining at least one of a first condition, as shown by block 2604, where both sides of a first segment at the first virtual boundary are within an updated area or both sides of the first segment are within an unupdated area, or where both sides of the first segment at the first virtual boundary are not within an updated area or both sides of the first segment are not within an unupdated area; performing a first type of filtering of at least one pixel adjacent to the first segment based on the determination of the first condition, as shown by block 2606; and performing a second different type of filtering of at least one pixel adjacent to the first segment based on the determination of the second condition, as shown by block 2608.

[0188] The second condition may include the case where one side of the first segment at the first virtual boundary lies within an updated area, and the other side of the first segment at the first virtual boundary lies within an unupdated area. The second different type of filtering may include not performing in-loop filtering on at least one pixel in the updated area when filtering at least one pixel requires the use of encoding information from the unupdated area. The second different type of filtering may include performing in-loop filtering on at least one pixel in the updated area, which includes using encoding information from the unupdated area derived from the updated area, or using encoding information from the unupdated area set to at least one default value, when filtering at least one pixel requires the use of encoding information from the unupdated area. The second different type of filtering may include filtering at least one pixel in the first area, which includes using encoding information from the second area derived from the first area, or using encoding information from the second area set to at least one value, when the encoding information from the second area is used to perform filtering on at least one pixel in the first area. The first type of filtering may include in-loop filtering. The second type of filtering may include in-loop filtering. The image may include a sequentially decoded updated image or an image being recovered. The method may further include embedding pixels from pixels in the first area on the first side of the first virtual boundary into the second area on the second side of the first virtual boundary, in response that pixels in the second area are used to perform filtering of at least one pixel in the first area. The method may further include replacing pixels in the second area on the second side of the first virtual boundary with pixels extrapolated from the first area on the first side of the first virtual boundary, in response that pixels in the second area are used to perform filtering of at least one pixel in the first area.The method may further include determining a first output of filtering at least one pixel in the first area on the first side of the first virtual boundary, if the encoded information of the first area and the encoded information of the second area on the second side of the first virtual boundary are available for filtering at least one pixel in the first area, and determining a second output of filtering at least one pixel in the first area, if the encoded information of the second area is not available for filtering at least one pixel in the first area. The method may further include determining the difference between the first and second outputs, and determining an output for filtering at least one pixel in the second area, using the difference or an approximation of the difference at least partially. The encoded information of the second area may include the output of filtering at least one pixel in the second area. The location of at least one pixel in the second area may correspond to the location of at least one pixel in the first area. The method may further include determining the difference between a first output and a second output, determining an initial output for filtering at least one pixel in the second area, and determining the final output for filtering at least one pixel in the second area by at least partially subtracting this difference from the initial output, wherein the encoded information of the second area includes the final output for filtering at least one pixel in the second area. The method may further include determining the final output for filtering at least one pixel in the second area by at least partially subtracting a weighted contribution of the difference from the initial output. At least one of the first or second filtering may include at least one of deblocking filtering, sample-adaptive offset edge-offset filtering, bilateral filtering for luminance, bilateral filtering for saturation, inter-component sample-adaptive offset filtering, adaptive loop filtering, or inter-component adaptive loop filtering.This method may further include disabling filtering of at least one pixel in a first area on the first side of the first virtual boundary up to a number of pixel positions from the first virtual boundary. This method may further include performing filtering of at least one pixel in a first area on the first side of the first virtual boundary up to a number of pixel positions from the first virtual boundary. This method may further include performing filtering of pixels adjacent to the first virtual boundary as the first virtual boundary moves in a first direction, and performing filtering of pixels adjacent to the second virtual boundary as the second virtual boundary moves in a second direction. The first direction may be the same as the second direction. The first direction may be different from the second direction. The first direction may be opposite to the second direction. The first virtual boundary may be spaced apart from the second virtual boundary. The first virtual boundary may be parallel to the second virtual boundary. The first virtual boundary may be perpendicular to the second virtual boundary. The first virtual boundary may be angled with respect to the second virtual boundary. The first virtual boundary may intersect with the second virtual boundary. The updated area may be in the center of the image, with at least two sides of the updated area adjacent to each of the unupdated areas. The updated area may be adjacent to a corner of the image, with at least two sides of the updated area adjacent to each of the unupdated areas. The updated area may be adjacent to both the first and second virtual boundaries. This method may further include determining at least one of a third condition, where both sides of the second segment at the second virtual boundary are within the updated area or both sides of the second segment are within the unupdated area, and a fourth condition, where both sides of the second segment at the second virtual boundary are not within the updated area or both sides of the second segment are not within the unupdated area. This method may further include performing a first type of filtering on at least one pixel adjacent to the second segment, based on the determination of a third condition.This method may further include performing a second different type of filtering on at least one pixel adjacent to the second segment based on the determination of the fourth condition. At least two updated areas may be provided that are spaced apart from each other at the same time. At least two unupdated areas may be provided that are spaced apart from each other at the same time.

[0189] According to one exemplary embodiment, an exemplary device is provided comprising at least one processor and at least one non-transient memory storing instructions, the instructions, when executed by at least one processor, cause the device to determine a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary; determine at least one of a first condition in which both sides of a first segment at the first virtual boundary are in an updated area or both sides of the first segment are in an unupdated area, or a second condition in which both sides of the first segment at the first virtual boundary are not in an updated area or both sides of the first segment are not in an unupdated area; perform a first type of filtering of at least one pixel adjacent to the first segment based on the determination of the first condition; and perform a second different type of filtering of at least one pixel adjacent to the first segment based on the determination of the second condition.

[0190] The second condition may include the case where one side of the first segment at the first virtual boundary lies within an updated area, and the other side of the first segment at the first virtual boundary lies within an unupdated area. The second different type of filtering may include not performing in-loop filtering on at least one pixel in the updated area when filtering at least one pixel requires the use of encoding information from the unupdated area. The second different type of filtering may include performing in-loop filtering on at least one pixel in the updated area, which includes using encoding information from the unupdated area derived from the updated area, or using encoding information from the unupdated area set to at least one default value, when filtering at least one pixel requires the use of encoding information from the unupdated area. The second different type of filtering may include filtering at least one pixel in the first area, which includes using encoding information from the second area derived from the first area, or using encoding information from the second area set to at least one value, when the encoding information from the second area is used to perform filtering on at least one pixel in the first area. The first type of filtering may include in-loop filtering. The second type of filtering may include in-loop filtering. The image may include a sequentially decoded updated image or an image being recovered. The instruction, when executed by at least one processor, may cause the device to embed pixels from pixels in the first area on the first side of the first virtual boundary into the second area on the second side of the first virtual boundary, in response to the fact that pixels in the second area are used to perform filtering of at least one pixel in the first area.The instruction, when executed by at least one processor, may cause the device to replace pixels in the second area on the second side of the first virtual boundary with pixels extrapolated from the first area on the first side of the first virtual boundary, in response that pixels in the second area are used to perform filtering of at least one pixel in the first area. The instruction, when executed by at least one processor, may cause the device to determine a first output for filtering at least one pixel in the first area on the first side of the first virtual boundary, if the encoded information for the first area and the encoded information for the second area on the second side of the first virtual boundary are available for filtering at least one pixel in the first area, and to determine a second output for filtering at least one pixel in the first area, if the encoded information for the second area is not available for filtering at least one pixel in the first area. The instruction, when executed by at least one processor, may cause the device to determine the difference between a first output and a second output, and to determine an output for filtering at least one pixel in the second area using at least partially the difference or an approximation of the difference. The encoded information for the second area may include the output for filtering at least one pixel in the second area. The position of at least one pixel in the second area may correspond to the position of at least one pixel in the first area. The instruction, when executed by at least one processor, may cause the device to determine the difference between a first output and a second output, to determine an initial output for filtering at least one pixel in the second area, and to determine a final output for filtering at least one pixel in the second area by at least partially subtracting this difference from the initial output, and the encoded information for the second area may include the final output for filtering at least one pixel in the second area.The instruction, when executed by at least one processor, may cause the device to determine the final output of filtering at least one pixel in a second area by at least partially subtracting a weighted difference contribution from the initial output. At least one of the first or second filtering may include at least one of deblocking filtering, sample-adaptive offset edge offset filtering, bilateral filtering for luminance, bilateral filtering for saturation, inter-component sample-adaptive offset filtering, adaptive loop filtering, or inter-component adaptive loop filtering. The instruction, when executed by at least one processor, may cause the device to disable filtering at least one pixel in a first area on a first side of a first virtual boundary up to a number of pixel positions from the first virtual boundary. The instruction, when executed by at least one processor, may cause the device to perform filtering at least one pixel in a first area on a first side of a first virtual boundary up to a number of pixel positions from the first virtual boundary. The instruction, when executed by at least one processor, may cause the device to filter pixels adjacent to the first virtual boundary as the first virtual boundary moves in a first direction, and to filter pixels adjacent to the second virtual boundary as the second virtual boundary moves in a second direction. The first direction may be the same as the second direction. The first direction may be different from the second direction. The first direction may be opposite to the second direction. The first virtual boundary may be spaced apart from the second virtual boundary. The first virtual boundary may be parallel to the second virtual boundary. The first virtual boundary may be perpendicular to the second virtual boundary. The first virtual boundary may be angled with respect to the second virtual boundary. The first virtual boundary may intersect with the second virtual boundary. The updated area may be in the center of the image, with at least two sides of the updated area adjacent to their respective unupdated areas. The updated area may be adjacent to a corner of the image, such that at least two of its sides are adjacent to the respective unupdated areas.The updated area may be adjacent to both the first and second virtual boundaries. The instruction, when executed by at least one processor, may cause the device to determine at least one of a third condition, where both sides of the second segment at the second virtual boundary are within an updated area or both sides of the second segment are within an unupdated area, and a fourth condition, where both sides of the second segment at the second virtual boundary are not within an updated area or both sides of the second segment are not within an unupdated area. The instruction, when executed by at least one processor, may cause the device to perform a first type of filtering of at least one pixel adjacent to the second segment based on the determination of the third condition. The device may be further configured to perform a second different type of filtering of at least one pixel adjacent to the second segment based on the determination of the fourth condition. At least two updated areas spaced apart from each other may be provided. At least two unupdated areas spaced apart from each other may be provided.

[0191] According to one exemplary embodiment, an exemplary apparatus is provided comprising: means for determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary; means for determining at least one of a first condition in which both sides of a first segment at the first virtual boundary are within an updated area or both sides of the first segment are within an unupdated area, or a second condition in which both sides of the first segment at the first virtual boundary are not within an updated area or both sides of the first segment are not within an unupdated area; means for performing a first type of filtering of at least one pixel adjacent to the first segment based on the determination of the first condition; and means for performing a second different type of filtering of at least one pixel adjacent to the first segment based on the determination of the second condition.

[0192] According to one exemplary embodiment, an exemplary non-transient program storage device readable by the device is provided, which tangibly embodies a program of instructions executable by the device to perform operations, the operations of which include determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary; determining at least one of a first condition in which both sides of a first segment at the first virtual boundary are in an updated area or both sides of a first segment are in an unupdated area, or a second condition in which both sides of a first segment at the first virtual boundary are not in an updated area or both sides of a first segment are not in an unupdated area; performing a first type of filtering of at least one pixel adjacent to the first segment based on the determination of the first condition; and performing a second different type of filtering of at least one pixel adjacent to the first segment based on the determination of the second condition.

[0193] See also Figure 27, according to one exemplary embodiment, an exemplary method is provided which, as shown by block 2702, determines a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary; performs filtering of pixels adjacent to the first virtual boundary as the first virtual boundary moves in a first direction, as shown by block 2704; and performs filtering of pixels adjacent to the second virtual boundary as the second virtual boundary moves in a second direction, as shown by block 2706. The first direction may be the same as the second direction. The first direction may be different from the second direction. The first direction may be opposite to the second direction. The first virtual boundary may be spaced apart from the second virtual boundary. The first virtual boundary may be parallel to the second virtual boundary. The first virtual boundary may be perpendicular to the second virtual boundary. The first virtual boundary may be angled with respect to the second virtual boundary. The first virtual boundary may intersect with the second virtual boundary. The updated area may be in the center of the image, with at least two sides of the updated area adjacent to each unupdated area. The updated area may be adjacent to a corner of the image, with at least two sides of the updated area adjacent to each unupdated area. The updated area may be adjacent to both the first and second virtual boundaries. This method may further include determining at least one of a third condition, where both sides of the second segment at the second virtual boundary are within the updated area or both sides of the second segment are within the unupdated area, and a fourth condition, where both sides of the second segment at the second virtual boundary are not within the updated area or both sides of the second segment are not within the unupdated area. Based on the determination of the third condition, this method may further include performing a first type of filtering on at least one pixel adjacent to the second segment. This method may further include performing a second different type of filtering on at least one pixel adjacent to the second segment, based on the determination of the fourth condition.At least two updated areas, spaced apart from each other, may be provided. At least two unupdated areas, spaced apart from each other, may be provided.

[0194] According to one exemplary embodiment, an exemplary device is provided comprising at least one processor and at least one non-transient memory storing instructions, which, when executed by at least one processor, causes the device to determine a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary; to perform filtering of pixels adjacent to the first virtual boundary as the first virtual boundary moves in a first direction; and to perform filtering of pixels adjacent to the second virtual boundary as the second virtual boundary moves in a second direction.

[0195] According to one exemplary embodiment, an exemplary apparatus is provided comprising means for determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary; means for performing filtering of pixels adjacent to the first virtual boundary as the first virtual boundary moves in a first direction; and means for performing filtering of pixels adjacent to the second virtual boundary as the second virtual boundary moves in a second direction.

[0196] According to one exemplary embodiment, an exemplary non-transient program storage device readable by the device is provided, which tangibly embodies a program of instructions executable by the device to perform operations, the operations of which include determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary; performing filtering of pixels adjacent to the first virtual boundary as the first virtual boundary moves in a first direction; and performing filtering of pixels adjacent to the second virtual boundary as the second virtual boundary moves in a second direction.

[0197] According to one exemplary embodiment, an exemplary method is provided for determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a second virtual boundary; determining at least one of a first condition, where both sides of a first segment in the first virtual boundary are in an updated area or both sides of the first segment are in an unupdated area, or a second condition, where one side of the first segment in the first virtual boundary is in an updated area and the other side of the first segment in the first virtual boundary is in an unupdated area; performing a first type of filtering on at least one pixel of the first segment based on the determination of the first condition; and performing a second different type of filtering on at least one pixel of the first segment based on the determination of the second condition.

[0198] According to one exemplary embodiment, an exemplary device is provided comprising at least one processor and at least one non-transient memory storing instructions, the instructions, when executed by at least one processor, cause the device to determine a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a second virtual boundary; determine at least one of a first condition, where both sides of a first segment in the first virtual boundary are in an updated area or both sides of a first segment are in an unupdated area, or a second condition, where one side of a first segment in the first virtual boundary is in an updated area and the other side of a first segment in the first virtual boundary is in an unupdated area; perform a first type of filtering on at least one pixel of the first segment based on the determination of the first condition; and perform a second different type of filtering on at least one pixel of the first segment based on the determination of the second condition.

[0199] According to one exemplary embodiment, an exemplary apparatus is provided comprising: means for determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a second virtual boundary; means for determining at least one of a first condition, where both sides of a first segment in the first virtual boundary are in an updated area or both sides of the first segment are in an unupdated area, or a second condition, where one side of the first segment in the first virtual boundary is in an updated area and the other side of the first segment in the first virtual boundary is in an unupdated area; means for performing a first type of filtering on at least one pixel of the first segment based on the determination of the first condition; and means for performing a second different type of filtering on at least one pixel of the first segment based on the determination of the second condition.

[0200] According to one exemplary embodiment, an exemplary non-transient program storage device readable by the device is provided, which tangibly embodies a program of instructions executable by the device to perform operations, the operations of which include determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a second virtual boundary; determining at least one of a first condition, where both sides of a first segment in the first virtual boundary are in an updated area or both sides of the first segment are in an unupdated area, or a second condition, where one side of the first segment in the first virtual boundary is in an updated area and the other side of the first segment in the first virtual boundary is in an unupdated area; performing a first kind of filtering of at least one pixel of the first segment based on the determination of the first condition; and performing a second different kind of filtering of at least one pixel of the first segment based on the determination of the second condition.

[0201] References to "computer," "processor," etc., should be understood to include not only computers with various architectures such as single-processor / multi-processor architectures and sequential / parallel architectures, but also special circuits such as field-programmable gate arrays (FPGAs), application-specific circuits (ASICs), signal processing devices, and other processing circuits. References to computer programs, instructions, code, etc., should be understood to include the programmable content of hardware devices, such as instructions for processors, or software or firmware for programmable processors, such as configuration settings for fixed-function devices, gate arrays, or programmable logic devices.

[0202] As used herein, the terms “circuitry,” “circuit,” and variations thereof may refer to any of the following: (a) implementations of hardware circuits, such as implementations in analog and / or digital circuits; (b) (where applicable) combinations of circuits and software (and / or firmware), such as (i) a combination of processors or (ii) a part of a processor / software that includes digital signal processors, software, and memory, which work together to cause a device to perform various functions; and (c) a microprocessor or part of a microprocessor that requires software or firmware for operation even if the software or firmware is not physically present. As a further example, as used herein, the term “circuit” may also refer simply to a processor (or more processors) or part of a processor and the software and / or firmware implementations associated with it (or them). The term “circuit” may also refer, for example, to a baseband integrated circuit or application processor integrated circuit for a mobile phone, or a similar integrated circuit in a server, cellular network device, or another network device, where applicable to a particular element. Circuitry and circuit may also be used to mean a function or process used to perform a method.

[0203] In the diagram, the arrows between individual blocks represent the operational connections between blocks and the direction of data flow in those connections.

[0204] It should be understood that the above description is merely illustrative. Various alternatives and modifications can be devised by those skilled in the art. For example, the features enumerated in various dependent claims can be combined with each other in any suitable combination. In addition, features from the various embodiments described above can be selectively combined into new embodiments. Accordingly, this description is intended to encompass all such alternatives, modifications, and differences that fall within the scope of the appended claims.

Claims

1. Determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary. The first condition is that both sides of the first segment at the first virtual boundary are within the updated area, or both sides of the first segment are within the unupdated area, or Determine at least one of the second conditions, which is when both sides of the first segment at the first virtual boundary are not within the updated area, or when both sides of the first segment are not within the unupdated area. Based on the determination of the first condition, perform a first type of filtering on at least one pixel adjacent to the first segment, A method comprising performing a second different type of filtering on at least one pixel adjacent to the first segment based on the determination of the second condition.

2. The method according to claim 1, wherein the second condition includes the case where one side of the first segment in the first virtual boundary is in an updated area and the other side of the first segment in the first virtual boundary is in an unupdated area.

3. The second different type of filtering described above is The method according to claim 2, wherein if filtering of the at least one pixel requires the use of the encoding information of the unupdated area, in-loop filtering is not performed on the at least one pixel in the updated area.

4. The second different type of filtering described above is If the filtering of the at least one pixel requires the use of the encoding information of the unupdated area, then for the at least one pixel in the updated area, Using the encoding information of the unupdated area derived from the updated area, or The method according to claim 2, comprising performing in-loop filtering, which includes one of using the encoding information of the unupdated area set to at least one default value.

5. The second different type of filtering described above is The method according to any one of claims 1 to 4, wherein the encoding information of the second area is used to perform filtering of at least one pixel of the first area, and includes using the encoding information of the second area derived from the first area, or using the encoding information of the second area set to at least one value.

6. The method according to any one of claims 1 to 5, wherein the first type of filtering includes in-loop filtering.

7. The method according to claim 1 or 6, wherein the second type of filtering includes in-loop filtering.

8. The method according to any one of claims 1 to 7, wherein the image includes a gradually decoded updated image or an image being recovered.

9. The method according to any one of claims 1 to 8, further comprising embedding pixels from the pixels in the first area on the first side of the first virtual boundary into the second area on the second side of the first virtual boundary, in response that pixels in the second area are used to perform filtering of at least one pixel in the first area.

10. The method according to any one of claims 1 to 9, further comprising replacing pixels in the second area on the second side of the first virtual boundary with pixels extrapolated from the first area on the first side of the first virtual boundary, in response that pixels in the second area are used to perform filtering of at least one pixel in the first area.

11. If the encoded information of the first area and the encoded information of the second area on the second side of the first virtual boundary are available for filtering at least one pixel of the first area, then a first output of the filtering of at least one pixel of the first area on the first side of the first virtual boundary is determined. The method according to any one of claims 1 to 10, further comprising determining a second output of the filtering of the at least one pixel in the first area if the encoded information of the second area is not available for the filtering of the at least one pixel in the first area.

12. Determining the difference between the first output and the second output, The method according to claim 11, further comprising using at least partially the difference or an approximation of the difference to determine the output of filtering for at least one pixel of the second area.

13. The method according to claim 12, wherein the encoded information of the second area includes the output of the filtering of at least one pixel of the second area.

14. The method according to claim 12 or 13, wherein the position of the at least one pixel in the second area corresponds to the position of the at least one pixel in the first area.

15. Determining the difference between the first output and the second output, Determining the initial output of filtering at least one pixel in the second area, Subtracting the difference at least partially from the initial output to determine the final output of the filtering for at least one pixel in the second area, The method according to any one of claims 11 to 14, further comprising determining that the encoded information of the second area includes the final output of the filtering of at least one pixel of the second area.

16. The method according to claim 15, further comprising determining the final output of the filtering of the at least one pixel in the second area by at least partially subtracting the weighted contribution of the difference from the initial output.

17. At least one of the first filtering or the second filtering is Deblocking filtering, Sample-adaptive offset edge offset filtering, Bilateral filtering for brightness, Bilateral filtering for saturation, Inter-component sample adaptive offset filtering, Adaptive loop filtering, or The method according to any one of claims 1 to 16, comprising at least one of intercomponent adaptive loop filtering.

18. The method according to any one of claims 1 to 17, further comprising disabling filtering of at least one pixel in the first area on the first side of the first virtual boundary up to a plurality of pixel positions from the first virtual boundary.

19. The method according to any one of claims 1 to 18, further comprising filtering at least one pixel of a first area on the first side of the first virtual boundary up to a plurality of pixel positions from the first virtual boundary.

20. The first virtual boundary moves in a first direction while filtering is performed on pixels adjacent to the first virtual boundary, The method according to any one of claims 1 to 19, further comprising performing filtering of pixels adjacent to the second virtual boundary while the second virtual boundary moves in a second direction.

21. The method according to claim 20, wherein the first direction is the same as the second direction.

22. The method according to claim 20, wherein the first direction is different from the second direction.

23. The method according to claim 22, wherein the first direction is opposite to the second direction.

24. The method according to any one of claims 20 to 23, wherein the first virtual boundary is spaced apart from the second virtual boundary.

25. The method according to any one of claims 20 to 24, wherein the first virtual boundary is parallel to the second virtual boundary.

26. The method according to any one of claims 20 to 23, wherein the first virtual boundary is perpendicular to the second virtual boundary.

27. The method according to any one of claims 20 to 23, wherein the first virtual boundary is angled with respect to the second virtual boundary.

28. The method according to any one of claims 20 to 23 or 26 to 27, wherein the first virtual boundary intersects the second virtual boundary.

29. The method according to any one of claims 2 to 28, wherein the updated area is located in the center of the image, with at least two sides of the updated area adjacent to each of the unupdated areas.

30. The method according to any one of claims 2 to 28, wherein the updated area is adjacent to a corner of the image, with at least two sides of the updated area adjacent to each of the unupdated areas.

31. The method according to any one of claims 2 to 28, wherein the updated area is adjacent to both the first and second virtual boundaries.

32. A third condition is when both sides of the second segment at the second virtual boundary are within the updated area, or when both sides of the second segment are within the unupdated area, and The method according to any one of claims 2 to 28, further comprising determining at least one of the fourth conditions, where both sides of the second segment at the second virtual boundary are not in an updated area, or where both sides of the second segment are not in an unupdated area.

33. The method according to claim 32, further comprising performing the first type of filtering of at least one pixel adjacent to the second segment based on the determination of the third condition.

34. The method according to claim 32 or 33, further comprising performing the second different type of filtering of at least one pixel adjacent to the second segment based on the determination of the fourth condition.

35. The method according to any one of claims 2 to 34, wherein at least two updated areas spaced apart from each other are provided.

36. The method according to any one of claims 2 to 35, wherein at least two unupdated areas are provided that are spaced apart from each other.

37. At least one processor, A device comprising at least one non-transient memory storing instructions, wherein when the instruction is executed by the at least one processor, the device provides Determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary. The first condition is that both sides of the first segment at the first virtual boundary are within the updated area, or both sides of the first segment are within the unupdated area, or Determine at least one of the second conditions, which is when both sides of the first segment at the first virtual boundary are not within the updated area, or when both sides of the first segment are not within the unupdated area. Based on the determination of the first condition, perform a first type of filtering on at least one pixel adjacent to the first segment, An apparatus that, based on the determination of the second condition, causes to perform a second different type of filtering on at least one pixel adjacent to the first segment.

38. The apparatus according to claim 37, wherein the second condition includes the case where one side of the first segment in the first virtual boundary is in an updated area and the other side of the first segment in the first virtual boundary is in an unupdated area.

39. The second different type of filtering described above is The apparatus according to claim 38, wherein if filtering of the at least one pixel requires the use of the encoding information of the unupdated area, in-loop filtering is not performed on the at least one pixel in the updated area.

40. The second different type of filtering described above is If the filtering of the at least one pixel requires the use of the encoding information of the unupdated area, then for the at least one pixel in the updated area, Using the encoding information of the unupdated area derived from the updated area, or The apparatus according to claim 38, comprising performing in-loop filtering, which includes one of using the encoding information of the unupdated area set to at least one default value.

41. The second different type of filtering described above is The apparatus according to any one of claims 37 to 40, comprising filtering at least one pixel of the first area, using the encoded information of the second area derived from the first area, or using the encoded information of the second area set to at least one value, when the encoded information of the second area is used to perform filtering of at least one pixel of the first area.

42. The apparatus according to any one of claims 37 to 41, wherein the first type of filtering includes in-loop filtering.

43. The apparatus according to claim 37 or 42, wherein the second type of filtering includes in-loop filtering.

44. The apparatus according to any one of claims 37 to 43, wherein the image includes a gradually decoded updated image or an image being recovered.

45. When the instruction is executed by the at least one processor, the device: The apparatus according to any one of claims 37 to 44, wherein, in response to pixels in the second area being used to perform filtering of at least one pixel in the first area, the apparatus causes pixels from the first area on the first side of the first virtual boundary to be embedded in the second area on the second side of the first virtual boundary.

46. When the instruction is executed by the at least one processor, the device: The apparatus according to any one of claims 37 to 45, which, in response to the fact that pixels in the second area are used to perform filtering of at least one pixel in the first area, causes the pixels in the second area on the second side of the first virtual boundary to be replaced with pixels extrapolated from the first area on the first side of the first virtual boundary.

47. When the instruction is executed by the at least one processor, the device: If the encoded information of the first area and the encoded information of the second area on the second side of the first virtual boundary are available for filtering at least one pixel of the first area, then a first output of the filtering of at least one pixel of the first area on the first side of the first virtual boundary is determined. The apparatus according to any one of claims 37 to 46, wherein if the encoded information of the second area is not available for filtering of the at least one pixel of the first area, the apparatus determines a second output of the filtering of the at least one pixel of the first area.

48. When the instruction is executed by the at least one processor, the device: Determining the difference between the first output and the second output, The apparatus according to claim 47, wherein the apparatus is configured to determine the filtering output of at least one pixel of the second area using at least partially the difference or an approximation of the difference.

49. The apparatus according to claim 48, wherein the encoded information of the second area includes the output of the filtering of at least one pixel of the second area.

50. The apparatus according to claim 12 or 13, wherein the position of the at least one pixel in the second area corresponds to the position of the at least one pixel in the first area.

51. When the instruction is executed by the at least one processor, the device: Determining the difference between the first output and the second output, Determining the initial output of filtering at least one pixel in the second area, Subtracting the difference at least partially from the initial output to determine the final output of the filtering for at least one pixel in the second area, The apparatus according to any one of claims 47 to 50, wherein the encoded information of the second area is used to perform the determination of the final output of the filtering of at least one pixel of the second area.

52. When the instruction is executed by the at least one processor, the device: The apparatus according to claim 51, wherein the weighted contribution of the difference is at least partially subtracted from the initial output to determine the final output of the filtering of the at least one pixel in the second area.

53. At least one of the first filtering or the second filtering is Deblocking filtering, Sample-adaptive offset edge offset filtering, Bilateral filtering for brightness, Bilateral filtering for saturation, Inter-component sample adaptive offset filtering, Adaptive loop filtering, or The apparatus according to any one of claims 37 to 52, comprising at least one of intercomponent adaptive loop filtering.

54. When the instruction is executed by the at least one processor, the device: The apparatus according to any one of claims 37 to 53, which causes the filtering of at least one pixel in the first area on the first side of the first virtual boundary to be disabled up to a plurality of pixel positions from the first virtual boundary.

55. When the instruction is executed by the at least one processor, the device: The apparatus according to any one of claims 37 to 54, wherein filtering of at least one pixel of a first area on the first side of the first virtual boundary is performed from the first virtual boundary to a plurality of pixel positions.

56. When the instruction is executed by the at least one processor, the device: The first virtual boundary moves in a first direction, and pixels adjacent to the first virtual boundary are filtered. The apparatus according to any one of claims 37 to 55, wherein the second virtual boundary moves in a second direction, and the apparatus performs filtering of pixels adjacent to the second virtual boundary.

57. The apparatus according to claim 56, wherein the first direction is the same as the second direction.

58. The apparatus according to claim 56, wherein the first direction is different from the second direction.

59. The apparatus according to claim 58, wherein the first direction is opposite to the second direction.

60. The apparatus according to any one of claims 56 to 59, wherein the first virtual boundary is spaced apart from the second virtual boundary.

61. The apparatus according to any one of claims 56 to 60, wherein the first virtual boundary is parallel to the second virtual boundary.

62. The apparatus according to any one of claims 56 to 59, wherein the first virtual boundary is perpendicular to the second virtual boundary.

63. The apparatus according to any one of claims 56 to 59, wherein the first virtual boundary is angled with respect to the second virtual boundary.

64. The apparatus according to any one of claims 56 to 59 or 62 to 63, wherein the first virtual boundary intersects with the second virtual boundary.

65. The apparatus according to any one of claims 38 to 64, wherein the updated area is located in the center of the image, with at least two sides of the updated area adjacent to each of the unupdated areas.

66. The apparatus according to any one of claims 38 to 64, wherein the updated area is adjacent to a corner of the image, with at least two sides of the updated area adjacent to the respective unupdated areas.

67. The apparatus according to any one of claims 38 to 64, wherein the updated area is adjacent to both the first and second virtual boundaries.

68. When the instruction is executed by the at least one processor, the device: A third condition is when both sides of the second segment at the second virtual boundary are within the updated area, or when both sides of the second segment are within the unupdated area, and The apparatus according to any one of claims 38 to 64, which causes the apparatus to determine at least one of the fourth conditions, where both sides of the second segment at the second virtual boundary are not in an updated area, or where both sides of the second segment are not in an unupdated area.

69. The apparatus according to claim 68, wherein, when the instruction is executed by the at least one processor, the apparatus is caused to perform the first type of filtering of at least one pixel adjacent to the second segment based on the determination of the third condition.

70. The apparatus according to claim 68 or 69, wherein, when the instruction is executed by the at least one processor, the apparatus is caused to perform the second different type of filtering of at least one pixel adjacent to the second segment based on the determination of the fourth condition.

71. The apparatus according to any one of claims 38 to 70, wherein at least two updated areas are provided that are spaced apart from each other.

72. The apparatus according to any one of claims 38 to 71, wherein at least two unupdated areas are provided that are spaced apart from each other.

73. A means for determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary, The first condition is that both sides of the first segment at the first virtual boundary are within the updated area, or both sides of the first segment are within the unupdated area, or Means for determining at least one of the second conditions, where both sides of the first segment at the first virtual boundary are not within the updated area, or where both sides of the first segment are not within the unupdated area, Means for performing a first type of filtering of at least one pixel adjacent to the first segment based on the determination of the first condition, An apparatus comprising means for performing a second different type of filtering of at least one pixel adjacent to the first segment based on the determination of the second condition.

74. A non-transient program storage device readable by the device, which tangibly embodies a program of instructions that can be executed using the device to perform an action, wherein the action is Determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include at least a first virtual boundary and a second virtual boundary. The first condition is that both sides of the first segment at the first virtual boundary are within the updated area, or both sides of the first segment are within the unupdated area, or Determine at least one of the second conditions, which is when both sides of the first segment at the first virtual boundary are not within the updated area, or when both sides of the first segment are not within the unupdated area. Based on the determination of the first condition, perform a first type of filtering on at least one pixel adjacent to the first segment, A non-transient program storage device, comprising performing a second different type of filtering on at least one pixel adjacent to the first segment based on the determination of the second condition.

75. Determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary, The first virtual boundary moves in a first direction while filtering is performed on pixels adjacent to the first virtual boundary, A method comprising performing filtering of pixels adjacent to the second virtual boundary while the second virtual boundary moves in a second direction.

76. The method according to claim 75, wherein the first direction is the same as the second direction.

77. The method according to claim 75, wherein the first direction is different from the second direction.

78. The method according to claim 77, wherein the first direction is opposite to the second direction.

79. The method according to any one of claims 75 to 78, wherein the first virtual boundary is spaced apart from the second virtual boundary.

80. The method according to any one of claims 75 to 79, wherein the first virtual boundary is parallel to the second virtual boundary.

81. The method according to any one of claims 75 to 78, wherein the first virtual boundary is perpendicular to the second virtual boundary.

82. The method according to any one of claims 75 to 78, wherein the first virtual boundary is angled with respect to the second virtual boundary.

83. The method according to any one of claims 75 to 78 or 81 to 82, wherein the first virtual boundary intersects the second virtual boundary.

84. The method according to any one of claims 75 to 83, wherein the updated area is located in the center of the image such that at least two sides of the updated area are adjacent to the respective unupdated areas.

85. The method according to any one of claims 75 to 83, wherein the updated area is adjacent to a corner of the image, with at least two sides of the updated area adjacent to each of the unupdated areas.

86. The method according to any one of claims 75 to 83, wherein the updated area is adjacent to both the first and second virtual boundaries.

87. A third condition is when both sides of the second segment at the second virtual boundary are within the updated area, or when both sides of the second segment are within the unupdated area, and The method according to any one of claims 75 to 83, further comprising determining at least one of the fourth conditions, where both sides of the second segment at the second virtual boundary are not in an updated area, or where both sides of the second segment are not in an unupdated area.

88. The method according to claim 87, further comprising performing the first type of filtering of at least one pixel adjacent to the second segment based on the determination of the third condition.

89. The method according to claim 87 or 88, further comprising performing the second different type of filtering of at least one pixel adjacent to the second segment based on the determination of the fourth condition.

90. The method according to any one of claims 87 to 89, wherein at least two updated areas spaced apart from each other are provided.

91. The method according to any one of claims 87 to 90, wherein at least two unupdated areas are provided that are spaced apart from each other.

92. At least one processor, A device comprising at least one non-transient memory storing instructions, wherein when the instruction is executed by the at least one processor, the device provides Determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary, The first virtual boundary moves in a first direction while filtering is performed on pixels adjacent to the first virtual boundary, A device that performs filtering of pixels adjacent to the second virtual boundary while the second virtual boundary moves in a second direction.

93. A means for determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary, Means for performing filtering of pixels adjacent to the first virtual boundary while the first virtual boundary moves in a first direction, An apparatus comprising means for performing filtering of pixels adjacent to the second virtual boundary while the second virtual boundary moves in a second direction.

94. A non-transient program storage device readable by the device, which tangibly embodies a program of instructions that can be executed using the device to perform an action, wherein the action is Determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a different second virtual boundary, The first virtual boundary moves in a first direction while filtering is performed on pixels adjacent to the first virtual boundary, A non-transient program storage device, comprising performing filtering of pixels adjacent to the second virtual boundary while the second virtual boundary moves in a second direction.

95. Determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a second virtual boundary, The first condition is that both sides of the first segment at the first virtual boundary are within the updated area, or both sides of the first segment are within the unupdated area, or Determining at least one of the second conditions where one side of the first segment in the first virtual boundary is in an updated area and the other side of the first segment in the first virtual boundary is in an unupdated area, Based on the determination of the first condition, perform a first type of filtering on at least one pixel of the first segment, A method comprising performing a second different type of filtering on at least one pixel of the first segment based on the determination of the second condition.

96. At least one processor, A device comprising at least one non-transient memory storing instructions, wherein when the instruction is executed by the at least one processor, the device provides Determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a second virtual boundary, The first condition is that both sides of the first segment at the first virtual boundary are within the updated area, or both sides of the first segment are within the unupdated area, or Determining at least one of the second conditions where one side of the first segment in the first virtual boundary is in an updated area and the other side of the first segment in the first virtual boundary is in an unupdated area, Based on the determination of the first condition, perform a first type of filtering on at least one pixel of the first segment, An apparatus that, based on the determination of the second condition, causes to perform a second different type of filtering on at least one pixel of the first segment.

97. A means for determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a second virtual boundary, The first condition is that both sides of the first segment at the first virtual boundary are within the updated area, or both sides of the first segment are within the unupdated area, or Means for determining at least one of the second conditions, where one side of the first segment in the first virtual boundary is in an updated area and the other side of the first segment in the first virtual boundary is in an unupdated area, Means for performing a first type of filtering on at least one pixel of the first segment based on the determination of the first condition, An apparatus comprising means for performing a second different type of filtering on at least one pixel of the first segment based on the determination of the second condition.

98. A non-transient program storage device readable by the device, which tangibly embodies a program of instructions that can be executed using the device to perform an action, wherein the action is Determining a plurality of virtual boundaries for at least a portion of an image, wherein the plurality of virtual boundaries include a first virtual boundary and a second virtual boundary, The first condition is that both sides of the first segment at the first virtual boundary are within the updated area, or both sides of the first segment are within the unupdated area, or Determining at least one of the second conditions where one side of the first segment in the first virtual boundary is in an updated area and the other side of the first segment in the first virtual boundary is in an unupdated area, Based on the determination of the first condition, perform a first type of filtering on at least one pixel of the first segment, A non-transient program storage device, comprising performing a second different type of filtering on at least one pixel of the first segment based on the determination of the second condition.

Images