Neural Network-Based Deblocking Filter

Abstract

A method and apparatus are provided for reducing artifacts in compressed images using a neural network-based deblocking filter. The method may include receiving at least one reconstructed image, each reconstructed image including one or more reconstructed blocks, and extracting a boundary region associated with a boundary of one or more reconstructed blocks in the at least one reconstructed image. The extracted boundary region may be input to a trained deblocking model to generate a boundary region having reduced artifacts, the trained deblocking model being trained on training data based on estimated compression by a neural image compression (NIC) network. Edge regions associated with the generated boundary region may be removed, and at least one reconstructed image having reduced artifacts may be generated based on the generated boundary region.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 299,227, filed January 13, 2022, and U.S. Patent Application No. 18 / 080,201, filed December 13, 2022, the disclosures of which are incorporated herein by reference in their entireties.

[0002] Embodiments of the present disclosure relate to a neural network-based deblocking filter for block-wise neural image compression (NIC). [Background technology]

[0003] In the related art, block-based intra prediction and residual coding mechanisms have proven very effective for compressing image frames in modern video coding systems such as the HEVC and VVC standards. The entire image is first divided into coding tree units (CTUs). For each CTU, the entire image is divided into blocks of various sizes, and a predictive block is generated by copying the boundary pixels of the previous compressed block along various angular directions. The residual between the original block and the predictive block is then compressed. The residual can be coded much more efficiently than the original pixels, achieving better coding performance.

[0004] ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) has been actively exploring potential needs for standardization of future video coding technologies. ISO / IEC JPEG has established a JPEG-AI group focused on AI-based end-to-end neural image compression using deep neural networks (DNNs). Large companies such as Google have funded specialized research projects for NICs. China's AVS standard also formed an AVS-AI specialized group to work on neural image and video compression technologies. Recent successes of these methods have led to increasing industrial interest in advanced neural image and video compression methodologies. Summary of the Invention [Means for solving the problem]

[0005] According to one aspect of the present disclosure, there may be provided a method for reducing artifacts in a compressed image using a neural network-based deblocking filter. The method may be executed by at least one processor and may include receiving at least one reconstructed image, each reconstructed image including one or more reconstructed blocks, extracting boundary regions associated with boundaries of the one or more reconstructed blocks in the at least one reconstructed image, inputting the extracted boundary regions into a trained deblocking model to generate artifact-reduced boundary regions, removing edge regions associated with the artifact-reduced boundary regions, and generating at least one artifact-reduced reconstructed image based on the artifact-reduced boundary regions.

[0006] According to one aspect of the present disclosure, a system for reducing artifacts in a compressed image using a neural network-based deblocking filter may be provided. The system may include at least one memory configured to store computer program code and at least one processor configured to access the computer program code and operate as directed by the computer program code. The program code may include: receiving code configured to cause the at least one processor to receive at least one reconstructed image, each reconstructed image comprising one or more reconstruction blocks; extraction code configured to cause the at least one processor to extract boundary regions associated with boundaries of one or more reconstruction blocks in the at least one reconstructed image; input code configured to cause the at least one processor to input the extracted boundary regions to a trained deblocking model to generate artifact-reduced boundary regions; removal code configured to cause the at least one processor to remove edge regions associated with the artifact-reduced boundary regions; and generation code configured to cause the at least one processor to generate at least one artifact-reduced reconstructed image based on the artifact-reduced boundary regions.

[0007] According to one aspect of the present disclosure, a non-transitory computer-readable medium is provided that, when executed by at least one processor, causes the at least one processor to implement a neural network-based deblocking filter: receive at least one reconstructed image, each reconstructed image including one or more reconstructed blocks; extract boundary regions associated with boundaries of the one or more reconstructed blocks in the at least one reconstructed image; input the extracted boundary regions to a trained deblocking model to generate artifact-reduced boundary regions; remove edge regions associated with the artifact-reduced boundary regions; and generate at least one artifact-reduced reconstructed image based on the artifact-reduced boundary regions.

[0008] Further features, nature and various advantages of the subject matter of the present disclosure will become more apparent in the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a simplified block diagram of a communication system according to one embodiment. [Figure 2] 2 is a block diagram illustrating example components of one or more devices of FIG. 1. [Figure 3A] 1 is a flowchart of an exemplary boundary strength derivation process according to one embodiment. [Figure 3B] FIG. 1 is a schematic diagram of an exemplary boundary strength derivation process according to one embodiment. [Figure 4A] FIG. 2 illustrates an exemplary structure of a neural network-based deblocking filter, according to one embodiment. [Figure 4B] FIG. 2 is another diagram illustrating an exemplary structure of a neural network-based deblocking filter, according to one embodiment. [Figure 5A] FIG. 1 illustrates an exemplary process for extracting a boundary region, according to one embodiment. [Figure 5B]FIG. 10 is another diagram illustrating an exemplary process for extracting a boundary region, according to one embodiment. [Figure 6A] FIG. 1 is a workflow diagram illustrating an exemplary process for reducing artifacts in compressed images using a neural network-based deblocking filter, according to one embodiment. [Figure 6B] FIG. 10 is another workflow diagram illustrating an exemplary process for reducing artifacts in a compressed image using a neural network-based deblocking filter, according to one embodiment. [Figure 6C] FIG. 10 is yet another workflow diagram illustrating an exemplary process for reducing artifacts in a compressed image using a neural network-based deblocking filter, according to one embodiment. [Figure 7] 1 is a flowchart of a process for reducing artifacts in a compressed image using a neural network-based deblocking filter, according to one embodiment. [Figure 8] 1 is a diagram of a video coding and decoding environment, according to one embodiment. [Figure 9] FIG. 1 is a diagram of a computer system suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] As mentioned above, the organization is actively exploring potential needs for standardization focused on AI-based end-to-end neural image compression using deep neural networks (DNNs).

[0011] 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The communication system 100 may include at least two terminals 140-130 interconnected via a network 150. For unidirectional data transmission, a first terminal 140 may code video data at a local location for transmission to the other terminal 130 via the network 150. The second terminal 130 may receive the coded video data of the other terminal from the network 150, decode the coded data, and display the recovered video data. Unidirectional data transmission may be common in media serving applications, etc.

[0012] 1 shows a second pair of terminals 110, 120 provided to support bidirectional transmission of coded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal 110, 120 may code video data captured at a local location for transmission to the other terminal over network 150. Each terminal 110, 120 may also receive coded video data transmitted by the other terminal, decode the coded data, and display the recovered video data on a local display device.

[0013] In FIG. 1 , terminals 140-120 may be illustrated as servers, personal computers, and smartphones, although the principles of the present disclosure are not so limited. Embodiments of the present disclosure contemplate application in laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. Network 150 represents any number of networks that convey coded video data between terminals 140-120, including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 150 may not be important to the operation of the present disclosure, unless otherwise described herein.

[0014] FIG. 2 is a block diagram of example components of one or more devices of FIG.

[0015] Device 200 may correspond to any one of terminals 110 to 140. As shown in FIG. 2, device 200 may include a bus 210, a processor 220, a memory 230, a storage component 240, an input component 250, an output component 260, and a communication interface 270.

[0016] Bus 210 includes components that enable communication between components of device 200. Processor 220 is implemented in hardware, firmware, or a combination of hardware and software. Processor 220 is a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC), or another type of processing component. In some implementations, processor 220 includes one or more processors that can be programmed to perform functions. Memory 230 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 220.

[0017] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include a hard disk (e.g., a magnetic disk, optical disk, magneto-optical disk, and / or solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium along with a corresponding drive.

[0018] Input components 250 include components that enable device 200 to receive information, such as via user input (e.g., a touchscreen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 250 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 260 include components that provide output information from device 200 (e.g., a display, a speaker, and / or one or more light-emitting diodes (LEDs)).

[0019] Communications interface 270 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable device 200 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections, etc. Communications interface 270 may enable device 200 to receive information from and / or provide information to another device. For example, communications interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0020] Device 200 may perform one or more processes described herein. Device 200 may perform these processes in response to processor 220 executing software instructions stored by a non-transitory computer-readable medium, such as memory (230) and / or storage component 240. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[0021] Software instructions may be loaded into memory 230 and / or storage component 240 from another computer-readable medium or from another device via communications interface 270. When executed, the software instructions stored in memory 230 and / or storage component 240 may cause processor 220 to perform one or more processes described herein. Additionally, or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0022] The number and arrangement of components shown in Figure 2 are provided as an example. In practice, device 200 may include additional, fewer, different, or differently arranged components than those shown in Figure 2. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0023] Given an input image x, the goal of NIC is to generate a compressed representation of x that is compact for storage and transmission, using the image x as input to a DNN encoder.

number

number

number

number

number

number

[0024] Deblocking Filter in HEVC - Boundary Strength Derivation In HEVC, the deblocking filter process may be performed per CU in the same order as the decoding process. Vertical edges may be filtered first (horizontal filtering), and then horizontal edges may be filtered (vertical filtering). For both luma and chroma components, filtering may be applied to 8x8 block boundaries determined to be filtered. In some embodiments, to reduce complexity, 4x4 block boundaries may not be processed.

[0025] The boundary strength (Bs) reflects the strength of the filtering process that may be required at the boundary: a Bs value of 2 indicates strong filtering, 1 indicates weak filtering, and 0 indicates no deblocking filtering.

[0026] In some embodiments, P and Q may be defined as blocks involved in filtering, where P may represent blocks located to the left (for vertical edges) or above (for horizontal edges) of the boundary, and Q may represent blocks located to the right (for vertical edges) or below (for horizontal edges) of the boundary. FIG. 3A may show a process 300 for how a Bs value may be calculated based on an intra-coding mode, the presence of non-zero transform coefficients, a reference picture, the number of motion vectors, and a motion vector difference. As an example, when the MV difference is below a threshold T, Bs is equal to 0. The threshold T is set to 1 pixel. In HEVC, the MV precision is 1 / 4 pixel, and the MV difference threshold is set to 4. In VTM, the MV precision is 1 / 16, and the MV difference is set to 16.

[0027] In some embodiments, Bs may be calculated on a 4x4 block basis, but remapped to an 8x8 grid. The boundary Bs of the 8x8 grid can be chosen by selecting the maximum of the two values ​​of Bs corresponding to the 8 pixels that make up the line of the 4x4 grid.

[0028] Deblocking filter in VVC In VTM5, the deblocking filtering process is almost the same as in HEVC, except for the addition of the following modifications: filter strength of the deblocking filter depending on the average luma level of the reconstructed samples, deblocking tC table extension, a strong deblocking filter for luma, a stronger deblocking filter for chroma, and luma deblocking on a 4x4 sample grid and chroma deblocking on an 8x8 sample grid.

[0029] Filter strength dependent on reconstructed average luma level In HEVC, the filter strength of the deblocking filter may be controlled by variables β and tC, which are derived from the averaged quantization parameter qPL. In VTM5, the deblocking filter may control its strength by adding an offset to qPL according to the luma level of the reconstructed sample. The reconstructed luma level LL is derived as shown in Equation (1), and the sample values ​​pi,k and qi,k for i=0..3 and k=0 and 3 are derived as shown in Figure 3B. LL=((p0,0+p0,3+q0,0+q0,3)≫2) / (1≪bitDepth) ...Formula (1)

[0030] The variable qPL may be derived as shown in equation (2), where QpQ and QpP represent the quantization parameters of the coding unit that includes samples q0,0 and p0,0, respectively. The offset qpOffset depends on the transfer function, and the value may be signaled in the SPS. qP L =((Qp Q +Qp P +1)≫1)+qpOffset ...Formula (2)

[0031] In VTM5, the maximum QP has increased from 51 to 63, and it is desired to reflect corresponding changes to the deblocking tables that derive the value of the deblocking parameter tC based on the block QP. Below is the updated tC table to accommodate the expanded QP range. tC=[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,2,2,2,2,3,3,3,3,4,4, 4,5,5,6,6,7,8,9,10,11,13,14,16,18,20,22,25,28,31,35,39,44,50,56,63,70,79,88,99]

[0032] Deblocking filter for luma If samples on one side of the boundary belong to a large block, a bilinear filter (a stronger deblocking filter) is used. Samples belonging to a large block are defined as having a width of 32 or more for vertical edges and a height of 32 or more for horizontal edges. Then, the block boundary samples pi, for i = 0 to Sp-1, and qi, for j = 0 to Sq-1, are replaced by linear interpolation as follows: p i '=(f i *Middle s,t +(64-f i )*P s +32)≫6), p i ±tcPD i ...Equation (3) q j '=(g j *Middle s,t +(64-g j )*Q s +32)≫6), q j ±tcPD j ...Equation (4)

[0033] where tcPD i and tcPD j The term is position-dependent clipping, and g j , f i , Middle s,t , P s and Q s is given below:

[0034] [Table 1]

[0035] The stronger luma filter mentioned above can be used only if all of condition 1, condition 2, and condition 3 are TRUE. Condition 1 is the "big block condition." This condition detects whether the samples on the P and Q sides belong to a big block. Conditions 2 and 3 can be as follows: Condition 2=(d<β)?TRUE:FALSE Condition 3 = StrongFilterCondition = (dpq is less than (β≫2), sp3 + sq3 is less than (3*β≫5), and Abs(p0-q0) is less than (5*t C +1)≫less than 1)?TRUE:FALSE

[0036] Deblocking filter for chroma In some embodiments, the following strong deblocking filter for chroma may be defined as shown in equations (5)-(7): p2'=(3*p3+2*p2+p1+p0+q0+4)≫3 ...Equation (5) p1'=(2*p3+p2+2*p1+p0+q0+q1+4)≫3 ...Equation (6) p0'=(p3+p2+p1+2*p0+q0+q1+q2+4)≫3 ...Equation (7)

[0037] The chroma filters disclosed above may perform deblocking on an 8x8 chroma sample grid. A strong chroma filter may be used on both sides of the block boundary. In some embodiments, a chroma filter may be selected when both sides of the chroma edge are 8 or more (in units of chroma samples) and the following decision, which has three conditions, is met: The first is a boundary strength and large block decision. The second and third are the same as the HEVC luma decision, which are an on / off decision and a strong filter decision, respectively. In the first decision, the boundary strength (Bs) is modified for chroma filtering, as shown in Table 2. The conditions in Table 1 are checked sequentially. If a condition is met, the remaining conditions with lower priority are skipped.

[0038] [Table 2]

[0039] Chroma deblocking may be performed when Bs is equal to 2, or when a large block boundary is detected and Bs is equal to 1. The second and third conditions may be the same as the HEVC luma strong filter decision.

[0040] Sub-block boundary deblocking filter In VVC, a deblocking filter may be enabled on a 4x4 grid for luma and an 8x8 grid for chroma. The deblocking filtering process may be applied to CU boundaries as well as sub-block boundaries. Sub-block boundaries may include prediction unit boundaries introduced by sub-block temporal motion vector prediction (STMVP) and affine modes, and transform unit boundaries introduced by sub-block transform (SBT) and intra-subpartitioning (ISP) modes.

[0041] In some embodiments, for SBT and ISP sub-blocks, the same logic as for TUs in the HEVC deblocking filter may be applied. The deblocking filter may be applied to TU boundaries if there are non-zero coefficients in any sub-block across the edge. For SbTMVP and affine sub-blocks on a 4x4 grid, the same logic as for PUs in the HEVC deblocking filter may be applied. For PU boundaries, the deblocking filter may be applied taking into account the difference between the motion vectors and the reference pictures of neighboring sub-blocks.

[0042] Aspects of the present disclosure relate to a neural network-based deblocking filter for a NIC, and more particularly to a block-wise deblocking filter mechanism for a NIC to deblock artifact regions within block boundaries.

[0043] A deblocking filter may be applied to the decoder side of the NIC. Images reconstructed by a block-by-block NIC contain interblock artifacts. As proposed in this disclosure, a deblocking filter can be used to minimize such artifacts. The deblocking DNN may include a set of convolutional layers or residual layers. An example of a residual block is block diagram 400 in FIG. 4A. The DNN used by image super-resolution may be used here (e.g., by changing the output size to be the same as the input size). An example of a network of residual blocks forming a neural network-based deblocking filter is block diagram 450 in FIG. 4B.

[0044] It will be understood that the present disclosure does not impose any limitations on the image compression method or the specific network structure of the deblocking filter, and furthermore, the present disclosure does not impose any limitations on the block size, the method of dividing an image into blocks, or the deblocking order of artifact regions.

[0045] In some embodiments, as an example, in block-based image compression, the reconstructed blocks contain artifacts within the boundaries of the blocks.

number

number

number

number

[0046] The deblocking method as disclosed herein can be used to reduce interblock artifacts. Figure 6A shows an example workflow 6000 of one embodiment of a neural network-based deblocking module. For ease of illustration, Figure 6A shows only four equally sized blocks. It will be understood that an image may be divided into multiple blocks, and each block may have a different size.

[0047] A DNN-based deblocking model may be used to reduce interblock artifacts (within block boundary regions). In one embodiment, one or more entire reconstructed images may be sent to the deblocking model. By default, the entire boundary region may be used as input to the deblocking module and also as the output of the deblocking module. Samples in the output boundary region may be used to replace samples before the deblocking module to generate a reconstructed image with reduced artifacts.

[0048] According to one aspect of the present disclosure, a region containing artifacts induced by neighboring blocks may be extracted (referred to herein as a boundary region). As shown in FIGS. 6A and 5A, the boundary region may be composed of two subregions from two neighboring blocks, and the size of the boundary region may be 2m×n or n×2m, where n is the length of the block and m is the height of the region taken from the block. The value of m may vary as an integer. For example, it may be 4 pixels. The boundary region may be sent to a deblocking module to reduce the artifacts. The deblocked region in the image may then be updated, as shown in FIG. 6A. In FIG. 5A, the boundary region is identified from two neighboring blocks. In some embodiments, the boundary region may also be composed of multiple blocks, as shown in FIG. 5B.

[0049] According to one aspect of the present disclosure, a process for reducing interblock artifacts may include removing edge regions of the output of a deblocking model to reduce blocking effects caused by the deblocking model itself. As an example, as shown in FIG. 6B, process 6400 includes removing edge regions of the output of a deblocking model. As shown in process 6400, the outer portions of the boundary region (lighter gray regions) may be removed from applying to samples before the deblocking filter. In other words, the output size of the deblocking may be smaller than the input size, where the removed portions come from the outer regions (both sides) of the input boundary region. The size of the edge region removed from the boundary region may vary in different embodiments. In one embodiment, a fixed size (length) of the removed edge region may be used, such as four samples wide on each side. In some embodiments, the size (length) of the removed edge region may be specified in the bitstream in a high-level syntax, such as at the picture level or sequence level.

[0050] 6A-6B disclose an embodiment using a single deblocking model to improve boundary regions. FIG. 6C shows a process 6800 for interblock artifact reduction using multiple neural network-based deblocking modules. The process 6800 may include utilizing a specific deblocking model for a particular type of boundary region to more efficiently remove and / or reduce artifacts.

[0051] As shown in process 6800, once the boundary regions are extracted, a classification module (either a DNN-based model, or a traditional method used in video codecs, or simply variance) may be used to classify the boundary regions into different categories. The boundary regions may then be sent to different deblocking models according to their categories. In some embodiments, the extracted boundary regions may be defined based on one of (1) the content of the boundary region and (2) the strength of the artifacts. As an example, a boundary region with high-frequency content may be sent to a different model than a model designed to deblock low-frequency content. In some embodiments, the frequency of a region may be measured by the maximum difference of different samples within the region. In one example, the difference of samples near the edge on one side of the boundary may be measured, and in another example, the difference of samples near the edge on the other side of the boundary may also be measured.

[0052] It will be appreciated that the boundary regions in the multi-model deblocking module are not limited to those depicted in Figures 6A-6C, and the deblocking regions depicted in Figures 6A-6B may also use multi-model deblocking methods.

[0053] Each of processes 6000, 6400, and 6800 is an embodiment for removing artifacts in a block-wise NIC reconstructed image. The goal may be to deblock pixels closer to the boundary more than pixels further from the boundary. The deblocking model may include several convolutional or residual layers. CNN-based attention mechanisms (e.g., non-local attention, SEnet) or residual nets (including a set of convnets and activation functions) may also be used.

[0054] In some embodiments, the encoder and decoder may be able to select between traditional deblocking methods or the neural network-based deblocking methods disclosed herein. The selection decision may be made at various levels, such as the slice / picture level, for a group of pictures, or at the sequence level. The selection may also be made by sending a signaling flag or may be inferred from content analysis.

[0055] In some embodiments, the encoder and decoder may be able to apply various levels of boundary strength on top of the proposed method, assuming that the DNN-derived adjustment on a pixel is at a default level of boundary strength. By analyzing the boundary conditions and block coding characteristics, different levels of Bs can be assigned to modify (e.g., expand or contract) the default adjustment.

[0056] One or more deblocking models (e.g., deblocking DNNs) may be trained according to embodiments of the present disclosure. In one embodiment, the weighting factors of the deblocking DNN may be initialized, for example, by using a pre-trained corresponding DNN model or by setting them to random values. Then, given an input training image x, a reconstructed image x with block artifacts may be

number

[0057] In one embodiment, the input of the deblocking DNN may be a vertical region or a horizontal region. In other embodiments, the input of the deblocking DNN may be only a vertical region or only a horizontal region. In some embodiments, a transposition operation may be applied to the extracted boundary artifact region used as input to the deblocking DNN, and the output of the deblocking DNN may be transposed back to its original shape.

[0058] In one embodiment, the training data is a reconstructed image that may be generated by estimated compression by a NIC network.

number

number

number

number

[0059] FIG. 7 shows a flowchart of an exemplary process 700 for reducing artifacts in compressed images using a neural network-based deblocking filter, according to one embodiment.

[0060] At least one reconstructed image may be received in operation 705. In some embodiments, each reconstructed image includes one or more reconstruction blocks.

[0061] A boundary region associated with a boundary of one or more reconstructed blocks in at least one reconstructed image may be extracted at operation 710. In some embodiments, the extracted boundary region may include at least two sub-regions from at least two adjacent blocks of the one or more reconstructed blocks.

[0062] At operation 715, the extracted boundary regions may be input to a trained deblocking model to generate artifact-reduced boundary regions (also referred to as generated boundary regions with reduced artifacts or generated boundary regions with reduced artifacts in this disclosure). In some embodiments, inputting the extracted boundary regions to the trained deblocking model may include classifying each extracted boundary region of the extracted boundary regions into a category based on at least one of the content of the each extracted boundary region or the intensity of artifacts within the each extracted boundary region. In some embodiments, based on the classification, the each boundary region may be input to a respective trained deblocking model of the one or more trained deblocking models to generate an artifact-reduced boundary region. In some embodiments, the trained deblocking model is trained on training data based on estimated compression by a neural image compression (NIC) network.

[0063] In some embodiments, the trained deblocking model is a neural network-based model. In some embodiments, the trained deblocking model may be trained based on a quality loss function that measures reconstruction quality. The training data may include vertical and horizontal extracted boundary regions from at least one of reconstructed images generated using estimated compression by a first neural network-based image compression network, reconstructed images generated using actual compression by a second neural network-based image compression network, or reconstructed images generated by a third neural network-based image compression network and then rounded.

[0064] Edge regions associated with the artifact-reduced boundary region may be removed at operation 720. In some embodiments, removing the edge regions may include removing the edge regions based on a predefined size, the predefined size being signaled in the high-level syntax bitstream.

[0065] At operation 725, a reconstructed image in which at least one artifact has been reduced (also referred to in this disclosure as a reconstructed image with reduced artifacts) may be generated based on the artifact-reduction boundary region. In some embodiments, generating the reconstructed image in which at least one artifact has been reduced may include replacing a boundary region corresponding to the artifact-reduction boundary region in the at least one reconstructed image to generate the reconstructed image in which the at least one artifact has been reduced.

[0066] 8 illustrates the arrangement of a video encoder and a video decoder in a streaming environment as an example of an application of the disclosed subject matter 800. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital television, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0067] The streaming system may include a capture subsystem 813, which may include a video source 801, such as a digital camera, that creates an uncompressed video sample stream 802. The sample stream 802, depicted with a bold line to emphasize its high data volume compared to an encoded video bitstream, may be processed by an encoder 803 connected to the camera 801. The encoder 803 may include hardware, software, or a combination thereof for enabling or implementing aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream 804, depicted with a thin line to emphasize its smaller amount of data compared to the sample stream, may be stored on a streaming server 805 for future use. One or more streaming clients 806, 808 may access the streaming server 805 to obtain copies 807, 809 of the encoded video bitstream 804. The client 806 may include a video decoder 810, which decodes the incoming copy 807 of the encoded video bitstream and creates an outgoing video sample stream 811 that may be rendered on a display 812 or other rendering device. In some streaming systems, the video bitstreams 804, 807, 809 may be encoded according to several video coding / compression standards. Examples of such standards include H.265 HEVC. A video coding standard informally known as Versatile Video Coding (VVC) is under development. The disclosed subject matter may be used in the context of VVC.

[0068] Embodiments of the present disclosure provide a flexible and general framework that accommodates various types of quality metrics.

[0069] The techniques of the embodiments of the present disclosure described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 9 illustrates a computer system 900 suitable for implementing embodiments of the subject matter of the present disclosure.

[0070] Computer software may be coded using any suitable machine code or computer language that may be subjected to mechanisms such as assembly, compilation, linking, etc. to create code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly or via interpretation, microcode execution, etc.

[0071] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.

[0072] 9 for computer system 900 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 900.

[0073] The computer system 900 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0074] The input human interface devices may include one or more (only one of each shown) of a keyboard 901, a mouse 902, a trackpad 903, a touchscreen 910, a data glove, a joystick 905, a microphone 906, a scanner 907, and a camera 908.

[0075] The computer system 900 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen 910, data gloves, or joystick 905, although haptic feedback devices that do not function as input devices may also be present). For example, such devices may include audio output devices (such as speakers 909, headphones (not shown)), visual output devices (such as screens 910, including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capabilities, each with or without haptic feedback capabilities, some of which may output two-dimensional visual output or output in more than three dimensions through means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0076] The computer system 900 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 920 with media such as CD / DVD 921, thumb drives 922, removable hard drives or solid state drives 923, legacy magnetic media such as tape or floppy disks (not shown), specialized ROM / ASIC / PLD-based devices such as security dongles (not shown), etc.

[0077] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0078] The computer system 900 may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, terrestrial broadcast TV, and vehicular and industrial networks including CANBus, etc. Particular networks generally require an external network interface adapter connected to a particular general-purpose data port or peripheral bus 949 (e.g., a USB port on computer system 900); others are commonly integrated into the core of computer system 900 by connection to a system bus as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 900 can communicate with other entities. Such communications can be unidirectional, receive only (e.g., broadcast TV), unidirectional transmit only (e.g., a CANbus to a particular CANbus device), or bidirectional, e.g., to other computer systems using local or wide-area digital networks. Such communications can include communications to a cloud computing environment 955. Particular protocols and protocol stacks can be used with each of these networks and network interfaces, as described above.

[0079] The aforementioned human interface devices, human-accessible storage devices, and network interface 954 may be connected to core 940 of computer system 900 .

[0080] The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, specialized programmable processing units in the form of field programmable gate arrays (FPGAs) 943, hardware accelerators for specific tasks 944, etc. These devices may be connected via a system bus 948, along with read-only memory (ROM) 945, random access memory 946, and internal mass storage devices 947, such as a non-user-accessible internal hard drive or SSD. In some computer systems, the system bus 948 is accessible in the form of one or more physical plugs, allowing expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus 948 or via a peripheral bus 949. Peripheral bus architectures include PCI, USB, etc. A graphics adapter 950 may also be included in the core 940.

[0081] The CPU 941, GPU 942, FPGA 943, and accelerator 944 may combine to execute specific instructions that may constitute the aforementioned computer code. That computer code may be stored in ROM 945 or RAM 946. Transient data may also be stored in RAM 946, while permanent data may be stored in, for example, internal mass storage device 947. Cache memory, which may be closely associated with one or more of the CPU 941, GPU 942, mass storage device 947, ROM 945, RAM 946, etc., may be used to enable fast storage and retrieval to any memory device.

[0082] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0083] By way of example, and not limitation, computer system 900 having the architecture, and specifically core 940, may achieve functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage devices as introduced above, as well as specific storage devices of the core 940 that are non-transitory in nature, such as core internal mass storage device 947 or ROM 945. Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core 940.

[0084] The computer-readable medium may include one or more memory devices or chips, depending on particular needs. Software enables the core 940, and in particular the processor therein (including a CPU, GPU, FPGA, etc.), to perform the particular processes described herein, or particular portions of the particular processes, including the definition of data structures stored in RAM 946 and the modification of such data structures by software-defined processes. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 944) that can operate in place of or in conjunction with software to perform the particular processes described herein, or particular portions of the particular processes. References to software may encompass logic, where appropriate, and vice versa. References to computer-readable medium may encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0085] While this disclosure describes several non-limiting exemplary embodiments, there are alterations, substitutions, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0086] 100 Communication Systems 110 Terminal 120 terminals 130 terminals 140 terminals 150 Network 200 devices 210 Bus 220 processors 230 memory 240 Storage Components 250 Input Components 260 Output Components 270 Communication Interface 801 Video Source 802 sample stream 803 Encoder 804 Video Bitstream 805 Streaming Server 806 Streaming Client 807 encoded video bitstream copy 808 Streaming Client 809 encoded video bitstream copy 810 Video Decoder 811 video sample stream 812 display 813 Capture Subsystem 900 Computer Systems 901 Keyboard 902 Mouse 903 Trackpad 905 Joystick 906 Microphone 907 Scanner 908 Camera 909 Speaker 910 Touchscreen 920 CD / DVD ROM / RW 921 CD / DVD and other media 922 thumb drive 923 Removable Hard Drive or Solid State Drive 940 cores 941 Central Processing Unit (CPU) 942 Graphics Processing Unit (GPU) 943 Field Programmable Gate Area (FPGA) 944 Hardware Accelerator 945 Read-Only Memory (ROM) 946 Random Access Memory (RAM) 947 Internal mass storage 948 System Bus 949 Peripheral Bus 954 network interface 955 Cloud Computing Environment

Claims

1. 1. A method for reducing artifacts in a compressed image using a neural network based deblocking filter, executed by at least one processor, the method comprising: receiving at least one reconstructed image, each reconstructed image including one or more reconstruction blocks; extracting a boundary region associated with a boundary of the one or more reconstructed blocks in the at least one reconstructed image; inputting the extracted boundary regions into a trained deblocking model to generate artifact-reducing boundary regions, the trained deblocking model being trained on training data generated by a Neural Image Compression (NIC) network, and inputting the extracted boundary regions into the trained deblocking model includes: classifying each of the extracted boundary regions into a category based on the frequency of content within the each extracted boundary region as measured by the maximum difference between different samples near one edge of the extracted boundary region; inputting each of the extracted boundary regions into a respective trained deblocking model of one or more trained deblocking models to generate an artifact-reduced boundary region based on the classification; inputting the extracted boundary regions into the trained deblocking model, removing edge regions associated with the generated boundary regions; and generating at least one reconstructed image having reduced artifacts based on the generated boundary region.

2. The method of claim 1 , wherein the extracted boundary region includes at least two sub-regions from at least two adjacent ones of the one or more reconstructed blocks.

3. 2. The method of claim 1, wherein removing the edge regions comprises removing the edge regions based on a predefined size, the predefined size being signaled in a high-level syntax bitstream.

4. The method of claim 1 , wherein removing the edge regions comprises removing edge regions from the generated boundary regions output by the deblocking model.

5. 2. The method of claim 1, wherein the trained deblocking model is trained based on a quality loss function that measures reconstruction quality, and the training data includes vertical and horizontal extracted boundary regions from images generated by the NIC network.

6. 2. The method of claim 1, wherein generating the at least one reconstructed image with reduced artifacts comprises replacing a boundary region corresponding to the generated boundary region in the at least one reconstructed image to generate the at least one reconstructed image with reduced artifacts.

7. at least one memory configured to store computer program code; at least one processor configured to access said computer program code and to operate as instructed by said computer program code; Including, 7. A system, wherein the computer program code is configured to cause the at least one processor to perform the method of any one of claims 1 to 6.

8. A computer program comprising computer code configured, when executed by at least one processor, to cause said at least one processor to perform the method of any one of claims 1 to 6.

Images