Method and related device for block-by-block neural image compression

Abstract

A method for block-by-block neural image compression includes: encoding blocks of an input image using a first neural network, wherein the encoded blocks are decoded by a decoder using a second neural network to generate reconstructed blocks; and performing intra prediction on the reconstructed blocks using a third neural network to generate predicted blocks. The method further includes: determining a difference between a block of the input image and the generated predicted blocks to generate a prediction residual; encoding the generated prediction residual using a fourth neural network, wherein the encoded prediction residual is decoded by a decoder using a fifth neural network; and adding the decoded prediction residual to the generated predicted blocks to generate restored predicted blocks.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 085,900, filed on September 30, 2020, U.S. Provisional Patent Application No. 63 / 085,908, filed on September 30, 2020, U.S. Provisional Patent Application No. 63 / 088,675, filed on October 7, 2020, and U.S. Patent Application No. 17 / 362,003, filed on June 29, 2021, all of which are hereby incorporated by reference in their entireties. Technical field

[0003] The present invention relates to the field of image compression technology, and in particular to methods and related devices for block - by - block neural image compression. Background art

[0004] Generally, instead of directly encoding the entire original image, block - based intra - prediction and residual coding mechanisms have proven to be very effective for compressing image frames in modern video coding systems such as in the High Efficiency Video Coding (HEVC) standard and the Versatile Video Coding (VVC) standard. First, the entire image is divided into Coding Tree Units (CTUs). For each CTU, the CTU can be divided into blocks of various sizes, and a prediction block is generated by replicating the boundary pixels of the previously compressed blocks along various angular directions. Then, the residual between the original block and the prediction block is compressed. The residual can be more effectively encoded compared to the original pixels, and better coding performance can be achieved.

[0005] ISO (International Organization for Standardization) / IEC (International Electrotechnical Commission) MPEG (Moving Picture Experts Group) (JTC 1 / SC 29 / WG 11) has been actively seeking potential requirements for the standardization of future video coding technologies. ISO / IEC JPEG (Joint Photographic Experts Group) has established the JPEG-AI (Joint Photographic Experts Group-Artificial Intelligence) group, focusing on AI-based end-to-end neural image compression using deep neural networks (DNNs). The Chinese AVS (Audio Video Coding Standard) has also established the AVS-AI (Audio Video Coding Standard-Artificial Intelligence) special group, which is engaged in neural image and video compression technologies. The success of the latest methods has led to an increasing industrial interest in advanced neural image and video compression methods.

[0006] Given an input image x, the goal of neural image compression (NIC) is to use the image x as the input to a DNN encoder to compute a compressed representation that is beneficial for storage and transmission. Then use as the input to a DNN decoder to reconstruct the image. Previous NIC methods have adopted a variational autoencoder (VAE) structure, in which the DNN encoder directly uses the entire image x as the input to the DNN encoder, and the entire image x passes through a set of network layers that work like a black box to compute the output representation. Accordingly, the DNN decoder takes the entire representation as the input to the DNN decoder, and the entire representation passes through another set of network layers that work like another black box to compute the reconstructed

[0007] In HEVC, deblocking filtering is performed on each coding unit (CU) in the same order as the decoding process. First, vertical edges are filtered (horizontal filtering), and then horizontal edges are filtered (vertical filtering). The filtering is applied to 8×8 block boundaries determined for both the luminance component and the chrominance component. To reduce complexity, 4×4 block boundaries are not processed.

[0008] Boundary Strength (Bs) reflects how strongly the boundary may need to be filtered. A Bs value of 2 indicates strong filtering, 1 indicates weak filtering, and 0 indicates no deblocking filtering.

[0009] P and Q are the blocks participating in the filtering, where P represents the block on the left side (in the case of vertical edges) or above (in the case of horizontal edges) of the boundary, and Q represents the block on the right side (in the case of vertical edges) or below (in the case of horizontal edges) of the boundary. Figure 1 Shows how to calculate the Bs value based on the intra coding mode, the presence of non-zero transform coefficients, reference pictures, the number of motion vectors, and the motion vector difference. For example, when the MV difference is below the threshold T, Bs is equal to 0. The threshold T is set to 1 pixel. In HEVC, the MV precision is 1 / 4 pixel, and the MV difference threshold is set to 4. In VTM (Versatile Video Coding and Test Model, VTM), the MV precision is 1 / 16, and the MV difference is set to 16.

[0010] Bs is calculated on a 4×4 block basis but remapped to an 8×8 grid. The maximum value of the two Bs values corresponding to 8 pixels formed by the lines in the 4×4 grid is selected as the Bs for the boundary in the 8×8 grid.

[0011] In VTM5, the deblocking filtering process is mostly the same as that in HEVC. However, the following modifications are added.

[0012] 1) The filtering strength of the deblocking filter depending on the average luminance level of the reconstructed samples

[0013] 2) Deblocking tC table extension

[0014] 3) A stronger deblocking filter for luminance

[0015] 4) A stronger deblocking filter for chrominance

[0016] 5) Luminance deblocking for the 4×4 sample grid and chrominance deblocking for the 8×8 sample grid

[0017] In HEVC, the filtering strength of the deblocking filter is controlled by variables β and tC derived from the average quantization parameter qPL. In VTM5, the deblocking filter controls the strength of deblocking filtering by adding an offset to qPL according to the luminance level of the reconstructed samples. The reconstructed luminance level LL is derived as follows:

[0018] LL = ((p0,0 + p0,3 + q0,0 + q0,3) >> 2) / (1 << bit depth) (Equation 1),

[0019] where the sample values pi,k and qi,k for i = 0...3 and k = 0 and 3 are derived as Figure 2 shown.

[0020] The variable qPL is derived as follows:

[0021] qPL = ((QpQ + QpP + 1) >> 1) + qpOffset (Equation 2),

[0022] where QpQ and QpP represent the quantization parameters of the coding units containing samples q0,0 and p0,0 respectively. The offset qpOffset depends on the transfer function and these values are signaled in the SPS (Sequence Parameter Set).

[0023] In VTM5, the maximum QP is changed from 51 to 63, and it is desired to reflect the corresponding change in the deblocking table, which derives the value of the deblocking parameter tC based on the block QP. The following is the updated tC table adapted to the extended QP range.

[0024] tC = [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,5,5,6,6,7,8,9,10,11,13,14,16,18,20,22,25,28,31,35,39,44,50,56,63,70,79,88,99] (Equation 3).

[0025] When the samples on either side of the boundary belong to a large block, a bilinear filter (stronger deblocking filter) is used. Samples belonging to a large block are defined as follows: for a vertical edge, the width is greater than or equal to 32; and for a horizontal edge, the height is greater than or equal to 32. Then, the block boundary samples pi for i = 0 to Sp - 1 and the block boundary samples qi for j = 0 to Sq - 1 are linearly interpolated and replaced as follows:

[0026] p i ′ = (f i *Middles，t +(64 - f i ) * P s + 32) >> 6), clipped to p i ±tcP i (Equation 4); and

[0027] q j ′ = (gj * Middle s，t +(64 - g j ) * Q s + 32) >> 6), clipped to q j ±tcPD j (Equation 5),

[0028] where tcPD i and tcPD j terms are position - dependent clipping, and g j , f i , Middle s，t , P s and Q s are given below:

[0029] Table 1 - Derivation of Stronger Deblocking Parameters for Luminance

[0030]

[0031]

[0032]

[0033] The stronger luminance filter mentioned above is used only when all of the conditions in Condition 1, Condition 2, and Condition 3 are true. Condition 1 is the "large block condition". This condition detects whether the samples on the P - side and the samples on the Q - side belong to a large block. Condition 2 and Condition 3 are determined by the following equations:

[0034] Condition 2 = (d < β)? true : false (Equation 6); and

[0035] Condition 3 = strong filter condition = (dpq less than (β >> 2), sp3 + sq3 less than (3 * β >> 5), and Abs(p0 - q0) less than (5 * t C + 1) >> 1)? true : false (Equation 7).

[0036] The following strong chrominance deblocking filter is defined:

[0037] p2′ = (3 * p3 + 2 * p2 + p1 + p0 + q0 + 4) >> 3 (Equation 8)

[0038] p1' = (2 * p3 + p2 + 2 * p1 + p0 + q0 + q1 + 4) >> 3 (Equation 9); and

[0039] p0' = (p3 + p2 + p1 + 2 * p0 + q0 + q1 + q2 + 4) >> 3 (Equation 10).

[0040] The above chroma filter performs deblocking on an 8×8 chroma sample grid. A strong chroma filter is used on both sides of the block boundary. Here, this chroma filter is selected when both sides of the chroma edge are greater than or equal to 8 (in terms of chroma samples), and then a determination is made under three conditions. The first determination is for the boundary strength and large blocks. The second and third determinations are basically the same as those for HEVC luma, which are the on / off determination and the strong filter determination respectively. In the first determination, bS is modified for chroma filtering as shown in Table 2. The conditions in Table 1 are checked in sequence. If a certain condition is met, the remaining conditions with lower priority are skipped.

[0041] Table 2 – Modified boundary strength

[0042]

[0043]

[0044] When bS is equal to 2 or bS is equal to 1, chroma deblocking is performed when a large block boundary is detected. The second and third conditions are basically the same as those for the HEVC luma strong filter determination.

[0045] In the current VVC working draft, the deblocking filter for luma is enabled for a 4×4 grid, and the deblocking filter for chroma is enabled for an 8×8 grid. The deblocking filter process is applied to CU boundaries and sub-block boundaries. Sub-block boundaries include prediction unit boundaries introduced by STMVP (Spatial-Temporal Motion Vector Prediction) and affine modes, and transform unit boundaries introduced by SBT (Sub-block Transform) and ISP (Intra Sub-Partition) modes.

[0046] For SBT sub-blocks and ISP sub-blocks, the same logic in the TU of the HEVC deblocking filter is applied. When there are non-zero coefficients in any sub-block across the edge, the deblocking filter is applied to the TU boundary.

[0047] For SbTMVP (Subblock-based Temporal Motion Vector Prediction) sub-blocks and affine sub-blocks on a 4×4 grid, the same logic in the PU of the HEVC deblocking filter is applied. For the PU boundary, the deblocking filter is applied considering the differences between the motion vectors of adjacent sub-blocks and the reference pictures.

[0048] It may be necessary to develop more efficient NIC technologies that can utilize the existing network architecture and achieve higher coding efficiency.

[0049] Intra prediction is a key mechanism widely used in image and video compression algorithms such as HEVC and VVC. The goal of prediction is to infer a pixel block based on previously reconstructed surrounding pixels. By encoding / decoding the residuals between the calculated original pixels and the predicted pixels, the compression method can be more efficient.

[0050] Intra prediction in traditional video codecs (HEVC, VVC) relies on the correlation between the block to be predicted and its neighborhood. For example, in VVC, according to the rate-distortion criterion, the prediction method selects one pattern from 35 fixed and simple prediction functions and propagates pixel values along a specified direction. Instead of simply propagating pixels through the causal relationship with surrounding pixels, previous proposals introduced a search method for finding the best match for the block to be predicted within the image. Further work exploited the self-similarity within the image that is defined as a linear combination of the k nearest patches in the neighborhood.

[0051] In recent years, recurrent neural networks have demonstrated the ability to learn reliable models for predicting image pixels. For example, a recurrent neural network sequentially updates its internal representation of the correlation between pixels in the known region of an image and then generates the next pixel in the unknown region of that image.

[0052] In addition, a fully connected (FC) layer-based method for intra prediction has been proposed. An FC neural network is capable of learning an end-to-end mapping from adjacent reconstructed pixels to the current block. The FC neural network takes multiple reference lines of pixels as the context region (input to the model). However, this method can only predict a very small number of pixels. In some previous work, a convolutional layer-based method for intra prediction was proposed. The convolutional layer-based method for intra prediction obtains adjacent reconstructed pixels and sends them to a convolutional neural network (CNN) for prediction.

[0053] The current prediction method is designed for the intra-frame prediction module of a video and cannot be directly applied to neural-based image compression. Additionally, the current neural-based prediction method divides the reconstructed pixels (context regions) into two separate regions, which may lead to the following drawbacks: 1) The current neural-based prediction method cannot learn the boundary region between the two separate context regions; 2) To combine the two context regions (or the latent representations transformed by the two context regions), the prediction method may require a large amount of computational resources and computational time. SUMMARY OF THE INVENTION

[0054] According to some embodiments, a method for block-based neural image compression is performed by at least one processor of an encoder, and the method includes: encoding a block of an input image using a first neural network, wherein the encoded block is decoded by a decoder using a second neural network to generate a reconstructed block; and performing intra-frame prediction on the reconstructed block using a third neural network to generate a predicted block. The method further includes: determining a difference between the block of the input image and the generated predicted block to generate a prediction residual; encoding the generated prediction residual using a fourth neural network, wherein the encoded prediction residual is decoded by the decoder using a fifth neural network; and adding the decoded prediction residual to the generated predicted block to generate a restored predicted block.

[0055] According to some embodiments, an encoder includes: a first encoding module configured to encode a block of an input image using a first neural network, wherein the encoded block is decoded by a decoder using a second neural network to generate a reconstructed block; an intra-frame prediction module configured to perform intra-frame prediction on the reconstructed block using a third neural network to generate a predicted block; a prediction residual generation module configured to determine a difference between the block of the input image and the generated predicted block to generate a prediction residual; a second encoding module configured to encode the generated prediction residual using a fourth neural network, wherein the encoded prediction residual is decoded by the decoder using a fifth neural network; and an addition module configured to add the decoded prediction residual to the generated predicted block to generate a restored predicted block.

[0056] According to some embodiments, an apparatus for block-based neural image compression is implemented as an encoder, and the apparatus includes: at least one memory configured to store program code; and at least one processor configured to read the program code and execute the program code to implement the above method.

[0057] According to some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by at least one processor of an encoder, implement the above method.

[0058] In summary, in the present invention, each neural network is used to encode, reconstruct, intra-predict, encode, and decode prediction residuals for image blocks. Compared with previous prediction methods, it has the following advantages. It can utilize existing network architectures and use prediction mechanisms to improve the encoding efficiency of NIC. The flexible and general framework adapts to different intra-prediction methods and different neural encoding methods for residuals in intra-prediction processing, and adapts to various types of quality metrics. Moreover, for the padding method and the transformation method, there is no need to divide the context area. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flowchart for calculating the boundary strength (Bs).

[0060] Figure 2 is a diagram of sample positions.

[0061] Figure 3 is a diagram of an environment in which the methods, apparatuses, and systems described herein can be implemented according to some embodiments.

[0062] Figure 4 is Figure 3 a block diagram of example components of one or more devices of.

[0063] Figure 5A is a block diagram of an encoder for block-by-block neural image compression according to some embodiments.

[0064] Figure 5B is according to some embodiments Figure 5A a block diagram of the intra-prediction component of the encoder of.

[0065] Figure 6 is a block diagram of a decoder for block-by-block neural image compression according to some embodiments.

[0066] Figure 7 is a block diagram of a post-processing component according to some embodiments.

[0067] Figure 8 is Figure 7 a diagram of the deblocking component of the post-processing component of.

[0068] Figure 9 is a diagram showing a method of extracting a boundary region according to some embodiments.

[0069] Figure 10 is a diagram of a multi-model deblocking component according to some embodiments.

[0070] Figure 11 is Figure 7 a diagram of the post-enhancement component of the post-processing component of.

[0071] Figure 12Diagram of a multi-model post-enhancement component according to some embodiments.

[0072] Figure 13 Diagram of a post-enhancement component according to some embodiments.

[0073] Figure 14 Diagram of overlapping regions of a post-enhancement component and a deblocking component according to some embodiments.

[0074] Figure 15 Block diagram of a device for DNN encoding and DNN prediction residual encoding according to some embodiments.

[0075] Figure 16 Block diagram of a device for DNN decoding and DNN prediction residual decoding according to some embodiments.

[0076] Figure 17A Diagram of a predicted block and reconstructed pixels.

[0077] Figure 17B Diagram showing a prediction method for two context regions according to some embodiments.

[0078] Figure 17C Diagram showing a fill prediction method according to some embodiments.

[0079] Figure 17D Diagram showing a transform prediction method according to some embodiments.

[0080] Figure 18 Flowchart of a method for per-block neural image compression according to some embodiments.

[0081] Figure 19 Block diagram of a device for per-block neural image compression according to some embodiments. Detailed embodiments

[0082] The present disclosure describes a neural image compression (NIC) framework for compressing an input image using a residual from block-based intra prediction via a DNN.

[0083] Figure 3 Diagram of an environment 300 in which the methods, devices, and systems described herein can be implemented according to some embodiments.

[0084] As Figure 3 shown, the environment 300 can include a user device 310, a platform 320, and a network 330. Devices in the environment 300 can be interconnected via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection.

[0085] User device 310 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with platform 320. For example, user device 310 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch), or a similar device. In some implementations, user device 310 may receive information from platform 320 and / or send information to platform 320.

[0086] Platform 320 includes one or more devices as described elsewhere herein. In some implementations, platform 320 may include a cloud server or a group of cloud servers. In some implementations, platform 320 may be designed to be modular such that software components can be swapped in or out. As such, platform 320 can be easily and / or quickly reconfigured for different uses.

[0087] In some implementations, as shown, platform 320 may be hosted in a cloud computing environment 322. It should be noted that while the implementations described herein describe platform 320 as being hosted in cloud computing environment 322, in some implementations, platform 320 may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0088] Cloud computing environment 322 includes an environment that hosts platform 320. Cloud computing environment 322 may provide services such as computing, software, data access, storage, etc., which do not require an end user (e.g., user device 310) to know the physical location and configuration of the systems and / or devices of hosted platform 320. As shown, cloud computing environment 322 may include a set of computing resources 324 (collectively referred to as "computing resources 324" and individually referred to as "computing resource 324").

[0089] Computing resources 324 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, computing resources 324 may host platform 320. Cloud resources may include: computing instances executed in computing resources 324, storage devices provided in computing resources 324, data transfer devices provided by computing resources 324, etc. In some implementations, computing resources 324 may communicate with other computing resources 324 via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection.

[0090] As further described in Figure 3As shown, computing resources 324 include a set of cloud resources, such as one or more applications ("Application, APP") 324-1, one or more virtual machines ("Virtual Machine, VM") 324-2, virtualized storage devices ("Virtualized Storage, VS") 324-3, one or more hypervisors ("Hypervisor, HYP") 324-4, and so on.

[0091] Application 324-1 includes one or more software applications that can be provided to and / or accessed by user device 310 and / or platform 320. Application 324-1 can eliminate the need to install and execute software applications on user device 310. For example, application 324-1 can include software associated with platform 320 and / or any other software that can be provided via cloud computing environment 322. In some implementations, one application 324-1 can send information to / receive information from one or more other applications 324-1 via virtual machine 324-2.

[0092] Virtual machine 324-2 includes a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. Virtual machine 324-2 can be a system virtual machine or a process virtual machine, depending on the degree of use and correspondence of virtual machine 324-2 to any real machine. A system virtual machine can provide a complete system platform that supports the execution of a full operating system ("OS"). A process virtual machine can execute a single program and can support a single process. In some implementations, virtual machine 324-2 can execute on behalf of a user (e.g., user device 310) and can manage the infrastructure of cloud computing environment 322, such as data management, synchronization, or long-duration data transfer.

[0093] Virtualized storage device 324-3 includes one or more storage systems and / or one or more devices that use virtualization technology within the storage system or device of computing resources 324. In some implementations, within the context of a storage system, the types of virtualization can include block virtualization and file virtualization. Block virtualization can refer to the abstraction (or separation) of logical storage from physical storage, such that the storage system can be accessed without regard to physical storage or heterogeneous architectures. The separation can allow the administrator of the storage system flexibility in how the administrator manages storage for end users. File virtualization can eliminate the dependency between data accessed at the file level and the location where the file is physically stored. This can enable optimization of storage usage, server consolidation, and / or the performance of non-disruptive file migration.

[0094] The hypervisor 324-4 can provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to execute simultaneously on a host computer such as the computing resource 324. The hypervisor 324-4 can present a virtual operating platform to the guest operating systems and can manage the execution of the guest operating systems. Multiple instances of various operating systems can share the virtualized hardware resources.

[0095] The network 330 includes one or more wired networks and / or wireless networks. For example, the network 330 can include a cellular network (e.g., a Fifth Generation (5G) network, a Long-Term Evolution (LTE) network, a Third Generation (3G) network, a Code Division Multiple Access (CDMA) network, etc.), a Public Land Mobile Network (PLMN), a Local Area Network (LAN), a Wide Area Network (WAN), a Metropolitan Area Network (MAN), a telephone network (e.g., a Public Switched Telephone Network (PSTN)), a private network, an ad-hoc network, an intranet, the Internet, a fiber-based network, etc. and / or a combination of these or other types of networks.

[0096] Figure 3 The number and arrangement of the devices and networks shown are provided as an example. In fact, compared with Figure 3 the devices and / or networks shown, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks. In addition, Figure 3 two or more of the devices shown can be implemented within a single device, or Figure 3 the single device shown can be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) in the environment 300 can perform one or more functions described as being performed by another set of devices in the environment 300.

[0097] Figure 4 is Figure 3 a block diagram of example components of one or more of the devices.

[0098] The device 400 can correspond to the user device 310 and / or the platform 320. As Figure 4As shown, device 400 may include bus 410, processor 420, memory 430, storage component 440, input component 450, output component 460, and communication interface 470.

[0099] Bus 410 includes components that permit communication among the components of device 400. Processor 420 is implemented in hardware, firmware, or a combination of hardware and software. Processor 420 is a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, processor 420 includes one or more processors that can be programmed to perform functions. Memory 430 includes random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device that stores information and / or instructions for use by processor 420 (e.g., flash memory, magnetic memory, and / or optical memory).

[0100] Storage component 440 stores information and / or software related to the operation and use of device 400. For example, storage component 440 may include a hard disk (e.g., a magnetic disk, optical disk, magneto-optical disk, and / or solid state disk), compact disc (CD), digital versatile disc (DVD), floppy disk, cassette tape, magnetic tape, and / or another type of non-transitory computer-readable medium, as well as a corresponding drive.

[0101] The input component 450 includes components that allow the device 400 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, the input component 450 may include sensors for sensing information (e.g., a Global Positioning System (GPS) component, an accelerometer, a gyroscope, and / or an actuator). The output component 460 includes components that provide output information from the device 400 (e.g., a display, a speaker, and / or one or more Light-Emitting Diodes (LEDs)).

[0102] The communication interface 470 includes transceiver-like components that enable the device 400 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection (e.g., a transceiver and / or separate receiver and transmitter). The communication interface 470 may allow the device 400 to receive information from another device and / or provide information to another device. For example, the communication interface 470 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a Radio Frequency (RF) interface, a Universal Serial Bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0103] The device 400 may perform one or more of the processes described herein. The device 400 may perform these processes in response to the processor 420 executing software instructions stored by a non-transitory computer-readable medium, such as the memory 430 and / or the storage component 440. The computer-readable medium is defined herein as a non-transitory memory device. The memory device includes storage space within a single physical storage device or storage space distributed across multiple physical storage devices.

[0104] The software instructions may be read into the memory 430 and / or the storage component 440 from another computer-readable medium or from another device via the communication interface 470. The software instructions stored in the memory 430 and / or the storage component 440, when executed, may cause the processor 420 to perform one or more of the processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with the software instructions to perform one or more of the processes described herein. Accordingly, the implementations described herein are not limited to any particular combination of hardware circuitry and software.

[0105] Figure 4 The number and arrangement of the components shown are provided as an example. In fact, compared with Figure 4Compared with the components shown, device 400 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a set of components (e.g., one or more components) of device 400 may perform one or more functions described as being performed by another set of components of device 400.

[0106] Methods and apparatuses for block-by-block neural image compression will now be described in detail.

[0107] This disclosure describes a block-based intra prediction and residual coding framework for NIC. Traditional prediction mechanisms are modified into NIC to significantly improve NIC coding efficiency. Instead of encoding the original pixels, the residual between the predicted block and the original block is encoded.

[0108] Figure 5A and Figure 6 Workflows of embodiments of an encoder 500 and a decoder 600 for NIC are described respectively. On the encoder side, given an input image x, the image x is first divided into n blocks {b0, b1, …, bn}. For block bi, it is compressed by two compression methods (i.e., a prediction-based compression method and a conventional DNN-based compression method), and the reconstructed block with less loss is selected to the decoder side. For the conventional DNN-based compression method, DNN encoder 510 directly encodes bi to generate which can be decoded by DNN decoder 520 in the future to generate a reconstructed block For the prediction-based compression method, represents a previously reconstructed block, and the previously reconstructed block is passed through a DNN of the intra prediction component 530 (“prediction DNN”) to calculate the predicted block b′ i . Adder 540 can calculate the prediction residual based on the difference between b i and b′ i DNN prediction residual encoder 550 encodes the residual to generate another encoded compact residual Then, the encoded residual is passed through DNN prediction residual decoder 560 to calculate the decoded prediction residual The decoded prediction residual is added back to b′ by adder 570 i to calculate a new recovered predicted block Block selection component 580 selects the reconstructed block (according to the two methods mentioned above) with less loss (compared with b i ). The selection signal s is signaledi Notify the decoder 600. Note that for block b i , or only one of (including useful information for reconstructing b i ) is sent to the decoder through the bitstream. For ease of representation, for block b i , the compact useful information in the bitstream that is conducive to storage and transmission is represented as u i .

[0109] The DNN of the intra prediction component 530 may include several convolutional layers and deconvolutional layers. To support different numbers of reconstructed blocks as context regions, fully connected layers may also be used in the DNN of the intra prediction component 530. For example, the DNN used for image super-resolution (e.g., by changing the output size to be the same as the input size) may be used here.

[0110] As Figure 6 described in, the decoder 600 reconstructs the image according to the block-level information in the bitstream including the block selection signal, the encoded block residual or the encoded block . The block selection signal indicates the method used to decode the current block. If the block uses a conventional DNN-based compression method, this method will be sent to the DNN decoder 520 to generate the corresponding reconstructed block and then this reconstructed block is added to a set of previous reconstructed blocks. If the block uses a prediction-based compression method, given a set of previous reconstructed blocks the DNN of the intra prediction component 530 (the same as on the encoder side) is used to calculate the predicted current block b' i , and the predicted current block b' i is added back to the decoded residual block through the adder 570 to generate the reconstructed block and then this reconstructed block is added to the said set of previous reconstructed blocks In some embodiments, the intra prediction component 530, the DNN decoder 520, and the DNN prediction residual decoder 560 on the decoder side use the same block-by-block correspondence mechanism as on the encoder side. The decoder 600 also has a deblocking DNN and a post-enhancement DNN. The reconstructed blocks form this which may contain artifacts between blocks. Then, is sent to the deblocking DNN, and the deblocking DNN is used to minimize such artifacts. For future loss reduction, the image is sent to the post-enhancement DNN to generate the final reconstructed The deblocking DNN and the post-enhancement DNN may include several convolutional layers. For example, the DNN used for image super-resolution may be used herein (e.g., by changing the output size to be the same as the input size).

[0111] In some embodiments, the DNN encoder 510 encodes the block b using the DNN encoder. i The DNN prediction residual encoder 550 encodes the residual r_b using the prediction residual encoding DNN. i The DNN decoder 520 decodes the encoded block The DNN prediction residual decoder 560 decodes the encoded residual using the prediction residual decoding DNN The DNN encoder and the DNN decoder adopt the VAE structure, and the prediction residual encoding DNN and the prediction residual decoding DNN also adopt the VAE structure. The structures of the encoder / decoder of the DNN and the prediction residual encoding / decoding DNN may be different or the same.

[0112] The present disclosure does not impose any restrictions on the specific network structures of the DNN for the intra prediction component 530, the encoder / decoder of the DNN and the prediction residual encoding / decoding DNN, the deblocking DNN, and the post-enhancement DNN. The present disclosure does not impose any restrictions on the block size, how to divide the image into blocks, and the scanning order of the blocks.

[0113] The following are some embodiments for implementing the encoder / decoder architecture of the prediction-based DNN mentioned above.

[0114] In some embodiments, block partitioning in conventional video coding systems such as HEVC and VVC may be adopted. The CTU (Coding Tree Unit) for the square array samples is considered the basic processing unit. The picture can be divided into a series of CTUs. Potential further partitioning is possible to enable the use of smaller blocks under the CTU. The size of the CTU can be 32×32, 64×64, 128×128, etc. inside the CTU. The blocks for implementing the encoder / decoder of the prediction-based DNN mentioned above can be the coding blocks or the prediction blocks inside the CTU.

[0115] In another embodiment, in addition to adopting those methods in the conventional video coding system, a simplified block partitioning method may also be used. For example, the picture can be divided into a series of M×N blocks. For each block, the block can be further divided into 4 smaller blocks using quadtree partitioning. Whether to split the block can be signaled in the bitstream or inferred by a learning-based method.

[0116] In some embodiments, the above-mentioned prediction-based method is used on the entire image (and the entire image is divided into blocks). In another embodiment, the image can be first segmented into several tiles, and for each tile, the tile is further divided into blocks, and image compression is performed in parallel at the tile level.

[0117] In some embodiments, the scan start block can be located at the upper left corner, upper right corner, lower left corner, and lower right corner. The scan order can be height-first and width-first.

[0118] In some embodiments, the prediction DNN takes all the reconstructed blocks as input to predict a block. In another embodiment, the prediction DNN uses partial reconstructed blocks (e.g., blocks adjacent to the block to be predicted) to predict a block.

[0119] In some embodiments, the prediction DNN includes three main components: 1) a set of multiple convolutional layers for transforming the reconstructed blocks into a latent space; 2) a fully connected layer, or a concatenation operation, or a spatial pyramid pooling layer for combining the latent spaces from different reconstructed blocks; and 3) a deconvolution network for generating the prediction of the target block from the combined latent space.

[0120] In some embodiments, the intra-frame prediction component 530 includes three components, as Figure 5B shown. First, the reconstructed blocks are grouped into one or more groups according to the positions of the reconstructed blocks. Note that Figure 5B three groups are shown as an example. In some embodiments, the number of groups ranges from 1 (i.e., all reconstructed blocks are in one group) to the number of blocks (i.e., each reconstructed block is in one group). In the transformation component 532, for each group, the transformation 532A, 532B, or 532N transforms the blocks into a latent space. The transformation component 532 can include several convolutional layers. The transformation component 532 can also include an attention mechanism capable of emphasizing the most important positions in the reconstructed blocks for the purpose of making a correct prediction. The combination component 534 combines the latent spaces together. Any operation or DNN layer with the ability to combine the latent spaces can be used here. For example, the any operation or DNN layer can be a fully connected layer, or a concatenation operation, or a spatial pyramid pooling layer. In the prediction component 536, the prediction module predicts a block according to the combined latent space generated by the combination component 534 Generally, the prediction module contains a set of deconvolutional layers. The prediction module can also include an attention mechanism capable of emphasizing the most important positions in the reconstructed blocks for the purpose of making a correct prediction. The transformation component 532, the combination component 534, and the prediction component 536 are DNNs learned during the training process.

[0121] Above, some methods for encoding and decoding pictures by using DNN were described. After sample reconstruction, the reconstructed blocks form The may contain artifacts between block boundaries. Then, is sent to a deblocking DNN, and the deblocking DNN is used to minimize such artifacts and generate To improve the quality of the decoded image, the image is sent to a post-enhancement DNN to generate a final reconstructed Figure 7 The workflow of an embodiment of the post-processing component 700 is shown. Details of embodiments of the deblocking component 710 and the post-enhancement component 720 will be provided separately in the following sections.

[0122] Regarding the deblocking component 710 and the post-enhancement component 720, the processing order can be swapped. That is, the post-enhancement component 720 can be used first, and then the deblocking component 710 can be used.

[0123] One method (or combination) of the following deblocking methods can be used to further reduce the artifacts between blocks. Figure 8 The workflow of an embodiment of the deblocking component 710 is described. For ease of description, Figure 8 there are only four blocks of equal size. Note that in general, an image can be divided into a large number of blocks, and the size of each block can be different.

[0124] To reduce the artifacts between blocks (in the block boundary region), a DNN-based deblocking model can be used. First, a region containing artifacts caused by adjacent blocks can be taken out from the block 810 (such a region is hereinafter referred to as the boundary region 820). As Figure 8 shown, the boundary region includes two sub-regions from two adjacent blocks, and the size of the boundary region is 2m×n or n×2m, where n is the length of the block and m is the height of the region taken out from the block. The value of m can vary as an integer. For example, m can be 4 pixels. The boundary region is sent to the deblocking component 710 to reduce the artifacts and generate a deblocked region 830. Then, the deblocked region is updated 840 in the image, as Figure 8 shown.

[0125] In Figure 8 case, the boundary region is identified from two adjacent blocks. However, as an example, as Figure 9 shown, the boundary region 910 can also include multiple blocks.

[0126] Figure 8 A method of using a single deblocking model to improve the boundary region is described. Figure 10Describes the workflow of the implementation of multi-model deblocking components 1040A, 1040B, and 1040N. The key concept is that for different types of boundary regions 1010, corresponding deblocking models are used to remove artifacts. As Figure 10 described, once the boundary region 1010 is extracted, a classification component 1020 (which can be a DNN-based model or a traditional method used in a video codec) is used to classify the boundary region into different categories. Then, the boundary region 1030 is sent to different deblocking components 1040A, 1040B, and 1040N according to the category of the boundary region 1030. In practice, the following metrics (but not limited to) can be used to define the type of boundary region. First, the content of the boundary region can be used. For example, a boundary region with high-frequency content is sent to a model different from the model designed for deblocking low-frequency content. Second, the intensity of the artifact can be used. Note that the boundary regions in the multi-model deblocking components 1040A, 1040B, and 1040N are not limited to Figure 10 the boundary regions described in Figure 9 The deblocking regions described in

[0127] Figure 8 and Figure 10 The main goal of the DNN deblocking model described in

[0128] is to remove the artifacts between blocks. Pixels near the boundary can be deblocked more compared to the pixels far from the boundary. The deblocking model can include several convolutional layers. A CNN-based attention mechanism (such as non-local attention, SEnet (Squeeze-and-Excitation Networks, SEnet)) and a residual network (including a set of convolutional neural networks (convnet) and activation functions) can be used. For example, a DNN used for image super-resolution can be used here (e.g., by changing the output size to be the same as the input size).

[0129] Above, methods for performing deblocking on boundary regions with the help of a DNN or other learning-based methods were discussed. Alternatively, the encoder and decoder may be able to apply various levels of boundary strength on top of the methods described herein assuming that the adjustments to the pixels exported by the DNN are at the default level of boundary strength. By analyzing the boundary conditions and block coding characteristics, different levels of BS can be assigned to expand or reduce the default adjustment.

[0130] After reducing artifacts by using the deblocking module, the reconstructed image is sent to the enhancement module to generate the final reconstructed image Figure 11 The workflow of an embodiment of the post-enhancement component 720 is described. For ease of description, Figure 11 There are only four blocks of equal size in the figure. Note that in general, an image can be divided into a large number of blocks, and the size of each block can be different.

[0131] To improve the quality of the image, a DNN-based post-enhancement model can be used as the post-enhancement component 720. First, the region that has not been sent to the deblocking model can be taken out from block 1110 (such a region is hereinafter referred to as the remaining block 1120). As Figure 11 shown, the remaining block 1120 includes a region from one block, and the size of the boundary region is (n - m) × (n - m), where n is the side length of the block, and m is the length of the region used for deblocking. The value of m is an integer. For example, m can be 4 pixels. The remaining block 1120 is sent to the post-enhancement component 720 to further improve the quality of this region and generate the enhanced block 1140. Then, the enhanced block 1140 is updated 1150 back to the image. Alternatively, the remaining block may partially or wholly overlap with the deblocking boundary region. In the case where the remaining block wholly overlaps with the deblocking boundary region, the remaining block can be the entire coded block.

[0132] The above embodiment describes a method of using a single post-enhancement model to improve the remaining blocks. Figure 12 The workflow of embodiments of the multi-model post-enhancement components 1230A, 1230B, and 1230N is described. The key concept is that for different types of remaining blocks 1120, corresponding post-enhancement models are used to improve the quality. As Figure 12As described in, once the remaining block 1120 is taken out, a classification component 1210 (which can be a DNN-based classification model or a traditional method used in a video codec) is used to classify the remaining block into different categories. Therefore, the classified remaining block 1220 is sent to different post-enhancement components 1230A, 1230B, and 1230N according to the category of the classified remaining block 1220. In practice, the following metrics (but not limited to) are used to define the type of the remaining block. For example, the content in the remaining block area can be used. In this example, the remaining block area with high-frequency content is sent to a different model from the model designed to enhance low-frequency content.

[0133] Instead of enhancing the image at the block level, a post-enhancement model that enhances the entire image at once can also be considered. Figure 13 Describes the workflow of an implementation of a post-enhancement component 1310 for the entire image 1320 including the remaining block and the boundary region. For ease of description, Figure 13 There are only four blocks of equal size in. Note that in general, an image can be divided into a large number of blocks, and the size of each block can be different. By enhancing the image in this way, the artifact boundary is further reduced.

[0134] Figures 11 to 13 The main function of the DNN enhancement model described in is to enhance the quality of the image. The post-enhancement component 720 can include several convolutional layers. A CNN-based attention mechanism (such as non-local attention, SEnet) and a residual network (including a set of convolutional neural networks and activation functions) can be used to achieve the enhancement goal. For example, a DNN used for image super-resolution can be used here (e.g., by changing the output size to be the same as the input size).

[0135] In the above method, if two modules have an overlapping region, the same sample is subjected to modifications from both the post-enhancement component 720 and the deblocking component 710. When this happens, the modified samples from the post-enhancement are represented as P1, P2,...; similarly, the modified samples from the deblocking are represented as D1, D2,.... The weighted average of both the Pi samples and the Di samples (i = 0, 1,...) is used to generate the final output sample represented as Ai, as Figure 14 shown. This process is called pixel blending at the boundary region. For example, A i (i = 0, 1,... N-1) are two representative samples after blending in the overlapping region of the post-enhancement region 1410 and the deblocking region 1420. The two representative samples can be derived as follows: A i = Σ(w i *P i +(1 - w i )*D i), where w i is a weight parameter designed for position i in this area. For different positions, the weights can be different. In one example, w i is always set to 0.5. In another example, the value of w i depends on the distances to two modules. Generally, on samples generated by the same module, positions closer to one module will be assigned more weights. In Figure 14 In one example, A1 = 5 / 8 * P1 + 3 / 8 * D1; A2 = 3 / 8 * P2 + 5 / 8 * D2.

[0136] The goal of the training process is to learn the prediction DNN, encoding DNN, decoding DNN, prediction residual encoding DNN, prediction residual decoding DNN, deblocking DNN, and post-enhancement DNN.

[0137] During the training process, first, the weight coefficients of the DNNs to be learned above are initialized. For example, they are initialized by only using a pre-trained corresponding DNN model or by setting the weight coefficients of the DNN model to be learned to random numbers. Then, given the input training image x, the input training image x undergoes Figure 5A the encoding process described in Figure 6 to generate encoding information to the bitstream, and then this encoding information undergoes the decoding process described in to calculate and reconstruct the image For NIC, there may be two competing goals: better reconstruction quality and less bit consumption. Use the quality loss function

[0138]

[0139] where E measures the distortion of the decoded block residuals compared to the original block residuals before encoding, which is used as a regularization loss for the DNNs for residual encoding / decoding and the DNNs for encoding / decoding. β is a hyperparameter used to balance the importance of the regularization loss.

[0140] Training with a large λ results in a compression model with less distortion but more bit consumption, while training with a small λ results in a compression model with more distortion but less bit consumption. To learn the weight coefficients of the DNN, the encoded DNN and the predicted residual encoded DNN are decomposed into two parts, as Figure 15 shown. Similarly, the decoded DNN and the predicted residual decoded DNN are decomposed into two parts, as Figure 16 shown.

[0141] Figure 15 FIG. is a block diagram of an apparatus 1500 for DNN encoding and DNN predicted residual encoding according to some embodiments. Figure 16 FIG.

[0141] is a block diagram of an apparatus 1600 for DNN decoding and DNN predicted residual decoding according to some embodiments.

[0142] As Figure 15 shown, the encoded DNN is decomposed into a neural encoder 1510 and an encoder 1520 thereafter. The corresponding processing modules are also decomposed into a neural encoding module and an encoding module thereafter. Similarly, the predicted residual encoded DNN is decomposed into a neural residual encoder 1530 and a residual encoder 1540 thereafter. The corresponding processing modules are also decomposed into a neural predicted residual encoding module and a predicted residual encoding module thereafter. In some embodiments, the neural encoder 1510 or the neural residual encoder 1530 is a DNN learned during the training process and deployed at the test stage. On the other hand, the encoder 1520 or the residual encoder 1540 is different for training and testing. At the test stage, the encoder 1520 or the residual encoder 1540 is an actual encoder that may include a quantizer followed by an entropy encoder. While at the training stage, the encoder 1520 or the residual encoder 1540 uses a statistical data sampler to approximate the effect of the actual quantizer and skips the entropy encoder for training.

[0143] Similarly, as Figure 16 shown, the decoded DNN is decomposed into a decoder 1610 and a neural decoder 1620 thereafter. The corresponding processing modules are also decomposed into a decoding module and a neural decoding module thereafter. The predicted residual decoded DNN is decomposed into a residual decoder 1630 and a neural residual decoder 1640 thereafter. The corresponding processing modules are also decomposed into a predicted residual decoding module and a neural predicted residual decoding module thereafter.

[0144] In some embodiments, the neural decoder 1620 or the neural residual decoder 1640 is a DNN that is learned during the training process and deployed at the test stage. On the other hand, the decoder 1610 or the residual decoder 1630 is different for training and testing. At the test stage, the decoder 1610 or the residual decoder 1630 is the actual decoder that may include an entropy decoder followed by a dequantizer. While at the training stage, the decoder 1610 or the residual decoder 1630 is skipped.

[0145] The present disclosure places no restrictions on the methods used for the neural encoder 1510, encoder 1520, decoder 1610, neural decoder 1620, neural residual encoder 1530, residual encoder 1540, residual decoder 1630, or neural residual decoder 1640. In other words, the NIC framework of the present disclosure can be adapted to different types of encoding DNNs, decoding DNNs, predictive residual encoding DNNs, and predictive residual decoding DNNs.

[0146] In some embodiments, the framework can generally use any DNN-based image compression method, such as the scale hyperprior encoder-decoder framework (or Gaussian mixture likelihood framework) and its variants, RNN-based recurrent compression methods and their variants.

[0147] In another embodiment, the framework can work with any existing image compression methods such as JPEG, JEPG2000, BPG.

[0148] It is worth mentioning that in embodiments of the learning process, the predictive DNN, encoding DNN, predictive residual encoding DNN, encoding DNN, predictive residual decoding DNN, deblocking DNN, and post-enhancement DNN can be updated together in an end-to-end (E2E) manner based on the gradients of the backpropagation mentioned above. In addition, any one of these DNNs can be learned separately. For example, the predictive DNN can be trained separately using some datasets (the same or different from the datasets used in the training process above), and these DNNs are fixed in the training process mentioned above, where only the weight coefficients of the remaining DNNs are updated.

[0149] As Figure 17A shown, to predict the block 1720 (of size w×h), the previously reconstructed pixels 1710 can be used as context. The context pixels are used as input to the prediction module, and the predicted samples of the block are generated. Hereinafter, the prediction method is discussed by how to form the context region according to the reconstructed pixels 1710.

[0150] Note that in all the following embodiments, the raster scan order is used, taking the top - left block as an example. In general, the starting block for scanning can be located at the top - left corner, top - right corner, bottom - left corner, and bottom - right corner. The scan order can be height - first and width - first.

[0151] Based on previous prediction experience in HEVC and VVC, the context region is around the block to be predicted. In some embodiments, the reconstructed pixel 1710 is regarded as two separate regions (i.e., the context region C above the prediction block 上方 and the context region C to the left of the prediction block 左侧 , in raster scan order, and the starting point is the top - left block). Figure 17B An embodiment of this prediction method is shown. As Figure 17B shown, the shape of the context region is "L" or "7". To predict a block based on the two context regions, the prediction model first transforms these two regions (C 上方 and C 左侧 ) into a latent representation, and then combines the two latent representations. The model generates the final block prediction based on the combined latent representation.

[0152] As Figure 17B shown, in some embodiments, the sizes of the left context and the above context can be 2h×w and 2w×h (two prediction block sizes). In other embodiments, the sizes of the left context and the above context can vary and may not be integer multiples of the prediction block size.

[0153] Instead of dividing the context region into C Figure 17B and C 左侧 as 上方 shown, in some embodiments, the context region is regarded as a rectangular region. For this, a padding operation may be required. Figure 17C An example is shown where the padding region is below C 上方 and to the right of C 左侧 . After padding, the padding region and the context region form a new rectangular region as the input to the prediction model. Note that in other embodiments, the padding region can be the same as the prediction block, or the padding region can include additional blocks (the idea is to combine the padding region and the context region to form a rectangular region, so the padding region can vary). Different from the prediction method with two context regions, the input to the prediction model in the padding method is a rectangular region. Then, the prediction model takes this input and transforms it into a latent representation, and thus generates the final prediction sample for the block.

[0154] The following are several embodiments of the padding method for padding the discussed padding region.

[0155] In some embodiments, zero padding can be used to fill the regions mentioned above.

[0156] In some embodiments, mean value filling can be used to fill the regions mentioned above. It can be calculated by taking the average of C 上方 and C 左侧 , or by taking the average of the entire image, or by taking the average of an image dataset (such as the zero-mean method used in ImageNET).

[0157] In some embodiments, pixels extending from C 上方 can be used to fill the region.

[0158] In some embodiments, pixels extending from C 左侧 can be used to fill the region.

[0159] In other embodiments, a combination (such as summing, averaging) of pixels extending from C 左侧 and C 上方 can be used to fill the region.

[0160] In some embodiments, in the case where it is assumed that the filled region is the current block to be predicted, the prediction block sample generation process in the conventional intra prediction method can be used to fill the filled region. Examples of such methods include the DC mode, PLANAR mode, unequal weight plane mode, PDPC (Position Dependent Prediction Combination), etc.

[0161] Other filling methods not mentioned in the present disclosure can also be used in the filling prediction method.

[0162] In all the above filling methods, the filled region can be larger than the prediction block or can be the same as the prediction block. In the case where the filled region is larger than the prediction block, the width of C 上方 will be larger than the prediction block; in the case where the filled region is the same as the prediction block, the width of C 上方 will be the same as the prediction block. Similarly, the height of the filled region can also be the same as or larger than the prediction block.

[0163] In a separation method different from filling, the surrounding context region can be transformed into a rectangular shape as follows.

[0164] As Figure 17D shown, in the transformation method, C 左侧 1730 is transformed into a horizontal shape 1740 (for ease of description, only 3 pixels are shown), and the transformed C 左侧 is filled and combined with C上方 the intermediate region 1750 between Figure 17D the gray pixels 1760 in part (a) of ) to form a rectangular context region 1770. Then, the shape of the context region is transformed into a rectangular region (from 'L' to '-'). The input to the prediction model is a rectangular region. Then, the prediction model takes this input and transforms it into a latent representation and thus generates the final prediction for the block.

[0165] In some embodiments, C 上方 is transformed into a vertical shape. After padding, a rectangular region (from 'L' to 'I') is obtained.

[0166] In some embodiments, zero padding can be used to pad the intermediate region.

[0167] In some embodiments, mean padding can be used to pad the gray region, and the same value can be used to pad the entire gray region. The value can be calculated by taking the average of C 上方 and C 左侧 or by taking the average of the entire image or by taking the average of an image dataset (such as the zero-mean method used in ImageNET).

[0168] In some embodiments, mean padding can be used to pad the intermediate region. Different values can be used to pad different rows. The value can be calculated by taking the average of the pixels in the same row in C 左侧 or by taking the average of the pixels in the same row in C 上方 or by combining (e.g., averaging, summing) the pixels in the same row in both C 左侧 and C 上方 for calculation.

[0169] In some embodiments, the pixels extending from the same row in C 上方 can be used to pad the intermediate region.

[0170] In some embodiments, the pixels extending from the same row in C 左侧 can be used to pad the intermediate region.

[0171] In some embodiments, the combination (e.g., averaging, summing) of the pixels extending from the same row in C 左侧 and the pixels extending from the same row in C 上方 can be used to pad the intermediate region.

[0172] In some embodiments, the combination (e.g., averaging, summing) of the pixels extending from the same row in C 左侧 and the pixels extending from the same row in C 上方The combination of the nearest pixels extending along the same row in (e.g., averaging, summing) is used to fill the intermediate region. That is, after the transformation, the adjacent (left and right) pixels in the same row are used to fill the gray region.

[0173] Alternatively, as Figure 17D shown in part (b) of, for different rows of the context region 1730, an additional number of pixels are used to equalize the total number of each row. Then, after the transformation 1780 and the shift 1790, a rectangular region can be established without filling the intermediate region. When the additional pixels are on the right or below the left context region, any of the above methods can be used to fill the left context region.

[0174] Other filling methods not mentioned in the present disclosure can also be used in the filling prediction method.

[0175] In C 上方 and C 左侧 When they have the same shape (e.g., when the prediction block is square, and the shape of the upper context region is 3×w×2w, and the shape of the left context region is 3×2w×w, where w is the width of the block), the following method can be used to combine the two context regions.

[0176] The two context regions can be transformed and combined into a 3×2w×2w context region and sent to the prediction model to generate the final prediction.

[0177] The upper context region can be transformed into 3×2w×w.

[0178] The left context region can be transformed into 3×w×2w.

[0179] The two context regions can be transformed and combined into a 6×W×2W or 6×2W×W region (channel-by-channel concatenation) and sent to the prediction model to generate the final prediction.

[0180] The upper context region can be transformed into 3×2w×w.

[0181] The left context region can be transformed into 3×w×2w.

[0182] Note that in other embodiments, first C 上方 and C 左侧 can be transformed into a latent representation, and then the above-mentioned method is used to combine the latent representation and then generate the final prediction.

[0183] Figure 18 is a flowchart of a method 1800 for multi-rate neural image compression using a stackable nested model structure according to some embodiments.

[0184] In some implementations, Figure 18 one or more of the processing blocks in Figure 18 may be executed by platform 320. In some implementations,

[0185] As Figure 18 shown, in operation 1810, method 1800 includes encoding blocks of an input image using a first neural network, where the encoded blocks are decoded by a decoder using a second neural network to generate reconstructed blocks.

[0186] In operation 1820, method 1800 includes performing intra prediction on the reconstructed blocks using a third neural network to generate predicted blocks.

[0187] In operation 1830, method 1800 includes determining a difference between a block of the input image and the generated predicted blocks to generate prediction residuals.

[0188] In operation 1840, method 1800 includes encoding the generated prediction residuals using a fourth neural network, where the encoded prediction residuals are decoded by a decoder using a fifth neural network.

[0189] In operation 1850, method 1800 includes adding the decoded prediction residuals to the generated predicted blocks to generate restored predicted blocks.

[0190] Method 1800 may further include: selecting, as a selected block, one of the reconstructed blocks and the generated restored predicted blocks that has a smaller loss relative to the block of the input image; if the selected block is a reconstructed block, sending a selection signal and the encoded blocks to the decoder, the selection signal indicating the reconstructed block, and the sent encoded blocks being decoded by the decoder based on the sent selection signal; and if the selected block is a restored predicted block, sending a selection signal and the encoded prediction residuals to the decoder, the selection signal indicating the restored predicted block, and the sent encoded prediction residuals being decoded by the decoder based on the sent selection signal.

[0191] Performing intra prediction on the reconstructed blocks may include: grouping a plurality of reconstructed blocks into one or more groups based on the positions of the plurality of reconstructed blocks; performing a transform on each of the one or more groups to generate one or more latent spaces; combining the generated one or more latent spaces; and performing intra prediction on the combined one or more latent spaces to generate predicted blocks.

[0192] Method 1800 may further include: removing a boundary region including artifacts from each of the plurality of reconstructed blocks; performing deblocking processing on the removed boundary regions using one or more neural networks to generate deblocked regions without artifacts; and updating each of the plurality of reconstructed blocks with the generated deblocked regions.

[0193] Method 1800 may further include: taking out the remaining block regions excluding the boundary regions from each of the plurality of reconstructed blocks; performing post-enhancement on the remaining block regions using at least one neural network to generate enhanced block regions; and updating the reconstructed block with the generated enhanced block regions.

[0194] Performing intra prediction on a reconstructed block may include: obtaining a first context region above the reconstructed block in the reconstructed pixels and a second context region to the left of the reconstructed block in the reconstructed pixels; performing a transformation on the obtained first context region and the obtained second context region respectively to generate a latent representation; combining the generated latent representations; and performing intra prediction on the combined latent representation to generate a predicted block.

[0195] Performing intra prediction on a reconstructed block may include: obtaining a first context region above the reconstructed block in the reconstructed pixels, a second context region to the left of the reconstructed block in the reconstructed pixels, and a padding region below the first context region and to the right of the second context region; combining the obtained first context region, the obtained second context region, and the obtained padding region to obtain a rectangular region; performing a transformation on the obtained rectangular region to generate a latent representation; and performing intra prediction on the generated latent representation to generate a predicted block.

[0196] Although Figure 18 example blocks of method 1800 are shown, in some implementations, compared with the blocks depicted in Figure 18 method 1800 may include additional blocks, fewer blocks, different blocks, or differently arranged blocks. Additionally or alternatively, two or more of the blocks of method 1800 may be executed in parallel.

[0197] Figure 19 is a block diagram of an apparatus 1900 for multi-rate neural image compression using a stackable nested model structure according to some embodiments.

[0198] As Figure 19 shown, apparatus 1900 includes a first encoding code 1910, a first execution code 1920, a determination code 1930, a second encoding code 1940, and an addition code 1950.

[0199] The first encoding code 1910 is configured to cause at least one processor to encode blocks of an input image using a first neural network, wherein the encoded blocks are decoded by a decoder using a second neural network to generate reconstructed blocks.

[0200] The first execution code 1920 is configured to cause at least one processor to perform intra prediction on the reconstructed blocks using a third neural network to generate predicted blocks.

[0201] The determination code 1930 is configured to cause at least one processor to determine a difference between a block of the input image and the generated predicted blocks to generate a prediction residual.

[0202] The second encoding code 1940 is configured to cause at least one processor to encode the generated prediction residuals using a fourth neural network, wherein the encoded prediction residuals are decoded by a decoder using a fifth neural network.

[0203] The addition code 1950 is configured to cause at least one processor to add the decoded prediction residuals to the generated predicted blocks to generate restored predicted blocks.

[0204] The program code may further include: a selection code configured to cause at least one processor to select, from among the reconstructed blocks and the generated restored predicted blocks, the one having a smaller loss relative to the block of the input image as a selected block; and a transmission code configured to: if the selected block is a reconstructed block, send a selection signal and the encoded blocks to the decoder, the selection signal indicating the reconstructed block, and the sent encoded blocks are decoded by the decoder based on the sent selection signal; and if the selected block is a restored predicted block, send a selection signal and the encoded prediction residuals to the decoder, the selection signal indicating the restored predicted block, and the sent encoded prediction residuals are decoded by the decoder based on the sent selection signal.

[0205] The first execution code 1920 may further be configured to cause at least one processor to: group a plurality of reconstructed blocks into one or more groups based on the positions of the plurality of reconstructed blocks; perform a transformation on each of the one or more groups to generate one or more latent spaces; combine the generated one or more latent spaces; and perform intra prediction on the combined one or more latent spaces to generate predicted blocks.

[0206] The program code may further include: a first removal code configured to cause at least one processor to remove a boundary region including artifacts from each of a plurality of reconstructed blocks; a second execution code configured to cause at least one processor to perform deblocking processing on the removed boundary region using one or more neural networks to generate a deblocked region without artifacts; and a first update code configured to cause at least one processor to update each of the plurality of reconstructed blocks with the generated deblocked region.

[0207] The program code may further include: a second removal code configured to cause at least one processor to extract the remaining block regions excluding the boundary region from each of the plurality of reconstructed blocks; a third execution code configured to cause at least one processor to perform post-enhancement on the remaining block regions using at least one neural network to generate an enhanced block region; and a second update code configured to cause at least one processor to update the reconstructed block with the generated enhanced block region.

[0208] The first execution code 1920 may further be configured to cause at least one processor to: obtain a first context region above a reconstructed block in the reconstructed pixels and a second context region to the left of the reconstructed block in the reconstructed pixels; perform a transformation on the obtained first context region and the obtained second context region respectively to generate a latent representation; combine the generated latent representations; and perform intra prediction on the combined latent representation to generate a predicted block.

[0209] The first execution code 1920 may further be configured to cause at least one processor to: obtain a first context region above a reconstructed block in the reconstructed pixels, a second context region to the left of the reconstructed block in the reconstructed pixels, and a padding region below the first context region and to the right of the second context region; combine the obtained first context region, the obtained second context region, and the obtained padding region to obtain a rectangular region; perform a transformation on the obtained rectangular region to generate a latent representation; and perform intra prediction on the generated latent representation to generate a predicted block.

[0210] Compared with previous E2E image compression methods, the present disclosure has the following advantages. Instead of encoding the original pixels, by encoding the residuals between the predicted blocks and the original blocks, a prediction mechanism is utilized to improve the NIC encoding efficiency. The flexible and general framework adapts to different intra prediction methods and different neural encoding methods for residuals in intra prediction processing. The flexible and general framework adapts to various types of quality metrics.

[0211] Compared with previous prediction methods, the present disclosure has the following advantages. These embodiments can be used in neural-based image compression methods. For the filling method and the transformation method, there is no need to divide the context area. For the filling method, the transformation method, and the method in special cases, the context is a rectangular area, and thus additional calculations for merging two separate areas can be reduced.

[0212] These methods can be used alone or in any order of combination. In addition, each of the method (or embodiment), the encoder, and the decoder can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0213] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure, or may be acquired from practice of the implementations.

[0214] As used herein, the term "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0215] It will be apparent that the systems and / or methods described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code for implementing these systems and / or methods does not limit the implementations. Thus, the operations and behaviors of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed based on the description herein to implement the systems and / or methods.

[0216] Even if combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or not disclosed in the specification. Although each of the dependent claims listed below may directly refer to only one claim, the disclosure of possible implementations includes each dependent claim combined with every other claim in the claim group.

[0217] No element, act, or instruction used herein is to be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Further, as used herein, the term "group" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with "one or more." The term "one" or similar language is used where only one item is intended. Also, as used herein, the terms "having," "comprising," "including," etc. are intended to be open-ended terms. Further, unless otherwise explicitly stated, the phrase "based on" is intended to mean "at least partially based on."

Claims

1. A method for block-by-block neural image compression, comprising: Encoding blocks of an input image using a first neural network, wherein the encoded blocks are decoded by a decoder using a second neural network to generate reconstructed blocks; Performing intra prediction on the reconstructed blocks using a third neural network to generate predicted blocks; Determining a difference between the blocks of the input image and the generated predicted blocks to generate prediction residuals; Encoding the generated prediction residuals using a fourth neural network, wherein the encoded prediction residuals are decoded by the decoder using a fifth neural network; and Adding the decoded prediction residuals to the generated predicted blocks to generate restored predicted blocks.

2. The method according to claim 1, further comprising: Selecting, from the reconstructed blocks and the generated restored predicted blocks, the one having a smaller loss relative to the blocks of the input image as a selected block; If the selected block is the reconstructed block, sending a selection signal and the encoded block to the decoder, the selection signal indicating the reconstructed block, and the sent encoded block being decoded by the decoder based on the sent selection signal; And If the selected block is the restored predicted block, sending a selection signal and the encoded prediction residuals to the decoder, the selection signal indicating the restored predicted block, and the sent encoded prediction residuals being decoded by the decoder based on the sent selection signal.

3. The method according to claim 1, wherein, Performing the intra prediction on the reconstructed blocks includes: Grouping the plurality of reconstructed blocks into one or more groups based on the positions of the plurality of reconstructed blocks; Performing a transform on each of the one or more groups respectively to generate one or more latent spaces; Combining the generated one or more latent spaces to obtain a combined latent space; and Performing the intra prediction on the combined latent space to generate the predicted blocks.

4. The method according to claim 1, further comprising: Removing boundary regions containing artifacts from each of the plurality of reconstructed blocks; Performing deblocking on the removed boundary regions using one or more neural networks to generate deblocked regions without the artifacts; And Updating each of the plurality of reconstructed blocks using the generated deblocked regions.

5. The method according to claim 4, further comprising: Taking out the remaining block regions excluding the boundary regions from each of the plurality of reconstructed blocks; Performing post-enhancement on the remaining block regions using at least one neural network to generate enhanced block regions; And Updating the reconstructed blocks using the generated enhanced block regions.

6. The method according to claim 1, wherein Performing the intra prediction on the reconstructed blocks includes: Obtaining a first context region above the reconstructed block in the reconstructed pixels and a second context region to the left of the reconstructed block in the reconstructed pixels; Performing a transform on the first context region and the second context region respectively to generate latent representations; Combining the generated latent representations; and Performing the intra prediction on the combined latent representations to generate the predicted blocks.

7. The method according to claim 1, wherein, Performing the intra prediction on the reconstructed blocks includes: Obtain a first context region above the reconstruction block in the reconstructed pixels, a second context region to the left of the reconstruction block in the reconstructed pixels, and a padding region below the first context region and to the right of the second context region; Combine the first context region, the second context region, and the padding region to obtain a rectangular region; Perform a transformation on the obtained rectangular region to generate a latent representation; and Perform the intra prediction on the generated latent representation to generate the prediction block.

8. An encoder, comprising: A first encoding module configured to encode a block of an input image using a first neural network, wherein the encoded block is decoded by a decoder using a second neural network to generate a reconstruction block; An intra prediction module configured to perform intra prediction on the reconstruction block using a third neural network to generate a prediction block; A prediction residual generation module configured to determine a difference between the block of the input image and the generated prediction block to generate a prediction residual; A second encoding module configured to encode the generated prediction residual using a fourth neural network, wherein the encoded prediction residual is decoded by the decoder using a fifth neural network; and An addition module configured to add the decoded prediction residual and the generated prediction block to generate a restored prediction block.

9. An apparatus for block-by-block neural image compression, the apparatus being implemented as an encoder, and the apparatus comprising: At least one memory configured to store program code; And At least one processor configured to read and execute the program code to implement the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable medium storing instructions, the instructions, when executed by at least one processor of an encoder, implementing the method according to any one of claims 1 to 7.

Images