Multi-scale neural image compression method, device and electronic equipment

Abstract

An embodiment of the present application provides a multi-scale neural image compression method, apparatus, and electronic device using intra-prediction residuals. The method includes: downsampling an input image, generating a current prediction image based on a previously recovered prediction image, and generating a prediction residual based on the difference between the downsampled input image and the generated current prediction image. The method further includes: encoding the generated prediction residual, decoding the encoded prediction residual, and generating a currently recovered prediction image based on the addition of the current prediction image and the decoded prediction residual. The method further includes: upsampling the currently recovered prediction image, generating a scale residual based on the difference between the input image and the upsampled currently recovered prediction image, and encoding the scale residual.

Description

[0001] Cross - reference to related applications

[0002] This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 074,405, filed on September 3, 2020, and U.S. Patent Application No. 17 / 333,319, filed on May 28, 2021, the disclosures of which are incorporated herein by reference in their entireties. Technical field

[0003] This application relates to video coding and decoding, and particularly to a multi - scale neural image compression method, apparatus, and electronic device. Background art

[0004] ISO / IEC MPEG (International Organization for Standardization / International Electrotechnical Commission Moving Picture Experts Group) (JTC 1 / SC29 / WG 11) has been actively seeking potential requirements for the standardization of future video coding and decoding technologies. ISO / IEC JPEG (Joint Photographic Experts Group) established the JPEG - AI group focusing on end - to - end neural image compression based on artificial intelligence (AI) using deep neural networks (DNN). The success of recent methods has brought more and more industrial benefits to advanced neural image and video compression methods.

[0005] Given an input image x, the goal of neural image compression (NIC) is to use the image x as the input to a DNN encoder to compute a compressed representation that is compact for storage and transmission, and then use as the input to a DNN decoder to reconstruct the image Previous NIC methods employed a variational autoencoder (VAE) structure, where the DNN encoder directly uses the entire image x as its input, and this input is passed through a set of network layers that work like a black box to compute an output representation Correspondingly, the DNN decoder takes the entire representation as its input, and this input is passed through another set of network layers that work like another black box to compute the reconstructed

[0006] ​It has been proven that, instead of directly encoding the entire original image, a block-based intra-frame prediction and residual encoding and decoding mechanism is very effective for compressing image frames in modern video coding standards such as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC). The entire image is divided into blocks of various sizes, and a predicted block is generated by replicating the boundary pixels of a previously compressed block along various different angular directions. Then, the residual between the original block and the predicted block is compressed. The residual can be encoded more efficiently than the original pixels, and better coding performance can be achieved.

[0007] Multi-scale processing is another effective mechanism for improving the efficiency of learning image representations for image reconstruction tasks such as NIC. By using different encoding and decoding strategies at different scales, better coding performance can be achieved. Summary of the Invention

[0008] According to an embodiment, a multi-scale neural image compression method using intra-frame prediction residuals, executed by at least one processor, and includes: downsampling an input image to obtain a downsampled input image; generating a current predicted image by intra-frame prediction based on a previously recovered predicted image; and generating a prediction residual based on the difference between the downsampled input image and the generated current predicted image. The method further includes: encoding the generated prediction residual to obtain an encoded prediction residual; decoding the encoded prediction residual to obtain a decoded prediction residual; and generating a current recovered predicted image based on the addition of the current predicted image and the decoded prediction residual. The method further includes: upsampling the current recovered predicted image to obtain an upsampled current recovered predicted image; generating a scale residual based on the difference between the input image and the upsampled current recovered predicted image; and encoding the scale residual to obtain an encoded scale residual.

[0009] According to an embodiment, an apparatus for multi-scale neural image compression using intra-prediction residuals includes: at least one memory configured to store program code; and at least one processor configured to read the program code and operate in accordance with the instructions of the program code. The program code includes: downsampling code configured to cause the at least one processor to downsample an input image to obtain a downsampled input image; first generation code configured to cause the at least one processor to generate a current predicted image by intra-prediction based on a previously recovered predicted image; and second generation code configured to cause the at least one processor to generate a prediction residual based on the difference between the downsampled input image and the generated current predicted image. The program code further includes: first encoding code configured to cause the at least one processor to encode the generated prediction residual to obtain an encoded prediction residual; first decoding code configured to cause the at least one processor to decode the encoded prediction residual to obtain a decoded prediction residual; and third generation code configured to cause the at least one processor to generate a current recovered predicted image based on the addition of the current predicted image and the decoded prediction residual. The program code further includes: upsampling code configured to cause the at least one processor to upsample the current recovered predicted image to obtain an upsampled current recovered predicted image; fourth generation code configured to cause the at least one processor to generate a scale residual based on the difference between the input image and the upsampled current recovered predicted image; and second encoding code configured to cause the at least one processor to encode the scale residual to obtain an encoded scale residual.

[0010] According to an embodiment, a non-transitory computer-readable medium storing instructions that, when executed by at least one processor for multi-scale neural image compression using intra-prediction residuals, cause the at least one processor to: downsample an input image to obtain a downsampled input image; generate a current predicted image by intra-prediction based on a previously recovered predicted image; and generate a prediction residual based on the difference between the downsampled input image and the generated current predicted image. When executed by at least one processor, the instructions further cause the at least one processor to: encode the generated prediction residual to obtain an encoded prediction residual; decode the encoded prediction residual to obtain a decoded prediction residual; and generate a current recovered predicted image based on the addition of the current predicted image and the decoded prediction residual. When executed by at least one processor, the instructions further cause the at least one processor to: upsample the current recovered predicted image to obtain an upsampled current recovered predicted image; generate a scale residual based on the difference between the input image and the upsampled current recovered predicted image; and encode the scale residual to obtain an encoded scale residual. Description of the Drawings

[0011] Figure 1Is a schematic diagram of an environment in which the methods, apparatuses, and systems described herein can be implemented according to an embodiment.

[0012] Figure 2 Is Figure 1 A block diagram of example components of one or more devices of

[0013] Figure 3 Is a block diagram of an encoder for multi-scale neural image compression using intra-prediction residuals according to an embodiment.

[0014] Figure 4 Is a block diagram of an apparatus for block-based intra-prediction according to an embodiment.

[0015] Figure 5 Is a block diagram of a decoder for multi-scale neural image compression using intra-prediction residuals according to an embodiment.

[0016] Figure 6 Is a block diagram of a decomposed scale residual coding DNN and a decomposed prediction residual coding DNN according to an embodiment.

[0017] Figure 7 Is a block diagram of a decomposed scale residual decoding DNN and a decomposed prediction residual decoding DNN according to an embodiment.

[0018] Figure 8 Is a flowchart of a multi-scale neural image compression method using intra-prediction residuals according to an embodiment.

[0019] Figure 9 Is a block diagram of an apparatus for multi-scale neural image compression using intra-prediction residuals according to an embodiment. Detailed Description

[0020] The present disclosure describes a NIC framework for compressing an input image by a DNN using multi-scale residuals from block-based intra-prediction.

[0021] Figure 1 Is a schematic diagram of an environment 100 in which the methods, apparatuses, and systems described herein can be implemented according to an embodiment.

[0022] As Figure 1 Shown, the environment 100 may include a user device 110, a platform 120, and a network 130. The devices of the environment 100 may be interconnected by a wired connection, a wireless connection, or a combination of wired and wireless connections.

[0023] User device 110 includes one or more devices that are capable of receiving, generating, storing, processing, and / or providing information related to platform 120. For example, user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a wearable device (e.g., smart glasses or a smart watch), or a similar device. In some embodiments, user device 110 may receive information from platform 120 and / or send information to platform 120.

[0024] Platform 120 includes one or more devices as described elsewhere herein. In some embodiments, platform 120 may include a cloud server or a group of cloud servers. In some embodiments, platform 120 may be designed to be modular such that software components can be swapped in or out according to specific needs. In this way, platform 120 can be easily and / or quickly reconfigured to have different uses.

[0025] In some embodiments, as shown, platform 120 may be hosted in cloud computing environment 122. It is noted that while the embodiments described herein describe platform 120 as being hosted in cloud computing environment 122, in some embodiments, platform 120 is not cloud-based (i.e., can be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0026] Cloud computing environment 122 includes an environment that hosts platform 120. Cloud computing environment 122 may provide services such as computing, software, data access, storage, etc., which do not require an end user (e.g., user device 110) to know the physical location and configuration of the systems and / or devices of hosted platform 120. As shown, cloud computing environment 122 may include a set of computing resources 124 (collectively referred to as "computing resources 124" and individually referred to as "computing resource 124").

[0027] Computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some embodiments, computing resources 124 may host platform 120. Cloud resources may include computing instances executed in computing resources 124, storage devices provided in computing resources 124, data transfer devices provided by computing resources 124, etc. In some embodiments, computing resources 124 may communicate with other computing resources 124 via a wired connection, a wireless connection, or a combination of wired and wireless connections.

[0028] Further as Figure 1As shown, the computing resources 124 include a set of cloud resources, such as one or more applications ("APP") 124-1, one or more virtual machines ("VM") 124-2, virtualized storage ("VS") 124-3, one or more hypervisors ("HYP") 124-4, etc.

[0029] The application 124-1 includes one or more software applications, which can be provided to and / or accessed by the user device 110 and / or the platform 120. The application 124-1 does not require the installation and execution of software applications on the user device 110. For example, the application 124-1 can include software related to the platform 120 and / or any other software that can be provided through the cloud computing environment 122. In some embodiments, an application 124-1 can send / receive information to / from one or more other applications 124-1 through the virtual machine 124-2.

[0030] The virtual machine 124-2 includes a software implementation of a machine (e.g., a computer) that executes programs, similar to a physical machine. The virtual machine 124-2 can be a system virtual machine or a process virtual machine, depending on the usage and correspondence of the virtual machine 124-2 to any real machine. The system virtual machine can provide a complete system platform that supports the execution of a complete operating system ("OS"). The process virtual machine can execute a single program and can support a single process. In some embodiments, the virtual machine 124-2 can execute on behalf of a user (e.g., the user device 110) and can manage the infrastructure of the cloud computing environment 122, such as data management, synchronization, or long-term data transfer.

[0031] The virtualized storage 124-3 includes one or more storage systems and / or one or more devices that use virtualization technology within the storage system or device of the computing resources 124. In some embodiments, within the context of the storage system, the types of virtualization can include block virtualization and file virtualization. Block virtualization can refer to the abstraction (or separation) of logical storage from physical storage so that the storage system can be accessed without considering the physical storage or heterogeneous structure. The separation can allow the administrator of the storage system to flexibly manage the storage of end users. File virtualization can eliminate the dependence between the data accessed at the file level and the location of the physical storage file. This can optimize the performance of storage usage, server consolidation, and / or uninterrupted file migration.

[0032] The hypervisor 124-4 can provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to execute simultaneously on a host computer such as computing resource 124. The hypervisor 124-4 can provide a virtual operating platform to the guest operating systems and can manage the execution of the guest operating systems. Multiple instances of various operating systems can share the virtualized hardware resources.

[0033] The network 130 includes one or more wired and / or wireless networks. For example, the network 130 can include a cellular network (e.g., a fifth generation (5G) network, a Long-Term Evolution (LTE) network, a third generation (3G) network, a Code Division Multiple Access (CDMA) network, etc.), a Public Land Mobile Network (PLMN), a Local Area Network (LAN), a Wide Area Network (WAN), a Metropolitan Area Network (MAN), a telephone network (e.g., a Public Switched Telephone Network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0034] Figure 1 The number and arrangement of the devices and networks shown are provided as examples. In fact, compared with Figure 1 the devices and / or networks shown, there can be more devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks with a different arrangement. Additionally, Figure 1 two or more of the devices shown can be implemented within a single device, or Figure 1 a single device shown can be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) in environment 100 can perform one or more functions described as being performed by another set of devices in environment 100.

[0035] Figure 2 is Figure 1 a block diagram of example components of one or more of the devices.

[0036] The device 200 can correspond to the user device 110 and / or the platform 120. As Figure 2As shown, device 200 may include bus 210, processor 220, memory 230, storage component 240, input interface 250, output interface 260, and communication interface 270.

[0037] Bus 210 includes components that permit communication among the components of device 200. Processor 220 is implemented in hardware, firmware, or a combination of hardware and software. Processor 220 is a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC), or another type of processing component. In some implementations, processor 220 includes one or more processors that can be programmed to perform functions. Memory 230 includes random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 220.

[0038] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include a hard disk (e.g., a magnetic disk, optical disk, magneto-optical disk, and / or solid state disk), compact disc (CD), digital versatile disc (DVD), floppy disk, cassette tape, magnetic tape, and / or another type of non-transitory computer-readable medium, as well as corresponding drives.

[0039] Input interface 250 includes components that permit device 200 to receive information, such as through a user input, e.g., a touch screen display, keyboard, keypad, mouse, button, switch, and / or microphone. Additionally or alternatively, input interface 250 may include sensors for sensing information (e.g., a global positioning system (GPS) component, accelerometer, gyroscope, and / or actuator). Output interface 260 includes components that provide output information from device 200, such as a display, speaker, and / or one or more light emitting diodes (LEDs).

[0040] Communication interface 270 includes transceiver-like components (e.g., a transceiver and / or separate receiver and transmitter) that enable device 200 to communicate with other devices, such as through a wired connection, wireless connection, or a combination of wired and wireless connections. Communication interface 270 may permit device 200 to receive information from and / or provide information to another device. For example, communication interface 270 may include an Ethernet interface, optical interface, coaxial interface, infrared interface, radio frequency (RF) interface, universal serial bus (USB) interface, Wi-Fi interface, cellular network interface, etc.

[0041] Device 200 may perform one or more processes described herein. Device 200 may perform these processes in response to a processor 220 executing software instructions stored by a non-volatile computer-readable medium (e.g., memory 230 and / or storage component 240). A computer-readable medium is defined herein as a non-volatile memory device. A memory device includes storage space within a single physical storage device or storage space distributed across multiple physical storage devices.

[0042] The software instructions may be read into the memory 230 and / or the storage component 240 from another computer-readable medium or from another device via a communication interface 270. When executed, the software instructions stored in the memory 230 and / or the storage component 240 may cause the processor 220 to perform one or more processes described herein. Additionally or alternatively, hardware wired circuitry may be used in place of or in combination with the software instructions to perform one or more processes described herein. Accordingly, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0043] Figure 2 The number and arrangement of the components shown are provided as an example. In fact, compared with the Figure 2 components shown, device 200 may include more components, fewer components, different components, or components arranged differently. Additionally or alternatively, a set of components of device 200 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of device 200.

[0044] A method and apparatus for multi-scale neural image compression using intra-prediction residuals will now be described in detail.

[0045] The embodiments described herein include a multi-scale block-based intra-prediction and residual encoding / decoding framework for NIC. Two mechanisms are used: encoding the residuals between the predicted blocks and the original blocks instead of encoding the original pixels; and adopting encoding / decoding strategies at different scales to greatly improve the encoding / decoding efficiency of NIC.

[0046] Figure 3 is a block diagram of an encoder 300 for multi-scale neural image compression using intra-prediction residuals according to an embodiment. Figure 4 is a block diagram of an apparatus 400 for block-based intra-prediction according to an embodiment. Figure 5 is a block diagram of a decoder 500 for multi-scale neural image compression using intra-prediction residuals according to an embodiment.

[0047] As Figure 3As shown, the encoder 300 includes a downsampler 310, an intra prediction section 320, a subtractor 330, a DNN prediction residual encoder 340, a DNN prediction residual decoder 350, an adder 360, an upsampler 370, a subtractor 380, and a DNN scale residual encoder 390.

[0048] On the encoder side, the downsampler 310 downsamples the input image x to generate a downsampled image x'. The previously recovered prediction image is generated by the intra prediction section 320 to produce a prediction image The subtractor 330 generates a prediction residual r', which is the difference between the downsampled image x' and the prediction image .

[0049] The DNN prediction residual encoder 340 encodes the prediction residual r' to generate an encoded prediction residual The encoded prediction residual is decoded by the DNN prediction residual decoder 350 to generate a decoded prediction residual The adder 360 adds the decoded prediction residual back to the prediction image to generate a newly recovered or current recovered prediction image

[0050] The upsampler 370 upsamples the recovered prediction image to generate an upsampled image The subtractor 380 generates a scale residual r, which is the difference between the original input image x and the upsampled image . The DNN scale residual encoder 390 encodes the scale residual r to generate an encoded scale residual

[0051] In an embodiment, the downsampler 310 uses a downsampling DNN that includes both a convolutional layer and a pooling layer to generate the downsampled image x' from the input image x. The upsampler 370 uses an upsampling DNN to generate an output higher-scale signal from the input lower-scale signal. The upsampling DNN may include several deconvolution layers. In an embodiment, any more complex DNN (such as a DNN for image super-resolution) may also be used as the upsampling DNN.

[0052] In an embodiment, the intra prediction section 320 uses a per-block intra prediction mechanism. The downsampled image x' (and the corresponding prediction residual r') is divided into multiple blocks, and the DNN prediction residual encoder 340 processes these blocks one by one based on a predefined scan order.

[0053] As Figure 4As shown, the apparatus 400 for block-based intra prediction includes an intra prediction section 410, a subtractor 420, a DNN prediction residual encoder 430, and a DNN prediction residual decoder 440.

[0054] Given a set of previously reconstructed or previously recovered prediction blocks The intra prediction section 410 (prediction DNN) generates a predicted image of the current prediction block The subtractor 420 subtracts the current prediction block from the ground truth block b′ of the downsampled input image x′ x to generate the corresponding prediction residual block of the prediction residual r′ The prediction DNN may include several convolutional layers and deconvolutional layers. For example, the DNN used for image super-resolution can be used here, for example, by changing the output size to be the same as the input size.

[0055] The residual block b′ r is encoded by the DNN prediction residual encoder 430 to generate the encoded prediction residual of the encoded prediction residual block The encoded prediction residual block is decoded by the DNN prediction residual decoder 440 to generate the decoded prediction residual of the decoded prediction residual block The decoded prediction residual block can be added back to the prediction block to generate the newly recovered or newly reconstructed block of the previously recovered predicted image Then, the newly reconstructed block is added to the set of previously reconstructed blocks and the intra prediction section 410 processes the next block.

[0056] Figure 5 As

[0057] shown, the decoder 500 includes a DNN scale residual decoder 510, a DNN prediction residual decoder 520, an intra prediction section 530, an adder 540, an upsampler 550, and an adder 560. Figure 3 On the decoder side, as described, the DNN scale residual decoder 510 generates the decoded scale residual based on the encoded scale residual The DNN prediction residual decoder 520 generates the decoded prediction residual based on the encoded prediction residual The intra prediction section 530 generates a predicted image ​​​​

[0058] The adder 540 adds the predicted image to the decoded prediction residual to generate the restored predicted image The upsampler 550 upsamples the restored predicted image using the upsampling DNN to generate the upsampled image The adder 560 adds the upsampled image back to the decoded scale residual to generate the final reconstructed image

[0059] In an embodiment, the intra prediction part 530 on the decoder side uses the same per-block intra prediction mechanism as on the encoder side. Given a set of previously reconstructed blocks the prediction DNN (the same as on the encoder side) is used to generate the current predicted block This current predicted block is added back to the decoded residual block of the decoded prediction residual to generate the reconstructed block Then, the reconstructed block is added to the set of previously reconstructed blocks and the intra prediction part 530 processes the next block.

[0060] In an embodiment, the DNN scale residual encoder 390 encodes the residual r using the scale residual encoding DNN. The DNN prediction residual encoder 340 encodes the prediction residual r′ using the prediction residual encoding DNN. The DNN scale residual decoder 510 decodes the encoded scale residual The DNN prediction residual decoder 350 or 520 decodes the encoded prediction residual The scale residual encoding DNN and the scale residual decoding DNN adopt the VAE structure, and the prediction residual encoding DNN and the prediction residual decoding DNN also adopt the VAE structure. The structures of the scale residual encoding / decoding DNN and the prediction residual encoding / decoding DNN may be different or the same.

[0061] The present disclosure imposes no restrictions on the network structures of the downsampling DNN, the upsampling DNN, the prediction DNN, the scale residual encoding / decoding DNN, and the prediction residual encoding / decoding DNN.

[0062] The goal of the training process is to learn the downsampling DNN, prediction DNN, scale residual encoding DNN, prediction residual encoding DNN, upsampling DNN, scale residual decoding DNN, and prediction residual decoding DNN. During the training process, for example, the weight coefficients of the above-mentioned DNNs to be learned are initialized using the corresponding pre-trained DNN models, or they are initialized by setting them to random numbers. Then, the input training image x passes through Figure 3 the encoding process described in to generate the encoded residual and Then, the encoded residual r^ and Figure 5 will pass through the decoding process described in to generate the decoded residual and and the reconstructed image For NIC, there are two competing objectives: better reconstruction quality and less bit consumption. The quality loss function

[0063]

[0064] is used to measure the reconstruction quality, which can be referred to as the distortion loss, such as peak signal-to-noise ratio (PSNR), multi-scale structural similarity (MS-SSIM), or a weighted combination of both. The rate loss is calculated to measure the bit consumption of the compressed representation. Therefore, a trade-off hyperparameter λ is used to optimize the joint rate-distortion (R-D) loss:

[0065] Figure 6 where Figure 7 measures the distortion of the decoded residual compared to the original residual before encoding, and this distortion is used as the regularization loss for the scale residual encoding DNN / scale residual decoding DNN and prediction residual encoding DNN / prediction residual decoding DNN. β is a hyperparameter used to balance the importance of the regularization loss.

[0066] Training with a larger hyperparameter λ results in a compressed model with less distortion but more bit consumption, and vice versa. To learn the weight coefficients of the DNNs, the scale residual encoding DNN and prediction residual encoding DNN are decomposed into two parts, as shown in Figure 6 Similarly, the scale residual decoding DNN and prediction residual decoding DNN are decomposed into two parts, as shown in Figure 7

[0067] As shown in Figure 6 ​As shown, the scale residual encoding DNN is decomposed into a neural scale residual encoder 610, followed by a scale residual encoder 620. The corresponding processing modules are also decomposed into a neural scale residual encoding module, followed by a scale residual encoding module. Similarly, the prediction residual encoding DNN is decomposed into a neural prediction residual encoder 630, followed by a prediction residual encoder 640. The corresponding processing modules are also decomposed into a neural prediction residual encoding module, followed by a prediction residual encoding module.

[0068] In an embodiment, the neural scale residual encoder 610 or the neural prediction residual encoder 630 is a DNN that is learned during the training process and deployed during the test phase. On the other hand, the scale residual encoder 620 or the prediction residual encoder 640 is different for training and testing. During the test phase, the scale residual encoder 620 or the prediction residual encoder 640 is the actual encoder, including a quantizer, followed by an entropy encoder. While during the training phase, the scale residual encoder 620 or the prediction residual encoder 640 uses a statistical data sampler to approximate the effect of the actual quantizer and skips the training of the entropy encoder.

[0069] Similarly, as Figure 7 shown, the scale residual decoding DNN is decomposed into a scale residual decoder 710, followed by a neural scale residual decoder 720. The corresponding processing modules are also decomposed into a scale residual decoding module, followed by a neural scale residual decoding module. The prediction residual decoding DNN is decomposed into a prediction residual decoder 730, followed by a neural prediction residual decoder 740. The corresponding processing modules are also decomposed into a prediction residual decoding module, followed by a neural prediction residual decoding module.

[0070] In an embodiment, the neural scale residual decoder 720 or the neural prediction residual decoder 740 is a DNN that is learned during the training process and deployed during the test phase. On the other hand, the scale residual decoder 710 or the prediction residual decoder 730 is different for training and testing. During the test phase, the scale residual decoder 710 or the prediction residual decoder 730 is the actual decoder, including an entropy decoder, followed by an inverse quantizer. While during the training phase, the scale residual decoder 710 or the prediction residual decoder 730 is skipped.

[0071] The present disclosure does not impose any restrictions on the methods for the neural scale residual encoder 610, the scale residual encoder 620, the scale residual decoder 710, the neural scale residual decoder 720, the neural prediction residual encoder 630, the prediction residual encoder 640, the prediction residual decoder 730, or the neural prediction residual decoder 740. In other words, the NIC framework of the present disclosure can accommodate different types of scale residual encoding DNNs, scale residual decoding DNNs, prediction residual encoding DNNs, and prediction residual decoding DNNs.

[0072] In an embodiment, the rate loss is further decomposed into:

[0073]

[0074] where each and is a rate loss associated with the bitrates of the compressed representations and respectively. The hyperparameter ρ balances the contributions of different loss terms. In an embodiment, an entropy estimation method is used to calculate these rate losses and where a DNN rate loss estimator (learned during the training phase and deployed during the testing phase) is used.

[0075] Moreover, the regularization loss is further decomposed into:

[0076]

[0077] where each or measures the recovered residual or is the reconstruction error (e.g., mean squared error (MSE) loss) compared to the residual or before DNN scale residual coding or before prediction residual coding. The hyperparameter γ balances the contributions of different loss terms. Based on both Equation (2) and Equation (3), for each training image x, the total loss of Equation (1) can be calculated. Then the gradient of the total loss of Equation (1) can be calculated, and based on this gradient, the weight coefficients of the learnable DNN can be updated through backpropagation.

[0078] In an embodiment of the learning process, the downsampling DNN, prediction DNN, scale residual coding DNN, prediction residual coding DNN, upsampling DNN, scale residual decoding DNN, and prediction residual decoding DNN can be updated together in an end-to-end (E2E) manner based on the above backpropagation gradient. Moreover, any one of these DNNs can be learned individually. For example, the downsampling DNN and the upsampling DNN can be trained separately using a certain dataset (the same or different from the dataset used in the above training process), and they can be fixed during the above training process where only the weight coefficients of the remaining DNNs are updated.

[0079] Figure 8 is a flowchart of a multi-rate neural image compression method 800 using intra prediction residuals according to an embodiment.

[0080] In some embodiments, Figure 8One or more processing blocks of can be executed by platform 120. In some embodiments, Figure 8 One or more processing blocks of can be executed by another device or group of devices (such as user device 110) that is separate from or includes platform 120.

[0081] As Figure 8 shown, in operation 810, method 800 includes downsampling an input image.

[0082] In operation 820, method 800 includes generating a current predicted image based on a previously recovered predicted image.

[0083] In operation 830, method 800 includes generating a prediction residual based on the difference between the downsampled input image and the generated current predicted image.

[0084] In operation 840, method 800 includes encoding the generated prediction residual.

[0085] In operation 850, method 800 includes decoding the encoded prediction residual.

[0086] In operation 860, method 800 includes generating a currently recovered predicted image based on the addition of the current predicted image and the decoded prediction residual.

[0087] In operation 870, method 800 includes upsampling the currently recovered predicted image.

[0088] In operation 880, method 800 includes generating a scale residual based on the difference between the input image and the upsampled currently recovered predicted image.

[0089] In operation 890, method 800 includes encoding the scale residual.

[0090] Method 800 may further include decoding the encoded scale residual and generating a reconstructed image based on the addition of the upsampled currently recovered predicted image and the decoded scale residual.

[0091] The downsampling of the input image, generating the current predicted image, encoding the generated prediction residual, decoding the encoded prediction residual, upsampling the currently recovered predicted image, encoding the scale residual, and decoding the encoded scale residual may each use a neural network.

[0092] The neural network can be trained by: determining a regularization loss based on the reconstruction error between the decoded scale residual and the encoded scale residual and the reconstruction error between the decoded prediction residual and the encoded prediction residual, and updating the weight coefficients of the neural network to minimize the determined regularization loss.

[0093] The neural network can be further trained by the following operations: determining a distortion loss based on the input image and the reconstructed image, and determining a rate loss based on the bitrates of the encoded scale residuals and the encoded prediction residuals. Updating the weight coefficients can include updating the weight coefficients of the neural network to minimize the determined distortion loss, the determined rate loss, and the determined regularization loss.

[0094] Generating the current predicted image can include generating a current prediction block based on a set of previously recovered prediction blocks of a previously recovered predicted image. Generating a prediction residual can include generating a prediction residual block based on the difference between the generated current prediction block and the true image block of the downsampled input image.

[0095] Encoding the generated prediction residuals can include encoding the generated prediction residual blocks, and decoding the encoded prediction residuals can include decoding the encoded prediction residual blocks.

[0096] Although Figure 8 example blocks of method 800 are shown, in some embodiments, method 800 can include Figure 8 blocks other than those depicted therein, fewer blocks than those, different blocks than those, or blocks arranged differently than those. Additionally or alternatively, two or more of the blocks of method 800 can be executed in parallel.

[0097] Figure 9 is a block diagram of a multi-rate neural image compression device 900 using intra-frame prediction residuals according to an embodiment.

[0098] As Figure 9 shown, the device 900 includes a downsampling code 910, a first generation code 920, a second generation code 930, a first encoding code 940, a first decoding code 950, a third generation code 960, an upsampling code 970, a fourth generation code 980, and a second encoding code 990.

[0099] The downsampling code 910 is configured to cause at least one processor to downsample an input image.

[0100] The first generation code 920 is configured to cause at least one processor to generate a current predicted image based on a previously recovered predicted image.

[0101] The second generation code 930 is configured to cause at least one processor to generate a prediction residual based on the difference between the downsampled input image and the generated current predicted image.

[0102] The first encoding code 940 is configured to cause at least one processor to encode the generated prediction residuals.

[0103] The first decoding code 950 is configured to cause at least one processor to decode the encoded prediction residual.

[0104] The third generation code 960 is configured to cause at least one processor to generate a current recovered prediction image based on the addition of the current prediction image and the decoded prediction residual.

[0105] The upsampling code 970 is configured to cause at least one processor to upsample the current recovered prediction image.

[0106] The fourth generation code 980 is configured to cause at least one processor to generate a scale residual based on the difference between the input image and the upsampled current recovered prediction image.

[0107] The second encoding code 990 is configured to cause at least one processor to encode the scale residual.

[0108] The program code may further include: a second decoding code configured to cause at least one processor to decode the encoded scale residual; and a fifth generation code configured to cause at least one processor to generate a reconstructed image based on the addition of the upsampled current recovered prediction image and the decoded scale residual.

[0109] The downsampling code 910, the first generation code 920, the first encoding code 940, the first decoding code 950, the upsampling code 970, the second encoding code 990, and the second decoding code may each use a neural network.

[0110] The neural network can be trained by: determining a regularization loss based on the reconstruction error between the decoded scale residual and the encoded scale residual and the reconstruction error between the decoded prediction residual and the encoded prediction residual, and updating the weight coefficients of the neural network to minimize the determined regularization loss.

[0111] The neural network can be further trained by: determining a distortion loss based on the input image and the reconstructed image, and determining a rate loss based on the bitrates of the encoded scale residual and the encoded prediction residual. Updating the weight coefficients may include updating the weight coefficients of the neural network to minimize the determined distortion loss, the determined rate loss, and the determined regularization loss.

[0112] The first generation code 920 may further be configured to cause at least one processor to generate a current prediction block based on a set of previously recovered prediction blocks of a previously recovered prediction image, and the second generation code 930 may further be configured to cause at least one processor to generate a prediction residual block based on the difference between the generated current prediction block and a ground truth image block of the downsampled input image.

[0113] The first encoding code 940 can be further configured to cause at least one processor to encode the generated prediction residual block, and the first decoding code 950 can be further configured to cause at least one processor to decode the encoded prediction residual block.

[0114] Compared with previous E2E image compression methods, the embodiments described herein have the following advantages. The embodiments utilize two mechanisms to improve NIC encoding and decoding efficiency: encoding the residual between the predicted block and the original block instead of encoding the original pixels; and using different encoding and decoding strategies at different scales. These embodiments include a flexible framework that adapts different intra-prediction methods and different neural encoding and decoding methods for the residuals from the scaling process and the residuals from the intra-prediction process. The embodiments further include a flexible framework that adapts various types of quality metrics.

[0115] These methods can be used alone or in any combination in any order. Additionally, each of the methods (or embodiments), encoders, and decoders can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0116] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible in light of the foregoing disclosure, or may be acquired from the practice of the implementations.

[0117] As used herein, the term component is intended to be broadly construed as hardware, firmware, or a combination of hardware and software.

[0118] It will be apparent that the systems and / or methods described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code used to implement these systems and / or methods is not a limitation on the implementations. Accordingly, the operations and behavior of the systems and / or methods are described herein without reference to specific software code—it being understood that software and hardware can be designed to implement the systems and / or methods based on the present description.

[0119] Even if combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. While each of the dependent claims listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.

[0120] The elements, acts, or instructions used herein shall not be construed as critical or essential, unless expressly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Further, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with "one or more." Where only one item is intended, the term "one" or similar language is used. Also, as used herein, the terms "has," "have," "having," etc. are intended to be open-ended terms. Further, the phrase "based on" is intended to mean "at least partially based on," unless expressly stated otherwise.

Claims

1. A multi-scale neural image compression method using intra-prediction residuals, characterized in that, The method includes: Downsampling the input image to obtain a downsampled input image; Generating a current predicted image by intra-frame prediction based on a previously recovered predicted image; Generating a prediction residual based on the difference between the downsampled input image and the generated current predicted image; Encoding the generated prediction residual to obtain an encoded prediction residual; Decoding the encoded prediction residual to obtain a decoded prediction residual; Generating a currently recovered predicted image based on the addition of the current predicted image and the decoded prediction residual; Upsampling the currently recovered predicted image to obtain an upsampled currently recovered predicted image; Generating a scale residual based on the difference between the input image and the upsampled currently recovered predicted image; and Encoding the scale residual to obtain an encoded scale residual; Decoding the encoded scale residual to obtain a decoded scale residual; wherein a neural network is used to encode the generated prediction residual, decode the encoded prediction residual, encode the scale residual, and decode the encoded scale residual, and the neural network is trained by the following operations: Determining a regularization loss based on a reconstruction error between the decoded scale residual and the scale residual before encoding, and a reconstruction error between the decoded prediction residual and the prediction residual before encoding; and Updating weight coefficients of the neural network to minimize the determined regularization loss.

2. The method according to claim 1, characterized in that, The method further includes: Generating a reconstructed image based on the addition of the upsampled currently recovered predicted image and the decoded scale residual.

3. The method according to claim 2, wherein The downsampling of the input image, the generation of the current predicted image, and the upsampling of the currently recovered predicted image respectively use a neural network.

4. The method according to claim 2, characterized in that, The neural network is further trained by the following operations: Determining a distortion loss based on the input image and the reconstructed image; and Determining a rate loss based on the bitrates of the encoded scale residual and the encoded prediction residual, wherein the updating of the weight coefficients includes: updating the weight coefficients of the neural network to minimize the determined distortion loss, the determined rate loss, and the determined regularization loss.

5. The method according to claim 1, wherein The generation of the current predicted image includes: generating a current prediction block based on a set of previously recovered prediction blocks of the previously recovered predicted image, and the generation of the prediction residual includes: generating a prediction residual block based on the difference between the generated current prediction block and a true image block of the downsampled input image.

6. The method according to claim 5, characterized in that, The encoding of the generated prediction residual includes: encoding the generated prediction residual block to obtain an encoded prediction residual block, and the decoding of the encoded prediction residual includes: decoding the encoded prediction residual block.

7. A multi-scale neural image compression device using intra-prediction residuals, characterized in that, The apparatus includes: At least one memory configured to store program code; and At least one processor configured to read the program code and operate according to the instructions of the program code, where the program code includes: Downsampling code configured to cause the at least one processor to downsample an input image to obtain a downsampled input image; A first generation code, configured to cause the at least one processor to generate a current predicted image by intra prediction based on a previously recovered predicted image; A second generation code, configured to cause the at least one processor to generate a prediction residual based on the difference between a downsampled input image and the generated current predicted image; A first encoding code, configured to cause the at least one processor to encode the generated prediction residual to obtain an encoded prediction residual; A first decoding code, configured to cause the at least one processor to decode the encoded prediction residual to obtain a decoded prediction residual; A third generation code, configured to cause the at least one processor to generate a currently recovered predicted image based on the addition of the current predicted image and the decoded prediction residual; An upsampling code, configured to cause the at least one processor to upsample the currently recovered predicted image to obtain an upsampled currently recovered predicted image; A fourth generation code, configured to cause the at least one processor to generate a scale residual based on the difference between the input image and the upsampled currently recovered predicted image; and A second encoding code, configured to cause the at least one processor to encode the scale residual to obtain an encoded scale residual; A second decoding code, configured to cause the at least one processor to decode the encoded scale residual to obtain a decoded scale residual; wherein a neural network is used to encode the generated prediction residual, decode the encoded prediction residual, encode the scale residual, and decode the encoded scale residual; the neural network is trained by: determining a regularization loss based on a reconstruction error between the decoded scale residual and the scale residual before encoding, and a reconstruction error between the decoded prediction residual and the prediction residual before encoding; and updating weight coefficients of the neural network to minimize the determined regularization loss.

8. The device according to claim 7, characterized in that, The program code further includes: A fifth generation code, configured to cause the at least one processor to generate a reconstructed image based on the addition of the upsampled currently recovered predicted image and the decoded scale residual.

9. The device according to claim 8, characterized in that, The downsampling code, the first generation code, and the upsampling code respectively use a neural network.

10. The device according to claim 8, characterized in that, The neural network is further trained by: determining a distortion loss based on the input image and the reconstructed image; and determining a rate loss based on bitrates of the encoded scale residual and the encoded prediction residual, and wherein updating the weight coefficients includes: updating the weight coefficients of the neural network to minimize the determined distortion loss, the determined rate loss, and the determined regularization loss.

11. The device according to claim 7, characterized in that The first generation code is further configured to cause the at least one processor to generate a current predicted block based on a set of previously recovered predicted blocks of the previously recovered predicted image, and The second generation code is further configured to cause the at least one processor to generate a prediction residual block based on the difference between the generated current predicted block and a true image block of the downsampled input image.

12. The device according to claim 11, wherein The first encoding code is further configured to cause the at least one processor to encode the generated prediction residual block to obtain an encoded prediction residual block, and The first decoding code is further configured to cause the at least one processor to decode the encoded prediction residual block.

13. An electronic device, characterized in that, Comprising: A memory for storing computer-readable instructions; A processor for reading the computer-readable instructions and performing the method according to any one of claims 1 to 6 according to the instructions of the computer-readable instructions.

14. A computer storage medium, characterized in that, Instructions are stored, and the instructions can be executed by at least one processor to perform the method according to any one of claims 1 to 6, generate a compressed image and store it.

Images