Dependent scalar quantization with replacement in neural image compression.

Abstract

A method, apparatus, and non-transitory storage medium for end-to-end neural image compression using dependent scalar quantization with permutation, comprising: receiving an input image; determining a permutation image based on the input image using a neural network-based permutation feature generator; compressing the permutation image; quantizing the compressed permutation image to obtain a quantized representation of the input image with higher compression capability by using a first dependent scalar quantizer; and entropy encoding the permutation image using a neural network-based encoder to generate a compressed representation of the quantized representation.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 212,460, filed June 18, 2021, and U.S. Patent Application No. 17 / 825,562, filed May 26, 2022, the disclosures of which are incorporated by reference in their entireties herein. [Background technology]

[0002] In recent years, ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) has been actively trying to standardize future video coding technologies, especially those that can leverage machine learning or artificial intelligence (ML / AI). ISO / IEC JPEG has established the JPEG-AI group, which focuses on AI-based end-to-end neural image compression. And China's AVS standard has also formed an AVS-AI special group to work on neural image and video compression technologies. Meanwhile, companies like Google are also funding specialized research projects for neural image compression (NIC).

[0003] In a neural network-based video or image coding framework, multiple models may be used, each of which requires a large dataset, and multiple machine learning models may be implemented. Conventional hybrid video codec frameworks may focus on training and optimizing each of the multiple models separately, which may increase the rate-distortion loss or increase the computational cost of the video or image coding framework, reducing the overall capacity of the image or video framework / process.

[0004] Therefore, there is a need for a way to optimize the coding framework and improve overall performance. Summary of the Invention [Means for solving the problem]

[0005] According to an embodiment, a method for neural image compression using dependent scalar quantization with permutation can be provided. The method can be executed by one or more processors. The method can include receiving an input image, determining a permutation image based on the input image using a neural network-based permutation feature generator, compressing the permutation image, quantizing the compressed permutation image to obtain a quantized representation of the input image with higher compression capability by using a first dependent scalar quantizer, and entropy encoding the permutation image using a neural network-based encoder to generate a compressed representation of the quantized representation.

[0006] According to an embodiment, an apparatus for neural image compression using dependent scalar quantization with permutation may be provided. The apparatus may include at least one memory configured to store program code and at least one processor configured to read the program code and operate as instructed by the program code. The program code may include a first receiving code configured to cause the at least one processor to receive an input image, a first determining code configured to cause the at least one processor to determine a permutation image based on the input image using a neural network-based permutation feature generator, a compressing code configured to cause the at least one processor to compress the permutation image, a quantizing code configured to cause the at least one processor to quantize the compressed permutation image to obtain a quantized representation of the input image with higher compression capability by using a first dependent scalar quantizer, and a first generating code configured to cause the at least one processor to entropy encode the permutation image using a neural network-based encoder to generate a compressed representation of the quantized representation.

[0007] According to an embodiment, a non-transitory computer-readable medium may be provided having instructions stored thereon that, when executed by at least one processor for neural image compression using dependent scalar quantization with permutation, may cause the at least one processor to receive an input image, determine a permutation image based on the input image using a neural network-based permutation feature generator, compress the permutation image, quantize the compressed permutation image to obtain a quantized representation of the input image with higher compression capability by using a first dependent scalar quantizer, and entropy encode the permutation image using a neural network-based encoder to generate a compressed representation of the quantized representation. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram of an environment in which the methods, apparatus, and systems described herein may be implemented, according to an embodiment. [Diagram 2] 2 is a block diagram of example components of one or more devices of FIG. 1. [Diagram 3] FIG. 2 is a diagram of an example dependent scalar quantization (DSQ) process in accordance with an embodiment. [Figure 4] FIG. 1 is a block diagram of an end-to-end neural image compression framework using dependent scalar quantization with replacement, according to an embodiment. [Figure 5A] 1 shows a flowchart of a method for end-to-end neural image quantization using dependent scalar quantization with replacement, according to an embodiment. [Figure 5B] 1 shows a flowchart of a method for end-to-end neural image quantization using dependent scalar quantization with replacement, according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Embodiments of the present disclosure relate to methods, apparatus, and systems for end-to-end (E2E) neural image compression (NIC), which may include receiving an input image, determining a replacement representation of the input image by performing transformation and quantization, and compressing the replacement representation. The E2E NIC framework can tune a deep neural network-based model / layer that generates the compressed representation by optimizing multiple quality metrics (e.g., rate-distortion capability) of the E2E NIC framework.

[0010] As mentioned above, traditional hybrid video codec frameworks may focus on separately training and optimizing machine learning models included in the image or video coding framework, resulting in a loss of overall capacity. On the other hand, the E2E NIC framework allows for joint optimization of image or video coding from input to output as a single module (with layers in between) to improve the final objective (e.g., minimizing rate-distortion loss). Thus, the E2E NIC framework can optimize the entire coding system to achieve better capacity and, in some cases, reduce the overall computational load of the framework.

[0011] In the E2E NIC framework, the quantization and compression processes may be particularly important. Although quantization may be a core process in image and video compression, quantization may also be a source of loss in compression quality. Thus, improvements in quantization efficiency may result in an improvement in the overall capacity of the image or video coding framework. According to embodiments of the present disclosure, utilizing a superior modification of the input image, a superior substitute or replacement image is one that can be better quantized and therefore better compressed. Thus, embodiments of the present disclosure relate to a novel E2E NIC framework that can utilize a more compressible replacement image using a neural network-based model to generate a replacement image, and subsequently quantize the replacement image for better compression capabilities. By utilizing this E2E NIC framework including dependent scalar quantization using a replacement image that is more suitable for compression, the overall coding capacity is improved and the compression loss introduced during the quantization of the original input image is reduced.

[0012] According to embodiments, the E2E NIC framework may be a deep neural network-based image or video coding method. The quantization process may utilize dependent scalar quantizers, and the quantized representation may be entropy coded to generate a compressed representation. In some embodiments, the E2E NIC framework may include any suitable neural network-based method, model, or layer. The embodiments disclosed herein are not intended to be limiting or exclusive. The E2E NIC framework may be pre-trained and fine-tuned using the methods disclosed herein. According to some embodiments of the present disclosure, the E2E NIC framework may be jointly trained and used for inference.

[0013] According to some embodiments, a process for neural network-based image compression may be as follows: Given an input image or video sequence x, based on the input x, a neural network-based encoder (e.g., a deep neural network (DNN)-based encoder) can compute a compressed representation f that is easier to store and transmit when compared to the input image x. The compressed representation f is a discrete-value quantized representation

number

number

number

number

number

[0014] Depending on the quality and characteristics of the input image, one or more side information, and one or more target quality criteria, the compressed representation of the input image may have losses that exceed a certain threshold. Also, in the above-mentioned neural network-based image compression process, quantization is a core step and one of the main causes of compression quality loss. Improving the efficiency of quantization can lead to significant performance gains in all image and video compression tasks. Therefore, according to an embodiment of the present disclosure, a method is provided for dependent scalar quantization with permutation of an input image that leverages more efficient quantization of a superior permutation image. This embodiment of the method improves the performance gains in all image and video compression tasks.

[0015] According to an embodiment of the present disclosure, a uniform scalar quantizer can be used as the quantizer during the encoding or inference phase. The uniform scalar quantizer may be replaced by a noise-injected quantizer during the training phase. During training of the E2E NIC model, the rate-distortion loss is calculated using a trade-off hyperparameter λ to estimate the rate-distortion loss of the reconstructed input image or video sequence.

number

number

[0016]

number

[0017] According to an embodiment of the present disclosure, a dependent scalar quantizer (DSQ) (e.g., a trellis-coded quantizer) can be used. The process of dependent scalar quantization can be a vector quantization process. DSQ is a quantizer that can be used to kTwo quantizers Q0 and Q1 may be utilized along with a state machine containing k states (k>0), and the state machine and its states are used to switch between these scalar coefficients. According to some embodiments, each state of the state machine may be associated with one of these scalar quantizers.

[0018] According to some embodiments, the DSQ may include manually designed quantization rules. The DSQ includes two quantizers Q0 and Q1 and a procedure for switching between them. Figure 3 provides an illustration of a DSQ mechanism that uses quantizers Q0 and Q1 in a DSQ design. The labels above the circles (e.g., A, B) indicate the associated states, and the labels below the circles indicate the associated quantization keys.

[0019] At the decoder side, the reconstructed numerical value x' is determined by an integer key k that multiplies the quantization step size Δ of either quantizer Q0 or Q1. The switching between quantizers Q0 and Q1 is performed with M=2 K The current DSQ state can be represented by a state machine with K states, K≧2 (hence M≧4), where each DSQ state can be associated with one of the quantizers Q0 or Q1. The current DSQ state is determined by the previous DSQ state and the current quantization key k i To encode the input stream x1, x2, ..., the possible transitions between quantizers Q0 and Q1 are K The problem can be represented by a trellis with DSQ states of k1, k2, .... Therefore, selecting the optimal sequence of quantization keys k1, k2, ... is equivalent to finding the path with the minimum rate-distortion (RD) cost. The problem can be solved by any suitable algorithm (e.g., the Viterbi algorithm).

[0020] According to an embodiment of the present disclosure, the E2E NIC framework may use an online training method to find an optimal replacement image for the input image for each input image or frame in a video sequence to be compressed, and then compress and quantize this replacement image instead of the input image. By quantizing the optimal replacement image or at least a good replacement image instead of the input image, the quantized representation achieves better compression and overall coding performance. According to an embodiment, the exemplary method of combining the generation of a replacement image with subsequent DSQ on the replacement image may be used to improve the compression performance of any suitable neural network-based E2E NIC framework.

[0021] According to embodiments of the present disclosure, the neural network-based image compression framework may include a pre-trained DNN model, and one or more model weights associated with the pre-trained model DNN model may be fixed. In some embodiments, one or more hyperparameters of the DNN model may be trained or fine-tuned.

[0022] According to an embodiment of the present disclosure, the E2E NIC framework and any models therein may have important hyperparameters: step size and number of steps. The step size may indicate the "learning rate" of online training. Online learning may include learning one or more models described herein in real-time. Images with different types of content may correspond to different step sizes to achieve the best optimization results. As an example, images of a particular resolution, images containing particular metadata (e.g., labels, features, etc.), or particular coding characteristics (e.g., prediction mode, CU size, block size, etc.) may correspond to different step sizes to achieve the best optimization results. The number of steps may indicate the number of updates operated. Target Loss Function

number

[0023] For example, compressed representation

number

number

[0024] In the related art, quantization may only include applying a rounding function to the coded features of the input image. However, according to embodiments of the present disclosure, quantization may include DSQ. Also, quantization may be performed on the coded features of the generated permuted image instead of the input image as in the related art. According to embodiments of the present disclosure, the overall loss may be observed for multiple iterations during training (e.g., mean square error (MSE), binary cross entropy (BCE), categorical cross entropy (CC), logarithmic loss, exponential loss, hinge loss, etc.). If the loss is consistent, stagnating, or exceeds a threshold number of iterations, training may be terminated to save time and resources. According to some embodiments, DSQ may be used to fine-tune a pre-trained model to obtain better compression capabilities.

[0025] According to some embodiments, the learning rate or step size may be altered depending on the output of the loss function. For example, if the loss is changing gradually, the step size may be increased significantly. Conversely, if the loss is changing rapidly, the step size may be changed gradually.

[0026] The embodiments of the present disclosure relate to a model of end-to-end neural image compression that improves compression performance by optimizing image compression as a system as a whole. The embodiments of the present disclosure enable better image compression using a neural network-based image substitution method and dependent scalar quantization. The present disclosure provides novel mechanisms, methods, and apparatus for combining a neural network-based substitution image generation method / model with dependent scalar optimization for effective end-to-end neural image compression. According to some embodiments, the end-to-end neural network-based substitution and / or dependent scalar quantization model may be pre-trained and then fine-tuned, or trained and used for inference simultaneously. This fine-tuning or joint training and inference of the neural network increases processing efficiency and reduces overhead.

[0027] FIG. 1 is a diagram of an environment 100 in which the methods, apparatus, and systems described herein may be implemented, according to an embodiment.

[0028] 1, environment 100 may include a user device 110, a platform 120, and a network 130. The devices of environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0029] User device 110 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with platform 120. For example, user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a wearable device (e.g., smart glasses or a smart watch, etc.), or a similar device. In some implementations, user device 110 may receive information from platform 120 and / or transmit information to the platform.

[0030] Platform 120 includes one or more devices as described elsewhere herein. In some implementations, platform 120 may include a cloud server or a group of cloud servers. In some implementations, platform 120 may be designed to be modular such that software components can be swapped in or out. As such, platform 120 may be easily and / or quickly reconfigured for different uses.

[0031] In some implementations, as shown, platform 120 may be hosted within a cloud computing environment 122. Notably, although the implementations described herein describe platform 120 as being hosted within cloud computing environment 122, in some implementations platform 120 may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0032] Cloud computing environment 122 includes an environment that hosts platform 120. Cloud computing environment 122 can provide services, such as computation, software, data access, storage, etc., that do not require end-user (e.g., user device 110) knowledge of the physical location and configuration of the systems and / or devices that host platform 120. As shown, cloud computing environment 122 may include a group of computing resources 124 (collectively referred to as “computing resources 124” and individually referred to as “computing resource 124”).

[0033] The computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, the computing resources 124 may host the platform 120. Cloud resources may include computational instances running within the computing resources 124, storage devices provided within the computing resources 124, data transfer devices provided by the computing resources 124, etc. In some implementations, the computing resources 124 may communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0034] As further shown in FIG. 1, the computing resources 124 include a group of cloud resources, such as one or more applications (“APP”) 124-1, one or more virtual machines (“VM”) 124-2, virtualized storage (“VS”) 124-3, and one or more hypervisors (“HYP”) 124-4.

[0035] The application 124-1 includes one or more software applications that may be provided to or accessed by the user device 110 and / or platform 120. The application 124-1 may eliminate the need to install and run a software application on the user device 110. For example, the application 124-1 may include software associated with the platform 120 and / or any other software that may be provided via the cloud computing environment 122. In some implementations, one application 124-1 may send and receive information to one or more other applications 124-1 via the virtual machine 124-2.

[0036] Virtual machine 124-2 includes a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. Virtual machine 124-2 can be either a system virtual machine or a processing virtual machine, depending on the use by virtual machine 124-2 and the degree of correspondence with any real machine. A system virtual machine may provide a complete system platform that supports the execution of a complete operating system ("OS"). A processing virtual machine may execute a single program and support a single process. In some implementations, virtual machine 124-2 may execute on behalf of a user (e.g., user device 110) and further manage the infrastructure of cloud computing environment 122, such as data management, synchronization, or long-term data transfer.

[0037] Virtualized storage 124-3 includes one or more storage systems and / or one or more devices that use virtualization techniques within the storage systems or devices of computing resources 124. In some implementations, in the context of storage systems, types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage such that it may be accessed regardless of whether the storage system is physical or heterogeneous. The separation may provide flexibility in how storage system administrators manage end-user storage. File virtualization may eliminate the dependency between data accessed at the file level and where the file is physically stored. This may enable optimization of storage usage, server consolidation, and / or smooth file migration.

[0038] The hypervisor 124-4 may provide hardware virtualization techniques that allow multiple operating systems (e.g., "guest operating systems") to run simultaneously on a host computer, such as the computing resource 124. The hypervisor 124-4 may present a virtual operating platform to the guest operating systems and further manage the execution of the guest operating systems. Multiple instances of different operating systems may share virtualized hardware resources.

[0039] Network 130 may include one or more wired and / or wireless networks, such as, for example, a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public local mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, an optical fiber-based network, etc., and / or a combination of these or other types of networks.

[0040] The number and arrangement of devices and networks shown in Figure 1 are provided as an example. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different arrangements of devices and / or networks than those shown in Figure 1. Furthermore, two or more devices shown in Figure 1 may be implemented within a single device, or a single device shown in Figure 1 may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) of environment 100 may perform one or more functions described as being performed by other sets of devices of environment 100.

[0041] FIG. 2 is a block diagram of example components of one or more devices of FIG.

[0042] The device 200 may correspond to the user device 110 and / or the platform 120. As shown in FIG. 2, the device 200 may include a bus 210, a processor 220, a memory 230, a storage component 240, an input component 250, an output component 260, and a communication interface 270.

[0043] The bus 210 includes components that enable communication between the components of the device 200. The processor 220 is implemented in hardware, firmware, or a combination of hardware and software. The processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other type of processing component. In some implementations, the processor 220 includes one or more processors that can be programmed to perform functions. The memory 230 includes random access memory (RAM), read only memory (ROM), and / or other types of dynamic or static storage devices (e.g., flash memory, magnetic memory, and / or optical memory) that store information and / or instructions for use by the processor 220.

[0044] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include a hard disk (e.g., a magnetic disk, optical disk, magneto-optical disk, and / or solid state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, cartridges, magnetic tape, and / or other type of non-transitory computer-readable medium along with a corresponding drive.

[0045] Input components 250 include components that enable device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 250 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 260 include components that provide output information from device 200 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0046] Communications interface 270 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communications interface 270 may enable device 200 to receive information from and / or provide information to other devices. For example, communications interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0047] The device 200 may perform one or more processes described herein. The device 200 may perform these processes in response to the processor 220 executing software instructions stored by a non-transitory computer-readable medium, such as the memory 230 and / or the storage component 240. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[0048] Software instructions may be loaded into memory 230 and / or storage component 240 from other computer-readable media or from other devices via communications interface 270. The software instructions stored in memory 230 and / or storage component 240, when executed, may cause processor 220 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0049] The number and arrangement of components shown in Figure 2 are provided as an example. In practice, device 200 may include additional, fewer, different, or differently arranged components relative to those shown in Figure 2. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by other sets of components of device 200.

[0050] FIG. 4 illustrates an example block diagram 400 of an end-to-end neural image compression using dependent scalar quantization with replacement (E2E NIC) framework, according to an embodiment.

[0051] As seen in FIG. 4, the block diagram 400 may include an encoder 402, a DSQ 404, an entropy coder 406, an entropy decoder 408, a decoder 410, a replacement feature optimizer 403, a hyperencoder 452, a second DSQ 454, a second entropy coder 456, a second entropy decoder 458, a hyperdecoder 460, and a context model 420.

[0052] According to an embodiment of the present disclosure, the E2E NIC framework may utilize block diagram 400 as follows: Given an input image or video sequence x, a permutation feature optimizer 403 can generate a permutation image x′ based on the input x, and a neural network-based encoder 402 can generate a compressed representation x′ that is easier to store and transmit when compared to the input image x.

number

number

number

number

number

number

number

[0053] According to some embodiments, the E2E NIC may include a hyper-prior model and a context model during the online training phase to further improve the compression capability. The hyper-prior model may be used to capture spatial dependencies of the latent representations generated between layers of the neural network. According to some embodiments, side information may be used by the hyper-prior model, which is generally generated by motion-compensated temporal interpolation of neighboring reference frames at the decoder side. This side information may be used to train and infer the E2E NIC framework. The hyper-encoder 452 may encode the permuted image x′ using a hyper-prior neural network-based encoder. The second DSQ 454 and the second entropy coder 456 may then generate a hyper-compressed representation of the hyper-encoded permuted image. The second entropy decoder 458 may decode the hyper-compressed representation to generate a hyper-reconstructed image, which may then be generated using the hyper-prior neural network-based hyper-decoder 460. The neural network-based context model 420 may be trained using the hyper-reconstructed permuted image and the quantized representation from the DSQ 404. The entropy encoder 406 and the entropy decoder 408 may use the context model 420 for encoding and recoding, respectively.

[0054] 5A-5B show a flowchart of a method for end-to-end neural image encoding using dependent scalar quantization with replacement, according to an embodiment. Figure 5A shows a process 500 for encoding, and Figure 5B shows a process 550 for decoding.

[0055] At operation 505, the framework may receive an input image. According to some embodiments, the input image may be an image of any suitable format. In some embodiments, the input image may be a portion of a sequence of video frames. By way of example, at 505, the framework may receive one or more input images.

[0056] At operation 510, a replacement image may be determined and / or compressed based on the input image using a neural network-based image compression framework. As an example, the replacement feature optimizer may generate a replacement image x' for the input image x. At operation 515, the replacement image may be encoded using the neural network-based encoder 402. Operations 510 and 515 may be performed in any order. According to some embodiments, the encoder 402 may encode the replacement image generated by the replacement feature optimizer 403. In some embodiments, the order may be reversed.

[0057] In operation 520, a quantized representation may be generated based on the permuted image encoded using the first dependent scalar quantizer to obtain a quantized representation of the input image having a higher compression capability using the first dependent scalar quantizer. According to an embodiment, the dependent scalar quantization may include a first quantizer, a second quantizer, and a state machine, the state machine enabling switching between the first quantizer and the second quantizer.

[0058] In operation 525, the replacement image may be entropy coded using a neural network based encoder to generate a compressed representation of the quantized representation. According to an embodiment, the best compressed representation may be a global minimum of the trade-off between the rate-distortion loss between the input image and the reconstructed image and the bit consumption rate of the compressed representation. The entropy coding may convert the quantized representation into a bitstream for storage and transmission.

[0059] At the decoding side of the operation, the compressed representation may be received at operation 530. At operation 535, the compressed representation may be decoded using a neural network-based decoder and / or an entropy decoder. At operation 540, a reconstructed image may be generated based on the decoded compressed representation.

[0060] According to an embodiment, a neural network-based image compression (E2E NIC) framework can include a pre-trained model, where one or more model weights associated with the pre-trained model are fixed. The pre-trained model can be fine-tuned using a first dependent scalar quantizer.

[0061] According to some embodiments, the neural network-based image compression framework may include a model, and training the model may include initializing a learning rate of the model. As the training progresses, the learning rate of the model may be adjusted a threshold number of times, and the adjustment may be based on image characteristics of one or more training images. The training may terminate based on any of the following conditions: determining that a difference in the learning rate between successive iterations is below a learning threshold, determining that an output loss of the loss function is consistent over a first number of iterations, or determining that the learning rate has been adjusted over a maximum number of iterations. According to some embodiments, the adjustment of the learning rate may be inversely correlated to the output loss of the loss function.

[0062] According to some embodiments, operations 505-540 may be performed using a device configured to execute code, with each operation corresponding to code, such as a received code, a determined code, a generated code, etc.

[0063] The embodiments of the present disclosure also provide the flexibility to adjust the learning-based substitution, quantization, encoding, and decoding schemes online or offline based on current data, supporting various types of learning-based quantization schemes, including DNN-based or traditional model-based schemes. The described method also provides a flexible and general framework to accommodate different DNN architectures and multiple quality metrics.

[0064] The proposed methods may be used separately or combined in any order. Furthermore, each of the methods (or embodiments) may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits) or may be implemented using software code (e.g., generate code, receive code, encode code, decode code, etc.). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium.

[0065] This disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible in light of the present disclosure or may be acquired from practice of the implementations.

[0066] As used herein, the term component is intended to be interpreted broadly as hardware, firmware, or a combination of hardware and software.

[0067] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0068] Although combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features may be combined in ways not specifically recited in the claims and / or disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.

[0069] No element, act, or instruction used herein may be construed as critical or essential unless expressly described as such. Additionally, the articles "a" and "an" as used herein include one or more items and may be used interchangeably with "one or more." Additionally, the term "set" as used herein includes one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with "one or more." When only one item is addressed, the term "one" or similar language is used. Additionally, terms such as "has," "have," "having," and the like, as used herein, are intended to be open-ended terms. Additionally, the phrase "based on" is intended to mean "based at least in part on," unless otherwise specified. [Explanation of symbols]

[0070] 100 Environment 110 User Devices 120 Platform 122 Cloud Computing Environment 124 computing resources 124-1 Application 124-2 Virtual Machine 124-3 Virtualized Storage 124-4 Hypervisor 130 Network 200 devices 210 Bus 220 Processor 230 Memory 240 Memory Components 250 Input Components 260 Output Components 270 Communication Interface 402 Encoder 403 Replacement Feature Optimizer 404 DSQ 406 Entropy Encoder, Entropy Coder 408 Entropy Decoder 410 Decoder 420 Context Model 452 HyperEncoder 454 2nd DSQ 456 Second Entropy Coder 458 Second Entropy Decoder 460 Hyper Decoder 500 processes 550 Process

Claims

1. 1. A method for neural image compression using dependent scalar quantization with permutation, the method being executed by one or more processors, the method comprising: receiving an input image, the input image including features; determining a replacement image for the input image based on the input image using a neural network based replacement feature generator; encoding the replacement image to obtain an encoded replacement image, the encoded replacement image including encoding features; quantizing the encoded permuted image including the coding features to obtain a quantized representation of the input image using a first dependent scalar quantizer, the quantized representation being more compressible than the input image; and entropy encoding the permuted image using a neural network based encoder to generate a compressed representation of the quantized representation.

2. The method comprises: receiving the compressed representation; decoding the compressed representation using entropy decoding; and The method of claim 1 , further comprising: generating a reconstructed image based on the decoded compressed representation using a neural network based decoder.

3. The method of claim 2 , wherein the best compressed representation is a global minimum of a trade-off between the rate-distortion loss between the input image and the reconstructed image and the bit consumption rate of the compressed representation.

4. The step of generating the condensed representation comprises: hyper-encoding the encoded permuted image using a hyper-a priori neural network based encoder; generating a hyper-compressed representation of the hyper-encoded encoded permuted image using a second dependent scalar quantizer and entropy coding; hyper-decoding the hyper-compressed representation to generate a hyper-reconstructed image using a hyper-a priori neural network based decoder; training a contextual neural network model based on the hyper-reconstructed image and the quantized representation; and generating the compressed representation of the quantized representation using entropy coding and the context neural network model; 2. The method of claim 1, comprising:

5. 2. The method of claim 1, wherein the step of quantizing includes a first quantizer, a second quantizer, and a state machine, the state machine enabling switching between the first quantizer and the second quantizer.

6. 2. The method of claim 1, wherein the neural image compression includes a pre-trained model, and one or more model weights associated with the pre-trained model are fixed.

7. The method of claim 6 , wherein the pre-trained model is fine-tuned using the first dependent scalar quantizer.

8. The neural image compression includes a trained model, and training the trained model includes: initializing a learning rate of the trained model; adjusting the learning rate of the trained model a threshold number of times, the adjusting being based on image characteristics of one or more training images; and The training, determining that a difference in the learning rate between successive iterations is below a learning threshold; determining that the output loss of the loss function is consistent over the first number of iterations; or determining that the learning rate has been adjusted a maximum number of iterations; The method of claim 1 , further comprising terminating based on at least one of:

9. 9. The method of claim 8, wherein the step of adjusting the learning rate is based inversely proportional to the output loss of the loss function.

10. Apparatus arranged to perform the method according to any one of claims 1 to 9.

11. A computer program for causing a computer to carry out the method according to any one of claims 1 to 9.

Images