Adaptive neural network image compression method, device, storage medium and equipment

Abstract

The present application discloses a meta-learning-based adaptive neural network image compression method, apparatus, storage medium, and computer device. The method comprises: using an original input image and target quality control parameters to generate a substitute input image and substitute target quality control parameters, wherein the substitute input image is a modified version of the input image and the substitute target quality control parameters are modified versions of the target quality control parameters; and using an encoding neural network to encode the substitute input image based on the substitute target quality control parameters to generate a compressed representation of the substitute input image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 176,745, filed with the U.S. Patent Office on April 19, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The embodiments of the present application relate to data processing technology, and in particular to a meta-learning-based adaptive neural network image compression method, device, storage medium and computer equipment. Background Art

[0004] ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) has been actively exploring potential needs for standardization of future video coding technologies. ISO / IEC JPEG has established the JPEG-AI group to focus on AI-based end-to-end neural network image compression (NIC) using neural networks (NN). Recent successful approaches to advanced neural network image and video compression are generating increasing industry interest.

[0005] Although some methods have shown improved performance, flexible bitrate control remains a challenging problem for previous NIC methods. For example, some methods may need to train multiple model instances with the goal of obtaining each desired trade-off between rate and distortion (the quality of the compressed image). All these multiple model instances can be stored and deployed at the decoder side to reconstruct the image according to different bitrates. In addition, these model instances cannot give arbitrarily smooth bitrate control because it is difficult to train and store an infinite number of model instances for each possible target bitrate. Some methods have studied multi-rate NIC, in which one model instance is trained to achieve compression at multiple predetermined bitrates. However, arbitrarily smooth bitrate control remains an unexplored open problem.

[0006] Furthermore, flexible target quality metric control is difficult for previous NIC methods because it requires training a separate model instance for each target quality metric (e.g., peak signal-to-noise ratio (PSNR), structural similarity metric (SSIM), combination of PSNR and SSIM, etc.). Smooth quality metric control (e.g., a weighted combination of PSNR and SSIM with arbitrary importance weights) remains an open problem. Summary of the Invention

[0007] The embodiments of the present application relate to a meta-learning-based adaptive neural network image compression method, apparatus, storage medium, and computer equipment, which can achieve image compression with arbitrary smooth target quality control (including smooth bit rate control, smooth quality metric control, etc.).

[0008] According to an embodiment of the present application, a meta-learning-based adaptive neural network image compression method is provided, comprising:

[0009] generating a substitute input image and substitute target quality control parameters using an original input image and target quality control parameters, wherein the substitute input image is a modified version of the input image and the substitute target quality control parameters are modified versions of the target quality control parameters; and

[0010] The substitute input image is encoded based on the substitute target quality control parameter using an encoding neural network to generate a compressed representation of the substitute input image.

[0011] According to an embodiment of the present application, a non-volatile computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by at least one processor for meta-learning-based adaptive neural network image compression, the at least one processor executes the above method.

[0012] According to an embodiment of the present application, a computer device is further provided, including a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above method.

[0013] According to an embodiment of the present application, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described method.

[0014] As can be seen from the above technical solutions, the method provided by the embodiments of the present invention achieves better compression by using a substitute input image instead of the original input image. In addition, by using a substitute target quality control parameter, the quality adaptation weight parameter can be improved, thereby enabling the substitute input image to be compressed better for the target compression quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram showing an environment for implementing the methods, devices, and systems described herein according to an embodiment of the present application is shown;

[0016] Figure 2 Shown Figure 1a block diagram of example components of at least one computer device;

[0017] Figure 3A and Figure 3B FIG4 shows a block diagram of a meta-NIC structure of an adaptive neural network image compression with smoothing quality control according to an embodiment of the present application;

[0018] Figure 4A FIG2 shows a block diagram of an adaptive neural network image compression device with smoothing quality control according to an embodiment of the present application;

[0019] Figure 4B The corresponding embodiment of the present application is shown Figure 4A A block diagram of the metaNIC encoder of the device shown;

[0020] Figure 4C The corresponding embodiment of the present application is shown Figure 4A a block diagram of the metaNIC decoder of the illustrated apparatus;

[0021] Figure 4D A block diagram of an apparatus for decoding an image after adaptive neural network image compression encoding with smoothing quality control according to an embodiment of the present application is shown;

[0022] Figure 5 A block diagram of a training apparatus for adaptive neural network image compression with smooth quality control during a training phase according to an embodiment of the present application is shown;

[0023] Figure 6 A flowchart of an adaptive neural network image compression method with smoothing quality control according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] The present application describes methods and apparatus for adaptively finding and / or generating alternative input images and alternative target quality control parameters for each image in a MetaNeural Image Compression (MetaNIC) framework to generate an optimal alternative input image and optimal alternative quality control parameters.

[0025] The meta-NIC system can adaptively calculate the quality-adaptive weight parameters of the underlying neural network image compression (NIC) model based on the current alternative input image and the target compression quality, so that a single meta-NIC model instance can achieve image compression with arbitrary smooth target quality control (including smooth bitrate control, smooth quality metric control, etc.).

[0026] The surrogate input image generated and used in the meta-NIC system is a superior variation of the original input image, resulting in better compression. Additionally, the generated surrogate target quality control parameters refine the calculated quality adaptation weight parameters to better compress the generated surrogate input image for the target compression quality. The decoder uses the learned surrogate target quality parameters to reconstruct the original input image from the encoded bitstream. Figure 1 is a schematic diagram of an environment for implementing the methods, devices, and systems described herein according to an embodiment of the present application.

[0027] like Figure 1 As shown, environment 100 may include user device 110, platform 120, and network 130. The devices of environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0028] User device 110 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with platform 120. For example, user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch), or the like. In some implementations, user device 110 may receive information from platform 120 and / or transmit information to platform 120.

[0029] The platform 120 includes one or more devices capable of generating audio output signals via a multi-band synchronized neural vocoder, as described elsewhere herein. In some implementations, the platform 120 may include a cloud server or a group of cloud servers. In some implementations, the platform 120 may be designed to be modular so that certain software components can be swapped in or out based on specific needs. In this way, the platform 120 can be easily and / or quickly reconfigured for different uses.

[0030] In some implementations, as shown, the platform 120 can be hosted in a cloud computing environment 122. It is worth noting that although the implementations described herein describe the platform 120 as being hosted in a cloud computing environment 122, in some implementations, the platform 120 is not cloud-based (i.e., can be implemented outside of a cloud computing environment) or can be partially cloud-based.

[0031] The cloud computing environment 122 includes an environment that hosts the platform 120. The cloud computing environment 122 can provide computing, software, data access, storage, and other services that do not require an end user (e.g., user device 110) to be aware of the physical location and configuration of one or more systems and / or devices hosting the platform 120. As shown, the cloud computing environment 122 can include a set of computing resources 124 (collectively, "computing resources 124" and individually, "computing resource 124").

[0032] Computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, computing resources 124 may be hosting platforms 120. Cloud resources may include computing instances executed in computing resources 124, storage devices provided in computing resources 124, data transfer devices provided by computing resources 124, and the like. In some implementations, computing resources 124 may communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0033] like Figure 1 As further shown in FIG, the computing resources 124 include a set of cloud resources, such as one or more application programs ("APP") 124-1, one or more virtual machines ("VM") 124-2, virtualized storage ("VS") 124-3, one or more hypervisors ("HYP") 124-4, etc.

[0034] Applications 124-1 include one or more software applications that can be provided to or accessed by user device 110 and / or sensor device 120. Applications 124-1 can eliminate the need to install and execute software applications on user device 110. For example, applications 124-1 can include software associated with platform 120 and / or any other software that can be provided via cloud computing environment 122. In some implementations, one application 124-1 can send and receive information to and from one or more other applications 124-1 via virtual machine 124-2.

[0035] Virtual machine 124-2 comprises a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. Virtual machine 124-2 can be a system virtual machine or a process virtual machine, depending on the use of virtual machine 124-2 and the degree of correspondence with any real machine. A system virtual machine can provide a complete system platform that supports the execution of a complete operating system ("OS"). A process virtual machine can execute a single program and can support a single process. In some implementations, virtual machine 124-2 can execute on behalf of a user (e.g., user device 110) and can manage the infrastructure of cloud computing environment 122, such as data management, synchronization, or long-duration data transfer.

[0036] Virtualized storage 124-3 includes one or more storage systems and / or one or more devices that use virtualization technology within the storage system or device of the computing resource 124. In some implementations, in the context of a storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage, such that the storage system may be accessed without regard to physical storage or heterogeneous structures. Separation may allow administrators of the storage system flexibility in how the administrator manages storage for end users. File virtualization may eliminate the dependency between data accessed at the file level and where the file is physically stored. This may enable optimization of storage usage, server consolidation, and / or performance of non-disruptive file migration.

[0037] Hypervisor 124-4 can provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to execute concurrently on a host computer such as computing resource 124. Hypervisor 124-4 can present a virtual operating platform to the guest operating systems and can manage the execution of the guest operating systems. Multiple instances of various operating systems can share virtualized hardware resources.

[0038] The network 130 includes one or more wired and / or wireless networks. For example, the network 130 may include a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-optic-based network, etc., and / or a combination of these or other types of networks.

[0039] Figure 1 The number and arrangement of devices and networks shown in FIG are provided as examples. In practice, there may be more than Figure 1The devices and / or networks shown may include more devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks. Figure 1 Two or more of the devices shown may be implemented in a single device, or Figure 1 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one set of devices (eg, one or more devices) of environment 100 may perform one or more functions described as being performed by another set of devices of environment 100.

[0040] Figure 2 Shown Figure 1 A block diagram of example components of at least one computer device.

[0041] Device 200 may correspond to user device 110 and / or platform 120. Figure 2 As shown, device 200 may include a bus 210 , a processor 220 , a memory 230 , a storage component 240 , an input component 250 , an output component 260 , and a communication interface 270 .

[0042] The bus 210 includes components that allow communication between components of the device 200. The processor 220 is implemented in hardware, firmware, or a combination of hardware and software. The processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 220 includes one or more processors that can be programmed to perform functions. The memory 230 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 220.

[0043] The storage component 240 stores information and / or software related to the operation and use of the device 200. For example, the storage component 240 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or other types of non-volatile computer-readable storage media, and corresponding drives.

[0044] Input components 250 include components that allow device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 250 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 260 include components that provide output information from device 200 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0045] The communication interface 270 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 270 can allow the device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 270 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0046] Device 200 can perform one or more of the processes described herein. Device 200 can perform these processes in response to processor 220 executing software instructions stored by a non-volatile computer-readable storage medium, such as memory 230 and / or storage component 240. Computer-readable storage media is defined herein as a non-volatile memory device. A memory device includes memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

[0047] The software instructions may be read into the memory 230 and / or storage component 240 from another computer-readable storage medium or from another device via the communication interface 270. When executed, the software instructions stored in the memory 230 and / or storage component 240 may cause the processor 220 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0048] Figure 2 The number and arrangement of components shown in FIG are provided as examples. In practice, the device 200 may include Figure 2 The components shown may be more components, fewer components, different components, or components arranged differently. Additionally or alternatively, one set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0049] The method and apparatus for adaptive neural network image compression with smooth quality control via meta-learning are described in detail below.

[0050] Embodiments of the present application describe methods and apparatus for an alternative meta-NIC framework that, on the one hand, supports arbitrary smooth quality control (including bitrate control, quality metric control, etc.) and simultaneously adaptively finds the best alternative input image and the best alternative quality control parameters for each input image.

[0051] The meta-NIC system can adaptively calculate the quality adaptation weight parameters of the underlying NIC model based on the current alternative input image and the target compression quality, so that a single meta-NIC model instance can achieve image compression with arbitrary smooth target quality control (including smooth bitrate control, smooth quality metric control, etc.).

[0052] Given an input image x of size (h, w, c), where h, w, c are the height, width, and number of channels, respectively, the NIC workflow can be described as follows. The input image x can be a regular image frame (t=1), a 4D video sequence containing more than one image frame (t>1), etc. Each image frame can be a color image (c=3), a grayscale image (c=1), an RGB+depth image (c=4), etc. Compute the compressed representation This compressed representation It is compact for storage and transmission. Then, based on the compressed representation Reconstruct the output image And the reconstructed output image Can be similar to the original input image x. Distortion loss Used to measure reconstruction error, such as peak signal-to-noise ratio (PSNR) or structural similarity index measure (SSIM). Calculation rate loss Expressed as measured compression The trade-off hyperparameter λ is used to form the joint rate-distortion (RD) loss:

[0053]

[0054] Using a large hyperparameter λ during training will result in a compressed model with less distortion but more bits consumed, and vice versa. Traditionally, for each predefined hyperparameter λ, a NIC model instance is trained, however, the NIC model instance does not work well for other values ​​of the hyperparameter λ. Therefore, in order to achieve multiple bit rates for the compressed stream, traditional methods may need to train and store multiple model instances. In addition, because it is difficult to train a model for every possible value of the hyperparameter λ in practice, traditional methods cannot achieve arbitrary smooth quality control such as smooth bitrate control. Similarly, traditional methods may need to train and store multiple model instances for each distortion metric or other metric, which may make it difficult to achieve smooth quality metric control.

[0055] Figure 3A and Figure 3B is a block diagram of meta-NIC architectures 300A and 300B for adaptive neural network image compression with smoothing quality control via meta-learning, according to an embodiment.

[0056] like Figure 3A As shown in , the metaNIC architecture 300A may include a shared encoding NN 305 , an adaptive encoding NN 310 , a shared decoding NN 315 , and an adaptive decoding NN 320 .

[0057] like Figure 3B As shown in , the metaNIC architecture 300B may include shared encoding layers 325 and 330 , adaptive encoding layers 335 and 340 , shared decoding layers 345 and 350 , and adaptive decoding layers 355 and 360 .

[0058] In the embodiment of the present application, the model parameters of the bottom NIC encoder and bottom NIC decoder can be divided into 4 parts: They represent shared encoding parameters (SEP), adaptive encoding parameters (AEP), shared decoding parameters (SDP), and adaptive decoding parameters (ADP), respectively. Figure 3A and Figure 3B Two embodiments of NIC network architectures are shown.

[0059] exist Figure 3A In , SEP, SDP, AEP and ADP are separate NN modules, and these separate modules are connected in sequence for network forward computation. Figure 3A The order in which these individual NN modules are connected is shown. Other orders can be used here.

[0060] exist Figure 3B In , parameter segmentation can be performed within the NN layer. Let Denote the SEP, AEP, SDP, and ADP of the i-th layer of the NIC encoder and the j-th layer of the NIC decoder, respectively. The NIC can calculate inference outputs based on the corresponding inputs of the SEP and AEP (or SDP and ADP), respectively, and these outputs can be combined (e.g., by addition, concatenation, multiplication, etc.) and then sent to the next layer.

[0061] Figure 3A An embodiment can be viewed as Figure 3B A situation in which For the shared encoding, the layers in NN 305 can be empty. For adaptive coding, the layers in NN 310 can be empty. For shared decoding, the layers in NN315 can be empty. For adaptive decoding, the layers in NN 320 may be empty. Therefore, in other embodiments, the layers may be combined Figure 3A and Figure 3B A network structure, wherein the NIC architecture includes pure shared encoding / decoding layers, and / or pure adaptive encoding / decoding layers, and hybrid layers with partially shared encoding / decoding parameters and partially adaptive encoding / decoding parameters.

[0062] like Figure 4A As shown in , the apparatus 400A includes a meta-NIC encoder 410 , a meta-NIC decoder 420 , a loss generator 405 , a back-propagation module 415 , a quantization and entropy encoder 425 , and a quantization and rate estimator 430 .

[0063] Figure 4A The overall workflow of the testing phase of the meta-NIC framework is shown in FIG. and denote the SEP and AEP of the i-th layer of the meta NIC encoder 410, respectively. These are exemplary notations, because according to some embodiments, for fully shared layers, is empty. In addition, for fully adaptive layers, is empty. Similarly, let and denote the SDP and ADP of the jth layer of the meta NIC decoder 420, respectively. These notations are also exemplary, because according to some embodiments, for a fully shared layer, is empty. In addition, for fully adaptive layers, In some embodiments, these symbols can be used to Figure 3A and Figure 3B Two embodiments of the present invention.

[0064] Given a surrogate input image x', and given a surrogate target quality control parameter Λ', the meta-NIC encoder 410 computes the compressed representation In some embodiments, the substitute input image x' may be a modified version of the original input image x. The substitute target quality control parameter Λ' may be a modified version of the original target quality control parameter Λ, which indicates a target compression quality, including a target quality metric, a target bit rate, etc. Both the substitute input image x' and the substitute target quality control parameter Λ' may be obtained through an iterative online learning process based on the original input image x and the original target quality control parameter Λ.

[0065] For the target quality control parameter Λ, let q be the quality metric … The quality metrics are usually expressed as a weighted combination of multiple quality metrics (such as PSNR, SSIM, etc.):

[0066]

[0067] Among them, the weight w i ≥0. In some embodiments, the target quality control parameter Λ may be a parameter including all weights w i and a single vector Λ=[w1,…w q ,λ]. When only a single quality metric is used When, for any i≠j, w i =1 and w j = 0. In some embodiments, Λ can be simplified to include only the target tradeoff hyperparameter λ: Λ = λ.

[0068] Figure 4B yes Figure 4A 4. A block diagram of a meta-NIC encoder 410 of an apparatus 400A is shown in FIG.

[0069] like Figure 4B As shown in FIG, the meta-NIC encoder 410 includes a SEP inference portion 412, an AEP prediction portion 414, and an AEP inference portion 416.

[0070] Figure 4B An example embodiment of the inference workflow of the meta-NIC encoder 410 of the i-th layer is given. In the meta-NIC encoder 410, the substitute input image x' is passed through the meta-NIC encoder NN. Let f(i) and f(i+1) denote the input tensor and output tensor of the i-th layer. Based on the current input f(i) (based on the substitute input image) and The SEP inference section 412 is based on a shared inference function To calculate one or more shared features g(i), the shared inference function can be modeled by using the forward calculation of the SEP in the i-th layer. Based on the current input f(i), shared features g(i), and the alternative target quality control parameter Λ', the AEP prediction section 414 calculates the estimate of the i-th layer The AEP prediction part 414 can be, for example, a NN including convolutional and fully connected layers, which is based on the original The current input f(i) and the alternative target quality control parameter Λ' are used to predict the updated estimate In some embodiments, the current input f(i) may be used as input to the AEP prediction portion 414. In some other embodiments, the shared feature g(i) may be used instead of the current input f(i). In other embodiments, the SEP loss may be calculated based on the shared feature g(i), and the gradient of the loss may be used as input to the AEP prediction portion 414. and shared features g(i), the AEP inference section 416 is based on the AEP inference function To compute the output tensor f(i+1), the AEP inference function is modeled by the forward computation using the estimated AEP in the i-th layer.

[0071] Notice, Figure 4B The workflow described in is an example notation. For layers fully shared with AEP, Can be empty, the modules related to AEP can be omitted, and f(i+1)=g(i). For layers fully adapted to SEP, It can be empty, the modules related to SEP can be omitted, and g(i)=f(i).

[0072] In the embodiment where the meta-NIC encoder 410 has a total of N layers, the output of the last layer is the compressed representation Compressed Representation can be sent to the meta NIC decoder 420. In some embodiments, the quantization and entropy encoder 425 compresses the representation After further compression into a compact code stream z, the compact code stream z is sent to the meta NIC decoder 420 .

[0073] Reference again Figure 4A , at the decoder side, let and Denote the SDP and ADP for the jth layer of the meta NIC decoder 420, respectively. Similar to the meta NIC encoder 410, this is an exemplary representation because for fully shared layers, is empty, and for fully adaptive layers, It's empty.

[0074] At the decoder side, dequantization and entropy decoding can be used to obtain the recovered compressed representation from the code stream z sent from the meta-NIC encoder 410 In some embodiments, the compressed representation The quantization and rate estimator 430 can be used to generate the recovered compressed representation

[0075] Recovery-based compressed representation and the alternative target quality control parameter Λ', the meta-NIC decoder 420 computes the reconstructed output image In the meta-NIC decoder 420, the recovered compressed representation Decode the NN via the meta-NIC. Let f(j) and f(j+1) denote the input and output tensors of the jth layer.

[0076] Figure 4C yes Figure 4A 4. A block diagram of the meta NIC decoder 420 of the apparatus 400A is shown in FIG.

[0077] like Figure 4C As shown in FIG, the meta NIC decoder 420 includes an SDP inference portion 422, an ADP prediction portion 424, and an ADP inference portion 426.

[0078] Figure 4C An example embodiment of an inference workflow for the metaNIC decoder 420 of layer j is given. Based on the current input f(j) (in some embodiments, based on the recovered compressed representation) and The SDP inference section 422 is based on the shared inference function To calculate one or more shared features g(j), the shared inference function is modeled by the network forward calculation using the SDP of the jth layer. Based on the current input f(j), the shared features g(j), and the alternative target quality control parameter Λ', the ADP prediction section 424 calculates the estimate of the jth layer The ADP prediction part 424 can be, for example, a NN with convolutional and fully connected layers, which is based on the original The current input f(j) and the alternative target quality control parameter Λ' are used to predict the updated estimate of In some embodiments, the current input f(j) may be used as an input to the ADP prediction portion 424. In some other embodiments, the shared feature g(j) may be used instead of the current input f(j). In other embodiments, the SDP loss may be calculated based on the shared feature g(j), and the gradient of the loss may be used as an input to the ADP prediction portion 424. and shared features g(j), the ADP inference section 426 is based on the ADP inference function To compute the output tensor f(j+1), the ADP inference function is modeled by the network forward computation using the ADP estimated in the jth layer.

[0079] Notice, Figure 4C The workflow described in is an exemplary notation. For layers fully shared with ADP, Can be empty, and the modules related to ADP can be omitted, and f(j+1)=g(j). For layers fully adaptive to SDP, It can be empty, and the SDP-related modules can be omitted, and g(j)=f(j).

[0080] In the embodiment where there are a total of M layers for the meta-NIC decoder 420, the output of the last layer is the reconstructed image output

[0081] In the above embodiments, the meta-NIC framework allows for arbitrary target quality control parameters Λ and / or alternative target quality control parameters Λ', and the processing workflow will compute the compressed representation and reconstructed output image to be compatible with the target quality control parameters Λ.

[0082] In some embodiments, the target quality control parameter Λ and / or the alternative target quality control parameter Λ' are the same for the encoder and decoder. In some other embodiments, the target quality control parameter Λ and / or the alternative target quality control parameter Λ' may be different for the meta-NIC encoder 410 and the meta-NIC decoder 420. In some embodiments, the target quality control parameter Λ and / or the alternative target quality control parameter Λ' differ between the meta-NIC encoder 410 and the meta-NIC decoder 420, and the meta-NIC decoder 420 attempts to adapt the compressed representation to a target quality that differs from the original encoding target quality.

[0083] According to an embodiment of the present application, when the AEP prediction part 414 and the ADP prediction part 424 perform prediction only on a predefined set of trade-off hyperparameters with or without considering the input f(i) or f(j), the meta-NIC model is simplified to a multi-rate NIC model that uses one model instance to adapt to the compression effects of multiple predefined bit rates.

[0084] Return Reference Figure 4A, the quantization and rate estimator 430 can simulate the real quantization and entropy encoding / decoding process by using a differentiable statistic sampler. In some embodiments, the quantization and rate estimator 430 can be part of the underlying NIC framework / architecture. In other words, when the underlying NIC architecture includes a quantization and rate estimator, the quantization and rate estimator 430 can correspond to the quantization and rate estimator of the underlying NIC architecture. The quantization and rate estimator 430 can be used to determine the recovered compressed representation and the estimated rate loss

[0085] Based on the reconstructed image and the original input image x, the loss generator 405 can calculate the distortion loss based on equation (2) The distortion loss can then be compared with the rate loss Combining the original target quality control parameter Λ, the total rate-distortion loss can be determined according to equation (1): According to an embodiment of the present application, the loss generator 405 may use or consider other adversarial or regularized losses, such as On or in compressed representation Any additional regularization term on .

[0086] Total rate distortion loss The back propagation module 415 can calculate the gradient of the loss Then the gradient of the loss Backward propagation to update the alternative input image x′ and the alternative target quality control parameter Λ′. For the next iteration in multiple online learning iterations, the loss is repeatedly calculated and the back-propagated loss gradient In some embodiments, the substitute input image x′ may be initialized to the original input image x, and the substitute target quality control parameter Λ′ may be initialized to the original target quality control parameter Λ′. The substitute input image x′ and the substitute target quality control parameter Λ′ are then updated through online iteration.

[0087] After T iterations, the online learning process is completed and the final alternative input image x′ and the final alternative target quality control parameter Λ′ are obtained. In some embodiments, T may refer to the maximum number of iterations. In some embodiments, the online learning process may be terminated when the change in the alternative input image x′ and the alternative target quality control parameter Λ′ is less than a preset threshold. The final alternative input image x′ and the final alternative target quality control parameter Λ′ are then passed through the meta NIC encoder 410 to generate a compressed representation Compressed Representation The compressed code stream z can be generated by quantization and entropy encoder 425. Quantization and entropy encoder 425 can compress the representation Performs actual quantization and entropy coding.

[0088] In some embodiments, the compressed code stream z and the final substitute target quality control parameter Λ′ may be sent to the meta NIC decoder 420. Furthermore, in some embodiments, the final substitute target quality control parameter Λ′ may be further encoded in a lossless manner before transmission.

[0089] Figure 4D is based on Figure 4A A partial view of an apparatus for decoding an image encoded by adaptive neural network image compression with smoothing quality control, according to an embodiment of apparatus 400A shown in FIG.

[0090] like Figure 4D As shown in , according to some embodiments, decoding an image encoded using adaptive neural network image compression with smoothing quality control includes: a meta-NIC decoder 420 and an inverse quantization and entropy decoder 428.

[0091] Figure 4D An example embodiment of the decoding workflow of the meta-NIC framework is given. At the decoder side, a compressed code stream z and a final alternative target quality control parameter Λ′ or a lossless compressed version of the alternative target quality control parameter Λ′ can be received. The dequantization and entropy decoder 428 uses the compressed code stream z to generate a recovered compressed representation In an embodiment, upon receiving the lossless compressed version of the surrogate target quality control parameter Λ′, the inverse quantization and entropy decoder 428 may recover the surrogate target quality control parameter Λ′. The meta-NIC decoder 420 then calculates the compressed representation based on the recovered The reconstructed image Alternative target quality control parameters Λ′, and Where j = 1, 2,…, M.

[0092] According to an embodiment of the present application, when the AEP prediction part 424 and the ADP prediction part 424 perform predictions only on a predefined set of tradeoff hyperparameters with or without considering the input f(i) or f(j), for a predefined set of combination weights (e.g., for the distortion metric Only w i =1and w j = 0 for i≠j), the meta-NIC model is simplified to the distortion measure A multi-rate NIC model that uses one model instance to adapt to the compression effects of multiple predefined bit rates.

[0093] According to some embodiments of the present application, the model parameters for the meta-NIC encoder 410 may be pre-trained and fixed during encoding and decoding, i.e. and (where i=1, ..., N), for the meta NIC decoder 420 and (where j = 1, ..., M), AEP predicts NN (denoted as Φ e Model parameters) and ADP prediction NN (denoted as Φ d ) model parameters), such as Figures 4A to 4D As described in .

[0094] Figure 5 is a block diagram of a training apparatus 500 for implementing adaptive neural network image compression with smooth quality control through meta-learning during a training phase according to an embodiment.

[0095] like Figure 5 As shown in , the training apparatus 500 includes a task sampler 510 , an inner loop loss generator 520 , an inner loop update part 530 , a meta-loss generator 540 , a meta-update part 550 , and a weight update part 560 .

[0096] The training process aims to learn Figure 4A MetaNIC Encoder 410 and as well as Figure 4A The MetaNIC decoder 420 and and AEP prediction NN (denoted as Φ e Model parameters) and ADP prediction NN (denoted as Φ d ) model parameters).

[0097] In an embodiment, a Model-Agnostic Meta-Learning (MAML) mechanism is used for training purposes. Figure 5 An example workflow for the meta-training framework is given. Other meta-training algorithms can be used here.

[0098] For training, there is a training data set Among them, each and the training trade-off hyperparameter Λ i Correspondingly, there are a total of K training quality control parameters (and therefore K training data sets). For training, there can be a finite set of weight combinations. For each weight combination w1=a1,…w q =a q , there can be k compromise hyperparameters λ iThe set consists of, where the quality control parameter Λ i Specify the values ​​of each item in the quality control parameter Λ, w1=a1,…w q =a q ,λ=λ i . In addition, there is verification data A collection of , where each and verify the quality control parameter Λ j Correspondingly, there are a total of P confirmation quality control parameters Λ j The validation quality control parameters may include values ​​that are different from those in the training set. The values ​​of the validation quality control parameters may also be the same as those from the training set.

[0099] The overall training goal is to learn a meta-NIC model that is broadly applicable to all (both training and future unseen) values ​​of the quality control parameters that correspond to a wide range of smoothness for the target compression quality. Assume that the NIC tasks with the target quality control parameters are drawn from a task distribution P(Λ). To achieve the above-mentioned training goal, the loss of the learned meta-NIC model is minimized over all training datasets for all training quality control parameters.

[0100] make Including all shared parameters in SEP and SDP, let Including all adaptive parameters in AEP and ADP. The MAML training process can have an outer loop and an inner loop for gradient-based parameter updates. For each outer loop iteration, the task sampler 510 first samples K ′ A set of training quality control parameters (K′≤K). Then, for each sampled training quality control parameter Λ i , the task sampler 510 is trained from the data set The sampled training data set In addition, the task sampler 510 samples a set of P′ (P′≤P) verification quality control parameters, and for each sampled verification quality control parameter Λ j , from the validation data set mid-sample validation data set Then, for each sample data Based on the current parameter Θ s 、Θ a , Φ e and Φ d Perform the meta-NIC forward calculation, and then the inner loop loss generator 520 calculates the accumulated inner loop loss

[0101]

[0102] Loss function L(x,Θ) s ,Θ a ,Φ e ,Φ d ,Λ i ) can include the RD loss of Equation (1) and another regularization loss (e.g., auxiliary loss for distinguishing the intermediate network outputs with different quality control parameters as targets). Then, based on the inner loop loss Given a step size α si and α ai As Λ i The quality control parameters / hyperparameters, the inner loop update part 530 calculates the updated task specific parameter updates:

[0103]

[0104] Accumulated internal circulation loss Gradient and gradient Can be used to calculate adaptive parameters and An updated version of .

[0105] Then, the meta-loss generator 540 calculates the outer meta-objective or loss for all sampled validation quality control parameters:

[0106]

[0107] in, Based on the use of parameters Φ s ,Φ a The meta-NIC forward calculation of , the loss calculated for the input x. Given a step size β aj and β sj As Λ j The meta-update part 550 updates the model parameters as follows:

[0108]

[0109] In some embodiments, Θ may not be updated in the inner loop. s , α si =0, Non-updates help stabilize the training process.

[0110] As for the parameters Φ of AEP prediction NN and ADP prediction NN e ,Φ d , the weight updating part 560 updates them using a regular training method. That is, according to the training and validation data Based on the current parameter Θ s、Θ a , Φ e ,Φ d , calculate all samples The loss L(x,Θ s ,Θ a ,Φ e ,Φ d ,Λ i ) and all samples The loss L(x,Θ s ,Θ a ,Φ e ,Φ d ,Λ j ). The gradients of all these losses can be accumulated (e.g., added) to be propagated back through Φ e ,Φ d Perform parameter updates on .

[0111] The embodiments of the present application are not limited to the above-mentioned optimization algorithms or loss functions for updating these model parameters. Any optimization algorithm or loss function known in the art for updating these model parameters can be used.

[0112] for Figure 4B In the case where the AEP prediction portion 414 of the meta-NIC model and the ADP prediction portion 424 of the meta-NIC model perform predictions only for a predefined set of training quality control parameters and / or a predefined set of metric combination weights, the verification quality control parameters can be the same as the training quality control parameters. The same MAML training procedure can be used to train the simplified meta-NIC model mentioned above (i.e., a multi-rate NIC model that uses a single model instance to accommodate the compression effects of multiple predetermined bit rates).

[0113] Embodiments of the present application allow the use of advanced alternative input images and advanced alternative target quality control parameters as altered versions of the original input images to adaptively improve the compression performance of each input data. The embodiments described herein enable image compression with arbitrary smooth quality control using meta-learning using only one meta NIC model instance. The methods and apparatus can be used for both multi-rate compression and smooth bitrate control with a single model. The embodiments described herein can be used for multi-metric and multi-rate compression, smooth bitrate control, and smooth quality metric control with a single model. Embodiments of the present application provide a flexible framework that adapts to a variety of underlying NIC models and meta-learning methods.

[0114] Figure 6 is a flow chart of a method for adaptive neural network image compression with smooth quality control through meta-learning, according to an embodiment.

[0115] In some implementations, Figure 6One or more process blocks of may be performed by platform 120. In some implementations, Figure 6 One or more process blocks of may be performed by another device or group of devices, such as user device 110, that is separate from or includes platform 120.

[0116] like Figure 6 As shown in FIG, in operation 610, method 600 includes receiving an original input image and target quality control parameters.

[0117] In operation 620, method 600 includes generating a substitute input image and substitute target quality control parameters using the original input image and the target quality control parameters. The substitute input image is a modified version of the input image, and the substitute target quality control parameters are modified versions of the target quality control parameters.

[0118] Generating the substitute input image includes performing a plurality of iterations, wherein performing each iteration includes: computing a rate-distortion loss using the target quality control parameters; computing a gradient of the rate-distortion loss using backpropagation; and updating the substitute input image and the substitute target quality control parameters based on the gradient of the rate-distortion loss. In some embodiments, performing a first iteration from the plurality of iterations may include: initializing the substitute input image to the input image; and initializing the substitute target quality control parameters to the target quality control parameters.

[0119] In some embodiments, the number of iterations may be a preset or predefined number.

[0120] At operation 630 , method 600 includes encoding the substitute input image based on the substitute target quality control parameters using an encoding neural network to generate a compressed representation.

[0121] In some embodiments, at operation 630 , an encoding neural network may be used to generate a compressed representation of the substitute input image based on the substitute input image and the substitute target quality control parameters.

[0122] Generating the compressed representation includes encoding the alternative input image using an encoding neural network. Encoding may include generating a first combined output based on a combination of a first shared encoding performed on the alternative input image using first shared encoding parameters and a first adaptive encoding performed on the alternative input image using first adaptive encoding parameters. Encoding may also include generating a second combined output based on a combination of a second shared encoding performed on the first combined output using second shared encoding parameters and a second adaptive encoding performed on the first combined output using second adaptive encoding parameters.

[0123] In some embodiments, encoding may include: performing a first shared encoding on an alternative input image using a first shared encoding layer having first shared encoding parameters; performing a first adaptive encoding on the alternative input image using a first adaptive encoding layer having first adaptive encoding parameters; combining the first shared encoded image and the first adaptively encoded image to generate a first combined output; and performing a second shared encoding on the first combined output using a second shared encoding layer having second shared encoding parameters. Encoding may also include performing a second adaptive encoding on the first combined output using the second adaptive encoding layer having the second adaptive encoding parameters.

[0124] Generating the first combined output may include generating a shared feature based on the substitute input image and the first shared encoding parameter, and generating an estimated first adaptive encoding parameter based on the substitute input image, the substitute target quality control parameter, the first adaptive encoding parameter, and the generated shared feature using a predictive neural network. The first combined output may then be generated based on the generated shared feature and the generated estimated first adaptive encoding parameter.

[0125] In some embodiments, when training the prediction neural network, a first loss for training data corresponding to the substituted target quality control parameter and a second loss for validation data corresponding to the substituted target quality control parameter may be generated based on the substituted target quality control parameter, the first shared encoding parameter, the first adaptive encoding parameter, the first shared decoding parameter, the first adaptive decoding parameter, and the prediction parameters of the prediction neural network. Training the prediction neural network may also include updating the prediction parameters based on the gradients of the generated first loss and the generated second loss.

[0126] In operation 630 , method 600 may further include generating a compressed code stream based on the compressed representation of the substitute input image using quantization and entropy coding.

[0127] like Figure 6 As shown in , in operation 640, method 600 may further include decoding the recovered compressed representation based on the received alternative target quality control parameters using a decoding neural network to reconstruct the output image. In some embodiments, decoding may be preceded by a receiving operation including receiving a recovered compressed code stream and an alternative target quality control parameter. Upon receiving the recovered compressed code stream and the alternative target quality control parameter, method 600 may include generating a recovered compressed representation based on the recovered compressed code stream and the received alternative target quality control parameter using inverse quantization and entropy decoding.

[0128] Decoding the recovered compressed representation may include decoding the recovered compressed representation using a decoding neural network. Decoding may include generating a first combined output based on a combination of first shared decoding performed on the recovered compressed representation using first shared decoding parameters and first adaptive decoding performed on the recovered compressed representation using first adaptive decoding parameters. Decoding may also include generating a second combined output based on a combination of second shared decoding performed on the first combined output using second shared decoding parameters and second adaptive decoding performed on the first combined output using second adaptive decoding parameters.

[0129] In some embodiments, decoding may include: performing a first shared decoding on the recovered compressed representation using a first shared decoding layer having first shared decoding parameters; performing a first adaptive decoding on the recovered compressed representation using a first adaptive decoding layer having first adaptive decoding parameters; combining the recovered compressed representation after the first shared decoding and the recovered compressed representation after the first adaptive decoding to generate a first combined output; and performing a second shared decoding on the first combined output using a second shared decoding layer having second shared decoding parameters. Decoding may also include performing a second adaptive decoding on the first combined output using the second adaptive decoding layer having the second adaptive decoding parameters.

[0130] Generating the first combined output may include generating a shared feature based on the recovered compressed representation and the first shared decoding parameter, and generating an estimated first adaptive decoding parameter based on the recovered compressed representation, the received substitute target quality control parameter, the first adaptive decoding parameter, and the generated shared feature using a predictive neural network. The first combined output may then be generated based on the generated shared feature and the generated estimated first adaptive decoding parameter.

[0131] When training the encoding neural network and the decoding neural network, an inner loop loss may be generated for the training data corresponding to the alternative target quality control parameters based on the alternative target quality control parameters, the first shared encoding parameters, the first adaptive encoding parameters, the first shared decoding parameters, and the first adaptive decoding parameters. The training may also include updating the first shared encoding parameters, the first adaptive encoding parameters, the first shared decoding parameters, and the first adaptive decoding parameters for the first time based on the gradient of the generated inner loop loss. Then, the training may include: generating a meta-loss for the verification data corresponding to the alternative target quality control parameters based on the alternative target quality control parameters, the first updated first shared encoding parameters, the first updated first adaptive encoding parameters, the first updated first shared decoding parameters, and the first updated first adaptive decoding parameters. In some embodiments, the training may also include: updating the first updated first shared encoding parameters, the first updated first adaptive encoding parameters, the first updated first shared decoding parameters, and the first updated first adaptive decoding parameters for a second time based on the gradient of the generated meta-loss.

[0132] Although Figure 6 Example blocks of an apparatus are shown, but in some implementations, the apparatus may include Figure 6 More blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in FIG. Additionally or alternatively, two or more blocks of a device may be combined.

[0133] The proposed methods can be used individually or combined in any order. Furthermore, each method (or embodiment), encoder, and decoder can be implemented by a processing circuit (e.g., at least one processor or at least one integrated circuit). In one example, the at least one processor executes a program stored in a non-volatile computer-readable storage medium.

[0134] The above disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0135] As used in this application, the term component is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0136] It is apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not a limitation of the implementation. Therefore, this application describes the operation and behavior of the systems and / or methods without reference to specific software code - it is understood that software and hardware can be designed to implement the systems and / or methods based on the description of this application.

[0137] Even though particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.

[0138] Unless expressly stated otherwise, no element, act, or instruction used herein should be construed as critical or essential. Furthermore, as used herein, the term "group" is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and is used interchangeably with "one or more." Furthermore, unless expressly stated otherwise, the phrase "based on" is intended to mean "based at least in part on."

Claims

1. An adaptive neural network image compression method based on meta-learning, characterized in that: include: Using the original input image x and the target quality control parameter Λ, generating an alternative input image x' and an alternative target quality control parameter Λ' includes: For each iteration: Based on the reconstructed image And the input image x, according to the following formula (1), calculate the distortion loss Where q is the quality metric The number of weights w i ≥0, the target quality control parameter Λ is the sum of all weights w i and a single vector of target trade-off hyperparameters λ, Λ = [w1,…w q ,λ]; The distortion loss and rate loss Combined, according to the following formula (2), the total rate distortion loss is calculated Wherein, given the alternative input image x' and the alternative target quality control parameter Λ', the compressed representation is calculated The rate loss Characterizing the compressed representation bit consumption; Using backpropagation, the total rate-distortion loss is calculated gradient; Based on the total rate-distortion loss , updating the alternative input image x' and the alternative target quality control parameter Λ'; wherein, when performing the first iteration, the alternative input image x' is initialized to the input image x, and the alternative target quality control parameter Λ' is initialized to the target quality control parameter Λ; and, The alternative input image x' is encoded based on the alternative target quality control parameter Λ' using an encoding neural network to generate a compressed representation of the alternative input image x'.

2. The method according to claim 1, characterized in that Also includes: The number of iterations is determined based on the update of the replacement input image x' being less than a first preset threshold.

3. The method according to claim 1, characterized in that Also includes: The number of the multiple iterations is determined based on that the update of the alternative target quality control parameter Λ' is less than a second preset threshold.

4. The method according to claim 1, wherein Also includes: A compressed code stream is generated based on the compressed representation of the alternative input image x' using quantization and entropy coding.

5. The method according to claim 1, wherein Also includes: receiving a recovered compressed code stream and the alternative target quality control parameter Λ'; Generate a recovered compressed representation based on the recovered compressed code stream and the substitute target quality control parameter Λ' using inverse quantization and entropy decoding; The recovered compressed representation is decoded using a decoding neural network based on the alternative target quality control parameter Λ' to reconstruct an output image.

6. The method according to claim 1, characterized in that The encoding of the substitute input image x' based on the substitute target quality control parameter Λ' using an encoding neural network comprises: performing a first shared encoding on the alternative input image x' using a first shared coding layer having first shared coding parameters; performing a first adaptive encoding on the alternative input image x' using a first adaptive coding layer having first adaptive coding parameters; and combining the first shared encoded image and the first adaptively encoded image to generate a first combined output; Using a second shared coding layer with second shared coding parameters, perform second shared coding on the first combined output; using a second adaptive coding layer with second adaptive coding parameters, perform second adaptive coding on the first combined output; combine the image that has undergone the second shared coding and the image that has undergone the second adaptive coding to generate a second combined output.

7. The method according to claim 6, characterized in that The combining the first shared coded image and the first adaptively coded image to generate a first combined output includes: generating a shared feature based on the alternative input image x' and the first shared encoding parameters; generating, using a predictive neural network, an estimated first adaptive coding parameter based on the substitute input image x', the substitute target quality control parameter Λ', the first adaptive coding parameter, and the shared feature; The first combined output is generated based on the shared feature and the estimated first adaptive encoding parameter.

8. A non-volatile computer-readable storage medium, characterized in that: At least one instruction is stored thereon, and when the at least one instruction is executed by at least one processor for adaptive neural network image compression based on meta-learning, the at least one processor executes the method according to any one of claims 1 to 7.

9. A computer device, characterized in that: The system comprises a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

Images