Method, apparatus, electronic device and computer-readable medium for feature replacement for end-to-end image compression
By using encoding and quantization neural networks in end-to-end image compression, combined with rate loss training, flexible bit rate and compression target control are achieved, solving the problem of multi-model dependency in existing technologies and reducing storage and computation costs.
Patent Information
- Application Number
- CN202180006010.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-04-28
- Filing Date
- 2021-05-07
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-05-07
AI Technical Summary
Existing end-to-end image compression technologies struggle to achieve flexible bit rate control and compression target control, requiring the training and deployment of multiple deep neural network models, resulting in excessively high storage and computation costs.
By using a first neural network to encode the input image to generate an encoded representation, using a second neural network to quantize the encoded representation to generate a compressed representation, and updating the encoded representation by training a determination rate loss, flexible bit rate control and compression target control are achieved.
It provides flexible bitrate control and compression target control capabilities, effectively reconstructing images that meet the user-defined desired bitrate, reducing dependence on multiple models, and lowering storage and computation costs.
Smart Images

Figure FDA0005057646880000011 
Figure FDA0005057646880000012 
Figure FDA0005057646880000013
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based upon and claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 023,694 filed on May 12, 2020, U.S. Provisional Patent Application No. 63 / 026,409 filed on May 18, 2020, and U.S. Patent Application No. 17 / 242,534 filed on April 28, 2021, the disclosures of which are incorporated herein by reference in their entirety. Technical Field
[0003] The present application relates to video processing technology, and more particularly to a method, device, electronic device and computer-readable medium for feature replacement for end-to-end image compression. Background Art
[0004] Standards groups and companies have been actively looking for potential needs for standardization of future video coding technologies. Some have focused on end-to-end (E2E) image and video compression based on artificial intelligence (AI).
[0005] Although the existing techniques have shown promising performance, the following two problems of E2E image compression still exist. First, it may be difficult to implement bitrate control, in which each deep neural network (DNN) model is trained with a target bitrate, because it may not work well if applied to compress images at other bitrates. Second, each DNN model may be trained to optimize certain target training losses, such as peak signal-to-noise ratio (PSNR) and / or structural similarity index measure (SSIM), which may not work well for other target training losses. In other words, in order to provide the flexibility of compressing input images at different bitrates or using different target metrics, multiple DNN models may need to be trained, stored, and deployed, which may be too expensive to be practical in terms of both storage and computation. Summary of the invention
[0006] In order to solve the above technical problems, embodiments of the present application provide a method, device, electronic device and computer-readable medium for feature replacement in end-to-end image compression.
[0007] According to one aspect of the embodiment, the present application provides a method for feature replacement for end-to-end image compression, which is executed by at least one processor and includes: encoding an input image using a first neural network to generate an encoded representation; and quantizing the generated encoded representation using a second neural network to generate a compressed representation. The generation process of the encoded representation is determined by training, and the training includes: determining a rate loss based on a bit rate of the generated compressed representation, and updating the generated encoded representation based on the determined rate loss.
[0008] According to another aspect of the implementation, the present application also provides a device for feature replacement for end-to-end image compression, including: an encoding module for encoding an input image using a first neural network to generate an encoded representation; a quantization module for quantizing the generated encoded representation using a second neural network to generate a compressed representation; and an update module for determining the generation process of the encoded representation through training, and the training includes: determining a rate loss based on the bit rate of the generated compressed representation, and updating the generated encoded representation based on the determined rate loss.
[0009] According to another aspect of the implementation, the present application also provides an electronic device, including a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the above method.
[0010] According to another aspect of the implementation, the present application also provides a non-transitory computer-readable medium storing instructions, which, when executed by at least one processor for feature replacement for end-to-end image compression, causes the at least one processor to perform the above method.
[0011] It can be seen that the embodiment of the present application provides a method for feature replacement for end-to-end image compression, which compresses the input image by learning an alternative coding feature representation via an E2E DNN. The learned alternative coding feature representation is a change of the original coding feature representation, and can be used to effectively reconstruct an image with improved compression performance, as well as provide the ability to control flexible bitrate and compression target. The input image can meet the desired bitrate defined by the user. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 is a diagram of an environment in which the methods, devices, and systems described herein may be implemented according to an embodiment of the present application.
[0013] Figure 2 yes Figure 1 A block diagram of example components of one or more devices in FIG.
[0014] Figure 3A 4 is a block diagram of a training device for feature replacement for end-to-end image compression during a training phase according to an embodiment of the present application.
[0015] Figure 3B 4 is a block diagram of a training apparatus for feature replacement for end-to-end image compression with bit rate constraints during a training phase according to an embodiment of the present application.
[0016] Figure 4 4 is a block diagram of a testing apparatus for feature replacement for end-to-end image compression during a testing phase according to an embodiment of the present application.
[0017] Figure 5 The figure is a flowchart of a method for feature replacement for end-to-end image compression according to an embodiment of the present application.
[0018] Figure 6 It is a block diagram of a device for feature replacement for end-to-end image compression according to an embodiment of the present application. DETAILED DESCRIPTION
[0019] The present disclosure describes a method and apparatus for compressing an input image by learning an alternative encoding feature representation via an E2E DNN. The learned alternative encoding feature representation is a modification of the original encoding feature representation and can be used to efficiently reconstruct an image with improved compression performance, as well as providing the ability for flexible bitrate control and compression target control. The input image can meet a desired bitrate defined by a user.
[0020] Figure 1 is a diagram of an environment 100 in which the methods, apparatuses, and systems described herein may be implemented, according to an embodiment.
[0021] like Figure 1 As shown, environment 100 may include user device 110, platform 120, and network 130. The devices of environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.
[0022] The user device 110 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with the platform 120. For example, the user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a wearable device (e.g., smart glasses or a smart watch), or a similar device. In some implementations, the user device 110 may receive information from the platform 120 and / or send information to the platform 120.
[0023] The platform 120 includes one or more devices as described elsewhere herein. In some implementations, the platform 120 may include a cloud server or a cloud server group. In some implementations, the platform 120 may be designed to be modular so that software components can be swapped in or out. In this way, the platform 120 may be easily and / or quickly reconfigured for different uses.
[0024] In some implementations, as shown, the platform 120 can be hosted in a cloud computing environment 122. It is worth noting that while the implementations described herein describe the platform 120 as being hosted in a cloud computing environment 122, in some implementations, the platform 120 may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.
[0025] Cloud computing environment 122 includes an environment hosting platform 120. Cloud computing environment 122 may provide computing, software, data access, storage, etc. services that do not require end users (e.g., user devices 110) to be aware of the physical location and configuration of (one or more) systems and / or devices hosting platform 120. As shown, cloud computing environment 122 may include a set of computing resources 124 (collectively, “computing resources 124” and individually, “computing resource 124”).
[0026] The computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, the computing resources 124 can host the platform 120. Cloud resources may include: computing instances executed in the computing resources 124, storage devices provided in the computing resources 124, data transmission devices provided by the computing resources 124, etc. In some implementations, the computing resources 124 can communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.
[0027] Further Figure 1 As shown, the computing resources 124 include a set of cloud resources, such as one or more applications ("APP") 124-1, one or more virtual machines ("VM") 124-2, virtualized storage devices ("VS") 124-3, one or more hypervisors ("HYP") 124-4, etc.
[0028] The applications 124-1 include one or more software applications that can be provided to or accessed by the user device 110 and / or the platform 120. The applications 124-1 can eliminate the need to install and execute software applications on the user device 110. For example, the applications 124-1 can include software associated with the platform 120 and / or any other software that can be provided via the cloud computing environment 122. In some implementations, one application 124-1 can send information to or receive information from one or more other applications 124-1 via the virtual machine 124-2.
[0029] The virtual machine 124-2 includes a software implementation of a machine (e.g., a computer) that executes programs similar to physical machines. The virtual machine 124-2 can be a system virtual machine or a process virtual machine, depending on the use and correspondence of the virtual machine 124-2 to any real machine. The system virtual machine can provide a complete system platform that supports the execution of a complete operating system ("OS"). The process virtual machine can execute a single program and can support a single process. In some implementations, the virtual machine 124-2 can execute on behalf of a user (e.g., a user device 110) and can manage the infrastructure of the cloud computing environment 122, such as data management, synchronization, or long-duration data transfer.
[0030] The virtualized storage device 124-3 includes one or more storage systems and / or one or more devices using virtualization technology within the storage system or device of the computing resource 124. In some implementations, within the context of the storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the extraction (or separation) of logical storage from physical storage so that the storage system can be accessed without considering physical storage or heterogeneous structures. Separation may allow administrators of the storage system flexibility in terms of administrator management of storage for end users. File virtualization may eliminate the dependency between data accessed at the file level and the location where the file is physically stored. This may enable optimization of storage usage, server consolidation, and / or performance of non-disruptive file migration.
[0031] Hypervisor 124-4 may provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to execute simultaneously on a host computer such as computing resource 124. Hypervisor 124-4 may present a virtual operating platform to the guest operating systems and may manage the execution of the guest operating systems. Multiple instances of various operating systems may share virtualized hardware resources.
[0032] The network 130 includes one or more wired and / or wireless networks. For example, the network 130 may include a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.
[0033] Figure 1 The number and arrangement of devices and networks shown are provided as examples. Figure 1 There may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks than those shown. Figure 1 Two or more of the devices shown may be implemented in a single device, or Figure 1 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one or more devices of one set of devices (eg, one or more devices) of environment 100 may perform one or more functions described as being performed by another set of devices of environment 100.
[0034] Figure 2 yes Figure 1 A block diagram of example components of one or more devices of the present invention.
[0035] Device 200 may correspond to user device 110 and / or platform 120. Figure 2 As shown, device 200 may include a bus 210 , a processor 220 , a memory 230 , a storage component 240 , an input component 250 , an output component 260 , and a communication interface 270 .
[0036] The bus 210 includes components that allow communication between components of the device 200. The processor 220 is implemented in hardware, firmware, or a combination of hardware and software. The processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 220 includes one or more processors that can be programmed to perform functions. The memory 230 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 220.
[0037] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-transitory computer-readable medium and a corresponding drive.
[0038] Input components 250 include components that allow device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 250 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 260 include components that provide output information from device 200 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).
[0039] The communication interface 270 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 270 can allow the device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.
[0040] Device 200 may perform one or more processes described herein. Device 200 may perform these processes in response to processor 220 executing software instructions stored by a non-transitory computer-readable medium (e.g., memory 230 and / or storage component 240). Computer-readable media are defined herein as non-transitory memory devices. Memory devices include memory space within a single physical storage device or memory space distributed on multiple physical storage devices.
[0041] The software instructions may be read into the memory 230 and / or storage component 240 from another computer-readable medium or from another device via the communication interface 270. When executed, the software instructions stored in the memory 230 and / or storage component 240 may cause the processor 220 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the implementations described herein are not limited to any particular combination of hardware circuitry and software.
[0042] Figure 2 The number and arrangement of components shown are provided as examples. Figure 2 The device 200 may include additional components, fewer components, different components, or differently arranged components than the components shown. Additionally or alternatively, a set of components (e.g., one or more components) of the device 200 may perform one or more functions described as being performed by another set of components of the device 200.
[0043] A method and apparatus for feature replacement for end-to-end image compression will now be described in detail.
[0044] This disclosure describes E2E image compression by learning alternative encoded feature representations.
[0045] Given an input image x of size (h, w, c), where h, w, c are height, width, and number of channels, respectively, the goal of E2E image compression is described as follows, where a compressed (quantized) representation y' is calculated that is compact for storage and transmission. Then, based on the calculated compressed representation y', the output image x' is reconstructed so that the reconstructed image x' and the original input image x can be similar. The loss function D(x, x') is used to measure the reconstruction error, i.e., distortion loss such as PSNR and / or SSIM. In an embodiment, the DNN that calculates the compressed representation y' may have two parts: an encoder that generates an encoded representation y based on the input image x, and a quantizer that generates a compressed representation y' based on the encoded representation y. The DNN that reconstructs the output image x' is a decoder.
[0046] Given an input image x, the device first goes through a training phase to learn an alternative encoding feature representation Then in the testing phase, Used to generate compressed (quantized) representation and reconstruct the image
[0047] Figure 3A is a block diagram of a training apparatus 300A for feature replacement for end-to-end image compression during a training phase according to an embodiment.
[0048] like Figure 3A As shown, the training device 300A includes: a training encoder 310, a training quantizer 320, a training decoder 330, a distortion loss component 340, a rate loss component 350 and a data updating component 360.
[0049] For each input image x, the training encoder 310, the training quantizer 320, and the training decoder 330 are DNNs, and the model weights of the training encoder, the training quantizer, and the training decoder are initialized or set based on the predetermined encoder, the predetermined quantizer, and the predetermined decoder, respectively. Then, a retraining / fine-tuning process is performed to calculate the updated alternative encoding feature representation The training quantizer 320 is based on the calculated alternative encoding feature representation Generate a new compressed (quantized) representation And train the decoder 330 based on the generated new compressed representation Generate a new reconstructed image The goal of the retraining / fine-tuning process is to optimize (minimize) the total loss The total loss consists of the following two parts:
[0050]
[0051] The distortion loss component 340 determines the same distortion loss that measures the reconstruction error The rate loss component 350 determines and compresses the representation The bit rate related rate loss The rate loss may be calculated using a predetermined rate loss estimator The hyperparameter λ balances the contribution of different loss terms. The data updating component 360 is based on the determined distortion loss The determined rate loss and the input hyperparameter λ to determine the total loss
[0052] The weights of a predetermined encoder, a predetermined quantizer, a predetermined decoder, and a predetermined rate loss estimator are pre-trained based on a set of training images. In detail, for each training image xtr , the same forward inference calculation is performed through encoding, quantization, and decoding to calculate the reconstructed image x tr '. Then, calculate the distortion loss D(x tr , x tr ') and rate loss R(x tr ). Then, given the pre-trained hyperparameter λ tr In the case of, the total loss L(x) can be calculated based on formula (1) tr , x tr '), the total loss L(x tr , x tr The gradient of ') is used to update the weights of the pre-trained encoder, pre-trained quantizer, pre-trained decoder, and pre-trained rate loss estimator through back-propagation.
[0053] During the training phase, for each input image x, the training encoder 310, the training quantizer 320, the training decoder 330, and the rate loss estimator are initialized in the retraining / fine-tuning process, and the training encoder 310 initializes the initial encoded representation y0 to the encoded representation y. Thereafter, for the tth iteration, the training quantizer 320 calculates the compressed (quantized) representation y t ', and the training decoder 330 is based on the quantized representation y t Calculate the reconstructed image x t '. The distortion loss component 340 calculates the distortion loss D(x, x t ') and rate loss R(y t '). Then, the data updating component 360 calculates the common loss L(x, y t ', x t ') to update the alternative encoding feature representation y through back propagation t+1 Finally, after T iterations, the optimal replacement encoding feature representation is output
[0054] Furthermore, the present disclosure describes a method for E2E image compression with bitrate constraints by learning alternative encoded feature representations.
[0055] The bit rate of the compressed representation y' is denoted by B(y'). A large bit rate B(y') may cause small distortion in the reconstructed image x', while a small bit rate B(y') may cause large distortion in the reconstructed image x'. R(y') represents the rate loss associated with the bit rate B(y'). Given a target rate loss R target The associated target bit rate B target In the case of , the compressed representation y' makes the bit rate B(y') as close as possible to the target bit rate B target . Therefore, the total loss can be defined as follows:
[0056] L(x,y',x')=etaL(R(y'),R target )+D(x,x') (2)
[0057] L(R(y'),R target )Measured rate loss R(y') and target rate loss R target The difference between the loss and η is a hyperparameter that balances the contribution of different loss terms. The hyperparameter η can be set to a large number based on experience to ensure that the actual bitrate is as close to the target bitrate as possible. In addition, the rate loss R(y') is related to the target rate loss R target The difference loss between can be set as follows:
[0058] L(R(y'),R target )=min(R(y')-R target ,∈) (3)
[0059] ∈ is a hyperparameter, which can be a very small number.
[0060] Figure 3B is a block diagram of a training apparatus 300B for feature replacement for end-to-end image compression with bitrate constraints during a training phase according to an embodiment.
[0061] like Figure 3B As shown, the training device 300B includes: a training encoder 310, a training quantizer 320, a training decoder 330, a distortion loss component 340, a rate loss component 350 and a data updating component 370.
[0062] Given an input image x and a target bitrate B target In the case of, during the training phase, the encoder 310 is trained to learn the alternative encoding feature representation Then, during the testing phase, use To generate the compressed representation and reconstruct the image In detail, for each input image x, the model weights of the training encoder 310, the training quantizer 320 and the training decoder 330 are initialized or set based on the predetermined encoder, the predetermined quantizer and the predetermined decoder, respectively. Then, a retraining / fine-tuning process is performed, in which the data updating component 370 calculates the updated alternative encoding feature representation The training quantizer 320 is based on the calculated alternative encoding feature representation Generate a new compressed (quantized) representation And train the decoder 330 based on the generated new compressed representation Generate a new reconstructed image The goal of the retraining / fine-tuning process is to optimize the total loss
[0063]
[0064] The rate loss may be calculated using a predetermined rate loss estimator Entropy estimation method.
[0065] The weights of a predetermined encoder, a predetermined quantizer, a predetermined decoder, and a predetermined rate loss estimator are pre-trained based on a set of training images. In detail, for each training image x tr , the same forward inference calculation is performed through encoding, quantization, and decoding to calculate the reconstructed image x tr '. Then, calculate the distortion loss D(x tr , x tr ') and rate loss R(y tr '). Then, given the pre-trained hyperparameter λ tr , the total loss L can be calculated tr (x tr ,y tr ', x tr '):
[0066] L tr (x tr ,y tr ', x tr ')=R(y tr ')+λ tr D(x tr , x tr '), (5)
[0067] The gradient of the total loss is used to update the weights of the pre-trained encoder, pre-trained quantizer, pre-trained decoder, and pre-trained rate loss estimator via back-propagation. tr ') is the rate loss, D(x tr , x tr ') are distortion losses and they are defined in the same way as in the above paragraph.
[0068] Figure 4 is a block diagram of a testing apparatus 400 for feature replacement for end-to-end image compression during a testing phase, according to an embodiment.
[0069] like Figure 4 As shown, the test device 400 includes: a test encoder 410 , a test quantizer 420 and a test decoder 430 .
[0070] In the test phase, the learned alternative encoding feature representation is used As input, the test quantizer 420 and the test decoder 430 are initialized. The test quantizer 420 computes the compressed (quantized) representation And through forward reasoning calculation, the test decoder 430 calculates the reconstructed image
[0071] The test quantizer 420 and the test decoder 430 may be the same as their corresponding training quantizer and training decoder, respectively. They may also be different. In an embodiment, the test decoder 430 is the same as the training decoder, while the training quantizer and the test quantizer 420 are different. In detail, the test quantizer 420 may adopt a general uniform quantization method, and the training quantizer may be a uniform sampler similar to the effect of the test quantizer 420, which avoids the difficulty of using discrete numbers for gradient back propagation.
[0072] The above embodiments provide great flexibility in bit rate control and target metric control. When the target bit rate changes, only the hyperparameter λ may need to be changed without retraining / fine-tuning the pretrained encoder, pretrained quantizer, and pretrained decoder. Similarly, when a compressed image that is optimal for different target metrics (e.g., with optimal PSNR and / or optimal SSIM) is to be obtained, only the way of calculating the distortion loss may be changed without retraining / fine-tuning the pretrained E2E image compression network.
[0073] Figure 5 is a flowchart of a method for feature replacement for end-to-end image compression according to an embodiment.
[0074] In some implementations, the platform 120 may execute Figure 5 In some implementations, Figure 5 One or more processing blocks of may be performed by another device or group of devices (eg, user device 110 ) separate from platform 120 or including platform 120 .
[0075] like Figure 5 As shown, in operation 510, method 500 includes encoding an input image using a first neural network to generate an encoded representation.
[0076] In operation 520, method 500 includes quantizing the generated encoded representation using a second neural network to generate a compressed representation.
[0077] The first neural network and the second neural network are trained by determining a rate loss based on a bit rate of the generated compressed representation and updating the generated encoded representation based on the determined rate loss.
[0078] In operation 530 , method 500 includes decoding the generated compressed representation using a third neural network to generate an output image.
[0079] The first neural network, the second neural network, and the third neural network may be trained by determining a distortion loss corresponding to a reconstruction error between an input image and a generated output image, and updating the generated encoded representation based on the determined distortion loss, the determined rate loss, and an input hyperparameter.
[0080] Updating the generated encoded representation may include optimizing the total loss based on equation (1).
[0081] The first neural network, the second neural network, and the third neural network may be trained by determining a minimum value between a difference between a determined rate loss and a target rate loss and a predetermined hyperparameter, determining a distortion loss corresponding to a reconstruction error between an input image and a generated output image, and updating the generated encoded representation based on the determined minimum value, the determined rate loss, and the input hyperparameter.
[0082] Updating the generated encoded representation may include optimizing the total loss based on equation (4).
[0083] The reconstruction error may include PSNR and / or SSIM.
[0084] although Figure 5 Example blocks of method 500 are shown, but in some implementations, Figure 5 Compared to the blocks depicted in , method 500 may include additional blocks, fewer blocks, different blocks, or blocks arranged differently. Additionally or alternatively, two or more blocks in the blocks of method 500 may be executed in parallel.
[0085] Figure 6 is a block diagram of an apparatus 600 for feature replacement for end-to-end image compression according to an embodiment.
[0086] like Figure 6 As shown, the apparatus 600 includes: an encoding code 610 , a quantization code 620 , and a decoding code 630 .
[0087] The encoding code 610 is configured to cause at least one processor to encode an input image using a first neural network to generate an encoded representation.
[0088] The quantization code 620 is configured to cause the at least one processor to quantize the generated encoded representation using a second neural network to generate a compressed representation.
[0089] The first neural network and the second neural network are trained by determining a rate loss based on a bit rate of the generated compressed representation and updating the generated encoded representation based on the determined rate loss.
[0090] The decoding code 630 is configured to cause at least one processor to decode the generated compressed representation using a third neural network to generate an output image.
[0091] The first neural network, the second neural network, and the third neural network may be trained by determining a distortion loss corresponding to a reconstruction error between an input image and a generated output image, and updating the generated encoded representation based on the determined distortion loss, the determined rate loss, and an input hyperparameter.
[0092] The generated encoded representation can be updated by optimizing the total loss based on equation (1).
[0093] The first neural network, the second neural network, and the third neural network may be trained by determining a minimum value between a difference between a determined rate loss and a target rate loss and a predetermined hyperparameter, determining a distortion loss corresponding to a reconstruction error between an input image and a generated output image, and updating the generated encoded representation based on the determined minimum value, the determined rate loss, and the input hyperparameter.
[0094] The generated encoded representation can be updated by optimizing the total loss based on equation (4).
[0095] The reconstruction error may include PSNR and / or SSIM.
[0096] Compared with previous E2E image compression methods, the above embodiments have the following advantages. The above embodiments can be regarded as a universal preprocessing that can be installed to any E2E image compression DNN model. In addition, for each individual input image, its alternative encoded feature representation can be optimized through a separate retraining / fine-tuning process based on the feedback of its loss, which can improve the compression performance.
[0097] In addition, the above embodiments can achieve flexible bitrate control without retraining the E2E image compression model or using multiple models. The above embodiments can achieve flexible target bitrate control without retraining the E2E image compression model or using multiple models. The above embodiments can easily change the target compression metric without retraining the E2E image compression model.
[0098] The proposed methods can be used alone or in combination in any order. In addition, each of the method (or implementation), encoder, and decoder can be implemented by a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.
[0099] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the exact forms disclosed.Modifications and variations may be made in light of the above disclosure or may be acquired from practice of the implementations.
[0100] As used herein, the term "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.
[0101] It will be apparent that the systems and / or methods described herein can be implemented by various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software codes, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.
[0102] Even if combinations of features are defined in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly refer to only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim group.
[0103] Any element, action or instruction used herein should not be interpreted as critical or necessary unless explicitly described as such. In addition, as used herein, the articles "a" and "an" are intended to include one or more items and can be used interchangeably with "one or more". In addition, as used herein, the term "group" is intended to include one or more items (e.g., related items, unrelated items, combinations of related items and unrelated items, etc.), and can be used interchangeably with "one or more". In the case of meaning only one item, the term "one" or similar language is used. Moreover, as used herein, the terms "have", "have", "contain" etc. are intended to be open terms. In addition, unless otherwise expressly stated, the phrase "based on" is intended to mean "based at least in part on".
Claims
1. A method for feature replacement for end-to-end image compression, characterized in that: The method comprises: encoding the input image using a first neural network to generate an encoded representation; and quantizing the generated encoded representation using a second neural network to generate a compressed representation, wherein the first neural network includes an encoder and the second neural network includes a quantizer; The generation process of the encoded representation is determined by training, and the training comprises the following steps: (a) quantizing the generated coded representation using the quantizer to generate a compressed representation; (b) determining a rate loss associated with the bit rate based on the bit rate of the compressed representation; (c) obtaining a new encoded representation using the encoder based on back propagation according to the rate loss; (d) Repeat steps (a) to (c) and generate the optimal encoding representation after T iterations.
2. The method according to claim 1, further comprising: The generated compressed representation is decoded using a third neural network to produce an output image.
3. The method according to claim 2, wherein: The training includes: determining a distortion loss corresponding to a reconstruction error between the input image and the generated output image; and The generated encoded representation is updated based on the determined distortion loss, the determined rate loss, and the input hyperparameters.
4. The method according to claim 3, wherein: Updating the generated encoded representation consists of optimizing the total loss based on: in, is the total loss, is based on the generated compressed representation The bit rate determines the rate loss, λ is a hyperparameter of the input, and is the input image x and the generated output image The reconstruction error between and corresponds to the distortion loss.
5. The method according to claim 2, wherein: The training includes: determining a minimum value between a difference between the determined rate loss and the target rate loss and a predetermined hyperparameter; determining a distortion loss corresponding to a reconstruction error between the input image and the generated output image; and The generated encoded representation is updated based on the determined minimum value, the determined rate loss, and the input hyperparameters.
6. The method according to claim 5, wherein: Updating the generated encoded representation consists of optimizing the total loss based on: in, is the total loss, η is the input hyperparameter, is based on the generated compressed representation The bit rate determines the rate loss, R target is the target rate loss, ∈ is the predetermined hyperparameter, and is the input image x and the generated output image The reconstruction error between and corresponds to the distortion loss.
7. The method according to any one of claims 3 to 6, wherein: The reconstruction error includes: peak signal-to-noise ratio (PSNR) and / or structural similarity index measure (SSIM).
8. A device for feature replacement for end-to-end image compression, characterized in that: The device comprises: an encoding module for encoding an input image using a first neural network to generate an encoded representation; a quantization module for quantizing the generated encoded representation using a second neural network to generate a compressed representation; wherein the first neural network includes an encoder and the second neural network includes a quantizer; and an updating module, configured to determine the generation process of the encoded representation by training, wherein the training comprises the following steps: (a) quantizing the generated coded representation using the quantizer to generate a compressed representation; (b) determining a rate loss associated with the bit rate based on the bit rate of the compressed representation; (c) obtaining a new encoded representation using the encoder based on back propagation according to the rate loss; (d) Repeat steps (a) to (c) and generate the optimal encoding representation after T iterations.
9. The apparatus according to claim 8, further comprising: A decoding module is used to decode the generated compressed representation using a third neural network to generate an output image.
10. The device according to claim 9, wherein: The training includes: determining a distortion loss corresponding to a reconstruction error between the input image and the generated output image; and The generated encoded representation is updated based on the determined distortion loss, the determined rate loss, and the input hyperparameters.
11. The device according to claim 10, wherein: The generated encoded representation is updated by optimizing the total loss based on the following formula: in, is the total loss, is based on the generated compressed representation The bit rate determines the rate loss, λ is a hyperparameter of the input, and is the input image x and the generated output image The reconstruction error between and corresponds to the distortion loss.
12. The device according to claim 9, wherein: The training includes: determining a minimum value between a difference between the determined rate loss and the target rate loss and a predetermined hyperparameter; determining a distortion loss corresponding to a reconstruction error between the input image and the generated output image; and The generated encoded representation is updated based on the determined minimum value, the determined rate loss, and the input hyperparameters.
13. The device according to claim 12, wherein: The generated encoded representation is updated by optimizing the total loss based on the following formula: in, is the total loss, η is the input hyperparameter, is based on the generated compressed representation The bit rate determines the rate loss, R target is the target rate loss, ∈ is the predetermined hyperparameter, and is the input image x and the generated output image The reconstruction error between and corresponds to the distortion loss.
14. The device according to any one of claims 10 to 13, wherein: The reconstruction error includes: peak signal-to-noise ratio (PSNR) and / or structural similarity index measure (SSIM).
15. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 6.
16. A non-transitory computer readable medium storing instructions, characterized in that: The instructions, when executed by at least one processor for feature replacement for end-to-end image compression, cause the at least one processor to perform the method according to any one of claims 1 to 6.
