Deep learning-based compression method using frequency decomposition
Through end-to-end neural network based on frequency decomposition, the spatial frequency of images and videos is decomposed into potential representations and grouped and encoded in the latent space, the problem of difficulty in reducing spatial redundancy and adapting to the quality of the region of interest in the prior art is solved. , realizes efficient image and video compression and decoding.
Patent Information
- Application Number
- CN202380035116.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-05
- Filing Date
- 2023-04-25
- Publication Date
- 2024-12-03
AI Technical Summary
Existing neural network-based image and video compression methods are difficult to reduce spatial redundancy and adapt to the quality decoding of regions of interest when learning nonlinear transformations, resulting in limited compression efficiency and quality scalability.
Using an end-to-end neural network based on frequency decomposition, the spatial frequency of the input content is decomposed into potential representations through frequency decomposition transformation, and grouped in the latent space. Each frequency group is entropy encoded and decoded to achieve quality scalability and asymmetry. decoding.
Improve the rate distortion performance of images and videos, realize the quality scalability and efficient compression of the region of interest, and enhance the decoder's control ability of reconstructed input quality.
Smart Images

Figure CN119072926A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 338,562, filed on May 5, 2022, which is incorporated herein by reference in its entirety. Technical Field
[0003] The present embodiments generally relate to a method and apparatus for compressing images and videos using artificial neural network (ANN) based tools. Background Art
[0004] In recent years, new image and video compression methods based on neural networks have been developed. In contrast to traditional methods that apply predefined prediction patterns and transforms, ANN-based methods rely on many parameters that are learned on large datasets during a training phase by iteratively minimizing a loss function. In the case of compression, the loss function is defined, for example, by a rate-distortion cost, where the "rate" represents an estimate of the bitrate of the encoded bitstream and the distortion quantifies the quality of the decoded video relative to the original input. Traditionally, the quality of the decoded input image is optimized, for example, based on a measure of the mean squared error or an approximation of the visual quality perceived by humans.
[0005] The Joint Video Exploration Team (JVET) between ISO / MPEG and ITU is currently investigating ANN-based tools to replace some modules of the latest video coding standard H.266 / VVC, as well as replacing the entire structure with an end-to-end autoencoder approach. Summary of the invention
[0006] According to one embodiment, a method of video encoding includes: decomposing at least a portion of an image into multiple frequency groups, wherein the frequency groups correspond to a set of frequency bands; for each frequency group in the multiple frequency groups, generating a corresponding latent representation in a latent space; and entropy encoding one or more of the corresponding latent representations.
[0007] According to another embodiment, a method of video decoding includes: obtaining one or more potential representations in a latent space, wherein each of the one or more potential representations corresponds to one of one or more frequency groups, wherein a frequency group corresponds to a set of frequency bands; obtaining the one or more frequency groups from the one or more potential representations; and combining the one or more frequency groups into at least a portion of an image.
[0008] According to another embodiment, a device for video encoding is provided, the device comprising one or more processors and at least one memory connected to the one or more processors, wherein the one or more processors are configured to: decompose at least a portion of an image into a plurality of frequency groups, wherein a frequency group corresponds to a set of frequency bands; for each frequency group in the plurality of frequency groups, generate a corresponding latent representation in a latent space; and entropy encode one or more of the corresponding latent representations.
[0009] According to another embodiment, a device for video decoding is provided, the device comprising one or more processors and at least one memory connected to the one or more processors, wherein the one or more processors are configured to: obtain one or more potential representations in a latent space, wherein each of the one or more potential representations corresponds to one of one or more frequency groups, wherein a frequency group corresponds to a set of frequency bands; obtain the one or more frequency groups from the one or more potential representations; and combine the one or more frequency groups into at least a portion of an image.
[0010] One or more embodiments also provide a computer program, the computer program including instructions, which when executed by one or more processors, cause the one or more processors to perform an encoding method or a decoding method according to any one of the embodiments described herein. One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for video encoding or decoding according to the methods described herein.
[0011] One or more embodiments further provide a computer-readable storage medium having video data generated according to the method described above stored thereon. One or more embodiments further provide a method and apparatus for transmitting or receiving video data generated according to the method described herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 A block diagram of a system is shown within which aspects of the present embodiments may be implemented.
[0013] Figure 2 An example of an end-to-end compression system is shown.
[0014] Figure 3A shows that only situation; Figure 3B shows that there is a bit stream and and Figure 3C shows that a potential representation of the entire group is available in the bitstream
[0015] Figure 4A shows a frequency analysis module according to an embodiment; and Figure 4B A frequency synthesis module according to an embodiment is shown.
[0016] Figure 5 The proposed compression method including the proposed encoder and decoder architecture according to an embodiment is shown.
[0017] Fig. 6A It shows that the wavelet transform is divided into four sub-bands (such as Figure 6B ), and X 1,LL It is further divided into four sub-bands (such as Figure 6C shown).
[0018] Fig. 7A shows a frequency decomposition (FD) layer when n=2 according to an embodiment; Figure 7B The implementation of the frequency decomposition layer is described; Figure 7C A generalized frequency decomposition layer according to an embodiment is shown.
[0019] Fig. 8A shows a frequency combination (FD) layer when n=2 according to an embodiment; Figure 8B The implementation of the proposed frequency combination layer is described; Figure 8C A generalized frequency combining layer according to an embodiment is shown.
[0020] Fig. 9A Shown decoded as input, and Fig. 9B shows the lower quality output generated Fig. 9C It shows that when the decoder is available and Decoding of both, and Fig.9D shows that the quality of the output is improved
[0021] Fig. 10A shows a ROI (Region of Interest) processed at the frequency synthesis stage according to one embodiment; Fig. 10B Shown include The encoded latent variables of the parts in ; and Fig. 10C The ROI in the picture is shown.
[0022] Fig.11A It shows that the enhancement information of object #1 is encoded as Fig. 11B The signaling of ROI information is shown.
[0023] Fig. 12A It shows that the enhancement information of object #1 and object #2 is encoded as and Fig. 12B The signaling of ROI information is shown.
[0024] Fig.13A The decoder according to the embodiment is shown Run inference on the encoder to request latent variables Fig. 13B The decoder according to the embodiment is shown Run inference on the encoder to request latent variables DETAILED DESCRIPTION
[0025] Figure 1 A block diagram of an example of a system that can implement various aspects and embodiments is shown. System 100 can be embodied as a device including various components described below, and is configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, and servers. The elements of system 100 can be embodied in a single integrated circuit, multiple ICs, and / or discrete components, either alone or in combination. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 100 are distributed over multiple ICs and / or discrete components. In various embodiments, system 100 is connected to other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 100 is configured to implement one or more aspects described in this application.
[0026] The system 100 includes at least one processor 110, which is configured to execute instructions loaded therein to implement, for example, various aspects described in the present application. The processor 110 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include non-volatile memory and / or volatile storage, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drive, and / or optical drive. As a non-limiting example, the storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device.
[0027] The system 100 includes an encoder / decoder module 130, which is configured to process data, for example, to provide encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a module that can be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module 130 may be implemented as a separate element of the system 100, or may be incorporated into the processor 110 as a combination of hardware and software known to those skilled in the art.
[0028] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described in this application may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during the execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from equations, formulas, operations, and operational logic processing.
[0029] In several embodiments, memory internal to the processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be a memory 120 and / or a storage device 140, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations (such as for MPEG-2, HEVC, or VVC).
[0030] Input to the elements of system 100 may be provided through various input devices, as indicated in block 105. Such input devices include, but are not limited to, (i) an RF section that receives an RF signal transmitted over the air, such as by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.
[0031] In various embodiments, the input device of block 105 has corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for the following operations: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select (for example) a signal frequency band, which may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired packet stream. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a frequency band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the desired frequency band again by filtering, down-conversion and perform frequency selection. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some and / or add other elements of similar or different functions in these elements. Adding element can include inserting element between existing elements, for example, inserting amplifier and analog-to-digital converter. In various embodiments, the RF part includes antenna.
[0032] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 100 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, in a separate input processing IC or in the processor 110 as desired. Similarly, aspects of USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 110 as desired. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 110 and an encoder / decoder 130 operating in conjunction with memory and storage elements, to process the data streams as desired for presentation on an output device.
[0033] The various elements of the system 100 may be disposed within an integrated housing in which the various elements may be interconnected and transmit data between them using suitable connection means 115 (eg, an internal bus known in the art, including an I2C bus, wiring, and a printed circuit board).
[0034] The system 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented, for example, within a wired and / or wireless medium.
[0035] In various embodiments, a Wi-Fi network such as IEEE 802.11 is used to stream data to the system 100. The Wi-Fi signals of these embodiments are received through a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet to allow streaming applications and other top communications. Other embodiments provide streaming data to the system 100 using a set-top box that delivers data through an HDMI connection of an input block 105. Still other embodiments provide streaming data to the system 100 using an RF connection of an input block 105.
[0036] The system 100 can provide output signals to various output devices, including a display 165, a speaker 175, and other peripheral devices 185. In various examples of embodiments, the other peripheral devices 185 include one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide functions based on the output of the system 100. In various embodiments, control signals are transmitted between the system 100 and the display 165, the speaker 175, or other peripheral devices 185 using signaling such as AV. Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output device can be communicatively coupled to the system 100 via a dedicated connection through the corresponding interface 160, 170, and 180. Alternatively, the output device can be connected to the system 100 using a communication channel 190 via the communication interface 150. The display 165 and the speaker 175 can be integrated in a single unit with other components of the system 100 in an electronic device (e.g., a television). In various embodiments, the display interface 160 includes a display driver, such as a timing controller (T Con) chip.
[0037] For example, if the RF input section 105 is part of a separate set-top box, the display 165 and speaker 175 may alternatively be separate from one or more of the other components. In various embodiments where the display 165 and speaker 175 are external components, the output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.
[0038] Figure 2 An example of an end-to-end compression system is shown. The input X to the encoder part of the network may include:
[0039] - images or frames of video;
[0040] - a part of an image;
[0041] - a tensor representing a set of images;
[0042] - A tensor representing a portion (crop) of a set of images.
[0043] In each case, the input may have one or more components, for example: monochrome, RGB, or YCbCr components.
[0044] like Figure 2 As shown, the input X is fed to the encoder network g a ()(210, also called analytical transformation). g a () is typically a sequence of convolutional layers with activation functions. Large strides for convolutional or space-to-depth operations can be used to reduce spatial resolution while increasing the number of channels. The encoder network can be viewed as a learned transformation. It should be noted that space-to-depth operations can be implemented by reshaping and permuting, for example, a tensor of size (N, H, W) is reshaped and permuted to (N*2*2, H / / 2, W / / 2), where " / / " represents an integer division operator by discarding the remainder.
[0045] Output of analytical transformation: Z = g a (X), a three-dimensional array or three-dimensional tensor (referred to as a tensor) of latent variables or latent representation Z is quantized (Q) and entropy encoded (EC) (220) into a binary stream (bitstream) for storage or transmission. It should be noted that in the following, it is assumed that EC embeds the quantization operation (Q).
[0046] The bit stream is entropy encoded (ED, 230) to obtain That is, the quantized version of Z. The decoder network g s () (240, also called synthetic transform) generates the reconstruction input: From the quantized latent representation The approximation of the original X is typically a sequence of upsampled convolutions (e.g., “deconvolution” or convolution followed by an upsampled filter) or depth-to-spatial operations. The decoder network can be viewed as learning an inverse transform, or a denoising and generative transform.
[0047] More specifically, the decoder network (210) is typically composed of a sequence of convolutional layers with strides, allowing the spatial resolution of the input to be reduced while increasing the depth (i.e., the number of channels of the input). Pooling (e.g., average pooling, max pooling, etc.) or squeezing operations (space to depth via reshaping and permutation) may also be used instead of strided convolutional layers. The encoder network can be viewed as a learned transformation.
[0048] The output of the analysis (mainly in the form of a 3-dimensional array, called a 3D tensor) is called a latent representation or a tensor of latent variables. From a broader perspective, a set of latent variables builds a latent space, which is also often used in the context of end-to-end compression based on neural networks.
[0049] The latent representation is quantized and entropy encoded for storage / transmission (220). A bitstream is a set of coded syntax elements and a payload representing bins of quantized symbols that is transmitted to a decoder.
[0050] The decoder first decodes the quantized symbols from the bitstream (230). The decoded latent representation is then transformed into pixels for output through a set of layers that typically consist of (de)convolutional layers (or depth-to-space squeezing operations). The decoder network (240) is therefore a learned inverse transform that operates on the quantized coefficients. The output of the decoder is a reconstructed image or a set of images.
[0051] It should be noted that more complex architectures exist, e.g., adding a "super autoencoder" (super prior) to the network to jointly learn a parameterized distribution of the latent representation as the output of the encoder. The approach proposed in this paper is not limited to the use of autoencoders. Any end-to-end differentiable encoder-decoder can be considered.
[0052] Despite the fact that there is no restriction on the input format of the autoencoder as described previously, most existing methods take as input the entire image or frame to be transformed into a latent representation Z, such as Figure 2 As shown in Figure 2. In such cases, the latent representation is represented as a 3D tensor in the latent space by (non)linear transformation with multiple convolutional layers followed by activation to transform the input image. This means that spatial redundancy is decomposed only by learning the transformation operation, which not only limits the coding efficiency but also the application of compression.
[0053] The proposed method aims to improve the rate-distortion performance of compressed images and / or videos using an end-to-end neural network (NN) based on frequency decomposition. In addition, the proposed method enables quality scalability or progressive decoding, for example, in the context of regions of interest of variable quality.
[0054] Since most state-of-the-art end-to-end NN-based image compression methods process the entire image by learning a nonlinear transformation, there is no way to reduce spatial redundancy except to learn a globally optimal transformation to decorrelate the input content. The entire image is transformed and compressed by learning the network; moreover, it is challenging to locally control the quality of the reconstructed input.
[0055] In contrast, most conventional hybrid encoders divide the image into non-overlapping blocks of different sizes to spatially adapt to the content. These encoders can then make decisions on prediction modes and transforms at the block level to minimize spatial redundancy from a rate-distortion optimization perspective. For example, the transform coefficients of each block can be quantized with various quantization step sizes, so that the quality of the reconstructed frame can be optimized for the target bit rate. In addition, by applying different quantization step sizes, the encoder is able to adaptively preserve the texture of the region of interest. Moreover, in the context of conventional image compression, the JPEG2000 standard stands out because it applies the wavelet transform to tiles (i.e., rectangular regions of the image) or the entire image, which enables smooth progressive decoding to adaptively quantize the coefficients across all spatial subbands.
[0056] In this document, an end-to-end NN-based image compression method is first proposed, which involves a frequency decomposition transform (such as a wavelet transform, whose kernel coefficients can be offline learnable or predefined), which decomposes the spatial frequencies of the input content so that various degrees of frequency are captured and grouped in the latent space to be encoded. This method enables the progressive transmission of groups of decomposed frequencies separated in the latent space and produced by the encoder as separate tensors. Then, a set of latent representations or tensors or a stack of latent channels composed of tensors can be decomposed and transmitted in separate bitstreams, thus achieving a quality scalability. The proposed decoder network can use as input a bitstream representing a "base latent representation" or "base tensor" containing low-frequency data to reconstruct a lower quality representation of the input content. Additional frequency data can be decoded from the "enhanced latent representation" or "enhanced tensor" to achieve higher quality.
[0057] It should be noted that in the context of conventional scalable video coding standards like MPEG SVC (Scalable Video Coding) and MPEG SHVC (scalable extension of the H.265 / HEVC standard), a layer refers to one of the outputs of the scalable video compression framework, e.g., in the context of quality scalability, the base layer has the lowest quality and can be refined using enhancement layers, which contain residual information to improve the quality of the reconstructed image. In the context of NN-based codecs, a layer has referred to an operation (convolution, fully connected, activation, etc.). In this document, the term "layer" is used for neural network operations, while a scalable "layer" is represented by its form, i.e., a base (respectively enhanced) "latent representation" or "tensor".
[0058] Additionally, the proposed method enables the decoder to improve the quality of the reconstructed input only in specific regions of interest (ROIs) (e.g., faces, moving objects, etc.) by encoding an additional enhanced latent representation containing the corresponding localized information.
[0059] To specify the ROI to the decoder, the position information can be encoded in the bitstream or can be derived from the decoder itself by performing a computer vision task using some pre-arrived latent representation (e.g., a base latent representation) or a lower quality reconstructed input as input.
[0060] As explained above, conventional image / video encoders divide the input image into large blocks, also called coding tree units (CTUs). Each CTU is then subdivided into smaller sub-blocks based on the effectiveness of prediction and transformation to remove redundancy in local textures. Each sub-block is encoded using a prediction mode selected by the encoder among a predefined canonical mode, a fixed linear transform (e.g., DCT-II), and quantization to minimize the rate-distortion criterion. These processes make image / video coding very flexible in terms of local optimization of bit rate and distortion.
[0061] Most end-to-end compression methods compress the entire input image at once using learned analysis and synthesized nonlinear transforms. In particular, these methods do not include adaptive mechanisms such as prediction modes and various quantization step sizes applied to the transformed coefficients. Therefore, the compression performance depends entirely on the behavior of the learned nonlinear transform. Since the entire input image is processed by a unique learned transform, there is not much flexibility to adapt the quality of the decoded ROI.
[0062] For adaptive output bitstreams, some solutions have been proposed, such as adjusting the transform by rescaling the intermediate latent representation with learned coefficients (per layer), or learning normalization layers per bitrate and switching them dynamically. However, this is limited to the use case of using the same model for a specific bitrate, and can only affect the scaling of the latent representation of the entire input, which has limited impact on improving compression efficiency. However, there are few methods to adaptively decode ROIs with the same model.
[0063] In this document, an end-to-end image / video compression method is proposed that decomposes the spatial frequencies of the input content into partitioned latent representations. The decomposed frequencies in the latent space are analyzed and grouped into separate latent representations or separate tensors, each of which is jointly optimized to be decoded progressively. Therefore, the decoder can decode the tensors in a scalable manner to progressively reconstruct the input. When the tensors are decoded, the tensors can be entropy decoded independently. The method achieves quality scalability by progressively transmitting the individual latent representations of the decomposed frequency data separated in the resulting latent space. The proposed decoder network uses a base latent representation or base tensor as input to reconstruct lower quality, and can potentially decode an additional enhanced latent representation or enhanced tensor containing more frequency data to achieve higher quality.
[0064] Furthermore, quality scalability of regions of interest (ROIs) is achieved, whereby the decoder takes as input only the corresponding latent representation in the augmentation tensor together with the latent representation already delivered to the decoder. The location of the ROI can be specified by explicitly signaling to the decoder. Alternative ROI approaches can be implemented on the decoder side by operating the computer vision algorithm using only the reconstructed image from the previously received latent representation as input or directly using the latent representation as input.
[0065] As mentioned above, an end-to-end NN-based compression method is introduced, which analyzes spatial frequencies through a learning process and groups corresponding latent representations in the latent space so that they can be segmented and transmitted in separate bitstreams. These bitstreams can then be decoded later in a scalable manner. Advantageously, spatial redundancy is reduced by using an explicit analysis of the (learnable) frequency decomposition process, which helps to separate and group related frequency data into multiple tensors or latent representations from various perspectives (e.g., reconstruction quality and multi-tasking). In addition, multiple latent representations can be decoded in a scalable manner. For example, the more latent representations with high-frequency data used by the decoder, the higher the quality of the input reconstruction. In addition, the quality of the reconstructed input can be locally improved by using only the portion of the latent representation corresponding to the position in the enhanced latent representation.
[0066] Progressive encoding in a scalable manner
[0067] exist Figure 3A , let X be the input image of the frequency analysis module (310), which outputs a set of latent representations {Z0, Z1, ..., Z n-1}, where Z i+1 Usually corresponds to frequencies above Z i It should be noted that {Z j} do not necessarily have the same dimensions. (corresponding to each corresponding Z after quantization j ) is entropy encoded (320, 322, 324) to form a bit stream together with necessary metadata, which is transmitted to a decoder.
[0068] At the decoder, the entropy decoder (330, 332, 334) outputs a It is reconstructed and used as input to the frequency synthesis module (340). In the proposed end-to-end method, it is not necessary to decode all to start the reconstruction process. For example, when only When , the remaining latent representations carrying higher frequency information are filled with zeros as input to reconstruct the image with the lowest quality. like Figure 3A The quality of the reconstructed input depends on the encoded information carried by the corresponding latent representation, which is learned through an offline training process.
[0069] Figure 3B shows the entropy coded and exists in the bitstream and is entropy decoded. The decoder then reconstructs It shows that the quality of the reconstructed input is higher than like Figure 3C As shown, when the potential representation of the entire group When available as input, it achieves the best quality of the reconstructed input. Therefore, the decoder reconstructs the
[0070] Figure 4A An embodiment of an encoder is presented that takes as input an image X and outputs a set of latent representations {Z0, Z1, ..., Z n-1 Specifically, the input passes through frequency decomposition (FD) layers (410, 420, 430, 440) which decompose the spatial frequencies and group them into multiple levels {Y0, Y1, ..., Y n-1 Then, each set of frequency data (Y i ) can be transformed by the corresponding analysis ga,k ()(450, 460, 470) is further transformed into a potential representation {Z0, Z1, ..., Z n-1}, where k = {0, 1, ..., n-1}. In general, a set of frequency data (frequency group) can be viewed as a set of multiple frequency bands. For example, Y0 contains low-frequency class bands, while Y1 includes relatively high-frequency class bands. Ideally, there is no redundant frequency information between these two frequency groups. It should be noted that four frequency decomposition layers are shown in FIG4, but any number of FD layers can be used.
[0071] Additional neural networks (such as hyper-prior encoder / decoders and autoregressive models) can also further process the latent representation {Z0, Z1, ..., Z n-1} to improve coding efficiency.
[0072] Figure 4B Renders the decoder as Figure 4A The decoder converts the entropy decoded As input. For each reconstructed latent representation The corresponding synthetic transformation g s,k ()(415, 425, 435) synthesize them separately into Each reconstructed frequency data set is used as input to the frequency combination (FC) layer (445, 455, 465, 475). The last frequency combination layer then outputs the reconstructed input This reconstructed input is also the output of the decoder.
[0073] Figure 5 An embodiment of the proposed compression method including the proposed encoder and decoder architectures that implement the described dual (sub) bitstream transmission is presented. Frequency perceptual analysis at the encoder takes as input an image X and transforms its content using frequency decomposition layers (510, 512, 514, 516). The last frequency decomposition layer (516) outputs tensors Y0 and Y1, from which two latent representations Z0 and Z1 are independently computed using generalized division normalization (GDN) layers (520, 525) followed by convolutional layers (530, 535). The two latent representations Z0 and Z1 are first quantized separately (540) and then entropy coded (550, 555) to generate a base bitstream and an enhancement bitstream, respectively.
[0074] At the decoder side, both bitstreams are entropy decoded (560, 570) to reconstruct the quantized latent representation and to be used as input to the frequency-aware synthesis module. Each latent representation is computed inversely compared to the encoding stage using a convolutional layer (570, 575) followed by an inverse GDN (IGDN, 580, 585). and and uses it as the input to the first frequency combination layer (590). After passing through several frequency combination layers (590, 592, 594, 596), the decoder outputs the reconstruction
[0075] It should be noted that a receiver can only decode the bitstream corresponding to the base layer. In this case, the decoder replaces This produces dimensions and The decoder then simply reconstructs the input with base quality
[0076] Frequency combination (decomposition) layer
[0077] Before further describing the details of the frequency combination (decomposition) layer, the typical process of wavelet transform is briefly explained, that is, transforming the input in the pixel domain into frequencies in the spatiotemporal domain.
[0078] Figure 6 shows an example of a two-level wavelet transform. By frequency decomposition, such as Fig. 6A The input shown First, it is divided into four sub-bands {X 1,LL , X 1,LH , X 1,HL , X 1,HH},like Figure 6B According to the properties of wavelet transform, the height and width of the input are reduced by half, so that X 1,j The dimension becomes where j∈{LL, LH, HL, HH}. The LL band includes low frequencies, and the HH band captures high frequencies in both directions. The LH band carries low frequencies in the horizontal direction but high frequencies in the vertical direction, while the HL band captures high frequencies in the horizontal direction but low frequencies in the vertical direction. 1,LL The next round of decomposition is performed on the same sub-band {X 2,LL , X 2,LH , X 2,HL , X 2,HH},like Figure 6C shown.
[0079] Fig. 7A An embodiment of the frequency decomposition (FD) layer when n=2 is presented. The input of the i-th layer uses two sets of frequency data In the special case when i=0 (i.e., the input to the first layer corresponds to the input image X), the only input is So ignore the remaining and its associated branches. For example, Fig. 7A As shown, when i = C, ignore The associated intra-frequency process (710) and inter-frequency process (720, 730). However, based on Through The associated inter-frequency processes (740, 750) generate And through The associated intra-frequency process (760) generates For other layers, process To generate the output
[0080] To generate output at each layer, each input undergoes two different processes: intra-frequency process and inter-frequency process. The intra-frequency process uses convolutional layers (710, 760) to calculate some information to keep within the frequency group and pass to the next frequency decomposition layer, while the inter-frequency process calculates frequency information to exchange between different frequency groups.
[0081] like Fig. 7A As shown, the frequency decomposition transform is explicitly processed by using a non-learning transform (such as wavelet transform, Fourier transform, discrete cosine transform, etc.) or a trainable version of these conventional transforms. and Then it goes through a typical convolutional layer in the "inter-frequency process" to and Exchange frequency information between them.
[0082] More specifically, if Fig. 7A As shown, and It is used as the input of the wavelet transform (720, 740), then passes through the convolution layer (730, 750) for the inter-frequency process, and then the output of the inter-frequency process flows into its adjacent group to be added (770, 775) to the corresponding output of the intra-frequency process with the convolution layer. In different wavelet transform modules (e.g., 720, 740), different wavelet transforms can be used, or they output different frequency band combinations. For example, for the wavelet (740) with 2-D discrete Haar wavelet transform, only the high-high (HH) frequency band is passed to the next layer. For the wavelet (720) with the same coefficients, only the low-low (LL) frequency band is passed to the next layer. In different convolution layers (e.g., 730, 750), different parameters can be used. The summed output is then processed separately with the activation function (780, 790) to generate the output and The activation function may be any type of non-linear function or even a linear function. Furthermore, an activation function (780) may be different from another activation function (790).
[0083] In this example, The wavelet transform (720) used as input can only calculate the frequency information corresponding to the LL frequency band, and further processes the LL frequency as input to the subsequent convolution layer to pass it to other frequency groups. Depending on the learning filter of the convolution layer, the calculated information can be subtracted from or added to the adjacent frequency groups.
[0084] Likewise, The wavelet transform (740) used as input can only calculate the frequency information corresponding to the HH frequency band, and further processes the HH frequency as the input of the subsequent convolution layer to pass it to other frequency groups. Depending on the learning filter of the convolution layer, the calculated information can be subtracted from or added to the adjacent frequency groups.
[0085] Figure 7B An implementation of a frequency decomposition layer is described. The input and output are depicted as 3D cubes because they generally correspond to 3D tensors. To make the intra-frequency process computation more precise, residual blocks (715, 775) are used. Initially, residual blocks were introduced as part of the ResNet architecture, and there are various layer configurations for building residual blocks. The residual block connects the output of a layer to the input of the block via a skip connection that references the layer input to learn a residual function instead of learning an unreferenced function. In this implementation, the skip connection consists of two convolutional layers and activation functions to retain some information that would be lost when using a non-residual block with a single branch with a regular convolutional layer. However, the proposed method does not impose any restrictions on the layer configuration of the residual block. Since the stride used by the residual block is 2 and the wavelet operation (735, 755) halves the resolution of the input in width and height, the resulting size of the output is smaller than the input. In addition, in this implementation, after the addition operation that combines the outputs of the intra-frequency process and the inter-frequency process, there is no activation function that is theoretically the same as the linear activation layer.
[0086] Figure 7C An embodiment of a generalized frequency decomposition (FD) layer is presented. The input of the i-th layer corresponds to a grouped frequency When i-0, it corresponds to the first layer interacting with the input image X, which is The only input of . Then, the remaining input and associated branches are ignored. After the first layer, the entire input of the i-th layer is processed To generate the output to use it as the input to the next i+1th layer. In the middle of the input, the calculated frequency information is exchanged between the upper and lower groups. So, for example, (k={0, 1, ..., n-1}) carries lower frequency than larger k, which is learned by successive FD layers during training.
[0087] Regarding wavelet modules, they can be implemented using convolutional layers with non-trainable kernels of traditional wavelet transform coefficients. For example, a wavelet module for low frequencies can use a convolutional layer whose kernel size is all one to capture the DC frequency of the input. For wavelet modules for capturing high frequencies, more complex designed wavelet transform coefficients (non-trainable) can be used as the kernel of the convolutional layer.
[0088] Fig. 8A An embodiment of the frequency combination (FC) layer when n=2 is presented. The input of the i-th layer uses two sets of frequency exist Figure 5 In the special case when i=3 (where the layer is assumed to generate outputs), Always the expected output Where k = {0, 1-}.
[0089] To generate an output at each layer, each input undergoes two different processes: an intra-frequency process and an inter-frequency process. The intra-frequency process uses the convolutional layer (810, 840) to calculate some information to keep within the frequency group and pass to the next frequency combination layer, while the inter-frequency process (820, 840) calculates frequency information to exchange between different frequency groups (850, 860). Fig. 8A As shown, and In the inter-frequency process, it is processed by the convolution layer to and The frequency information is exchanged between the convolution layers. Different parameters may be used in different convolution layers (e.g., 820, 830). The summed outputs are then processed separately using activation functions (870, 880) to generate outputs and
[0090] Figure 8BThe implementation of the proposed frequency combination layer is described. In the inter-frequency process, the input is upsampled by a factor of 2 using PixelShuffle (835, 855, see W. Shi et al., "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network", Proceedings of the IEEE CVPR Conference, 2016, pp. 1874-1883), followed by a convolutional layer (845, 865), thereby taking into account multiple frequency features across channels. Since the residual block (815, 875) and the PixelShuffle layer with upsampling operations double the resolution of the input, the size of the output is larger than the input. As in the decomposition layer, after the addition operation of summing the outputs of the intra-frequency process and the inter-frequency process, there is no activation layer that is theoretically the same as the linear activation layer.
[0091] In the example, only yes Figure 5 In the case of the decoder input in , the proposed decoder treats the remaining input as a constant value (e.g., zero), and thus involves the entire branch to compute the output. Unlike the FD layer, no engineered transformation is explicitly used here, as the convolutional layer can figure out how to correctly synthesize the output when trained for a given optimization function.
[0092] Figure 8C An embodiment of a generalized frequency combination (FC) layer is presented. The input to the i-th layer is a grouped frequency exist Figure 5 When i=3 (where the layer is assumed to generate output), Always the expected output where k = {0, 1, ..., n-1}. For example, in the case where only yes Figure 5 In the case of the decoder input in , our proposed decoder treats the remaining input as a constant value (e.g., zero), and thus involves the entire branch to compute the output. After the first layer, the entire input of the i-th layer is processed To generate the output to use it as the input of the next i+1th layer. In the middle of the input, the calculated frequency information is exchanged between the upper and lower groups to synthesize the output image.
[0093] Scalability (or progressive decoding)
[0094] FIG9 shows an example of dual-layer quality scalability according to an embodiment. Fig. 9AAs shown, only the Decoding as input produces lower quality output like Fig. 9B As shown, All latent variables in are assigned zero. Fig. 9C As shown, when available at the decoder and When both are used, and The decoding of the input together improves the output (expressed as ) quality, such as Fig.9D As shown. Hence, the same decoder architecture is used for different quality levels. To clean up the input while providing scalability, on the decoder side, all input ports of the FC should be enabled, but the elements of the input can be zero (for the first FC layer) or vary depending on the desired degree of quality of the output image.
[0095] Quality enhancement in regions of interest
[0096] In one embodiment, we propose to encode and transmit only the latent representation The region of interest (ROI) in the image / video (e.g., the face area, in Fig. 10C The decoder thus receives only the portion containing The encoded latent variables of the part (1030) in are as follows Fig. 10B As shown in Figure 2, where C×l×m is the dimension of the tensor corresponding to object #1 in the latent space. is used as input to the frequency synthesis stage, setting the remaining latent variables to 0(1010), as Fig. 10A As shown. Therefore, the tensor of size C×L×M Feed to the frequency synthesis (1020) to use object #1 The enhanced ROI is used to reconstruct the input image.
[0097] Signaling for ROI
[0098] There are several ways to instruct the decoder to reform the input tensor to use as the ROI information required for the frequency synthesis decoder input. In one embodiment, when the enhancement information of object #1 is encoded as like Fig.11A As shown, the proposed method involves explicitly signaling semantic information, which includes the latent representation The corresponding position information in the original dimension The tensor dimensions of and possible additional semantic information that may be needed in the encoded bitstream (e.g., object category or class ID), such as Fig. 11B Therefore, the decoder is used to parse The encoded ROI information in the bitstream of , and used to specify the latent representation of the reconstruction In potential expression Another enhancement data for another ROI of the second object can also be provided in a separate bitstream. Encoding and transmission, such as Fig. 12A In this case, common information (such as the dimensions of the original tensor) can be referenced The previously encoded data, such as Fig. 12B shown.
[0099] In another embodiment, based on the decoder The inference result of running on , only the corresponding latent variables are requested from the encoder like Fig.13A In this case, the receiver system includes a decoder (1320) and a computer vision algorithm (or network, 1330) that signals the position information of the original input scale and the size of the ROI (1340) to the encoder. The encoder then compresses only the corresponding latent variables Then, when receiving the encoded latent variable The ROI information is also used in the decoder to appropriately fill the remaining latent variables with zeros (1310) and construct the input tensor
[0100] In another embodiment, it is proposed to avoid reconstruction Instead, use The inference is performed as input to a properly trained computer vision network. Therefore, the receiver system still signals the location of the ROI and its area size to the encoder. Therefore, the encoder compresses the corresponding latent variables and transmits the encoded latent variables to the decoder, such as Fig. 13B shown.
[0101] Various numerical values are used in this application. The specific values are only for illustrative purposes, and the described aspects are not limited to these specific values.
[0102] Various methods are described herein, and each of these methods includes one or more steps or actions that realize the described method.Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined.Additionally, in various embodiments, terms such as "first", "second" can be used to modify elements, parts, steps, operations, etc., such as, for example, "first decoding" and "second decoding".Unless specifically required, the use of such terms does not imply that the modified operation is sorted.Therefore, in this example, the first decoding does not need to be performed before the second decoding, but can occur in a time period overlapping with the second decoding, such as before, during, or during the second decoding.
[0103] Various implementations involve decoding. "Decoding" as used in this application may encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, and inverse transformation. Based on the context of the specific description, whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to generally refer to a broader decoding process will be clear and will be well understood by those skilled in the art.
[0104] Various implementations involve encoding.In a similar manner to the above discussion about "decoding", "encoding" as used in this application may encompass all or part of a process performed on an input video sequence, for example, to produce an encoded bitstream.
[0105] The implementations and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if only discussed in the context of a single implementation form (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., devices or programs). The device can be implemented, for example, with appropriate hardware, software, and firmware. The method can be implemented in, for example, a device, for example, a processor that generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, for example, a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.
[0106] Reference to "one embodiment" or "an embodiment" or "an implementation" or "implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations appearing in various places throughout this application are not necessarily all referring to the same embodiment.
[0107] Additionally, the present application may refer to “determining” various information. Determining information may include one or more of the following: for example, estimating information, calculating information, predicting information, or retrieving information from a memory.
[0108] Furthermore, the present application may refer to "accessing" various information. Accessing information may include one or more of: for example, receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.
[0109] Additionally, the present application may involve "receiving" various information. Like "accessing," receiving is a broad term. Receiving information may include one or more of: for example, accessing information or retrieving information (e.g., retrieving from a memory). Moreover, "receiving" is often involved in one way or another during operation, e.g., storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.
[0110] It should be understood that, for example, in the case of "A / B", "A and / or B", and "at least one of A and B", the use of any of the following " / ", "and / or", and "at least one" is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As yet another example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to as many listed items as are apparent to those of ordinary skill in this and related arts.
[0111] As will be apparent to one of ordinary skill in the art, implementations may generate signals formatted to carry a variety of signals that may be, for example, stored or transmitted. Information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.
Claims
1. A video encoding method, comprising: decomposing at least a portion of the image into a plurality of frequency groups, wherein the frequency groups correspond to a set of frequency bands; For each frequency group in the plurality of frequency groups, generating a corresponding latent representation in a latent space; and One or more of the corresponding latent representations are entropy encoded.
2. The method according to claim 1, wherein: The generation is performed by a sequence of convolutional layers with activation functions.
3. The method according to claim 1 or 2, wherein: The decomposition is performed by a plurality of decomposition layers, wherein each decomposition layer performs an intra-frequency process and an inter-frequency process.
4. The method according to claim 3, wherein: The intra-frequency process is performed by a set of convolutional layers.
5. The method according to claim 3, wherein: The inter-frequency process includes: Performing a frequency decomposition transform on an input data set associated with a frequency group, followed by one or more convolutional layers to form an output data set; and The output data set is added to a result of an intra-frequency process of another frequency group adjacent to the frequency group.
6. The method according to claim 5, wherein: The frequency decomposition transform corresponds to a wavelet transform, a Fourier transform, a discrete cosine transform or a learning transform.
7. The method according to any one of claims 1 to 6, further comprising: Position information of at least a region of interest of the image is signaled.
8. A method for video decoding, comprising: obtaining one or more latent representations in a latent space, wherein each of the one or more latent representations corresponds to a frequency group of one or more frequency groups, wherein a frequency group corresponds to a set of frequency bands; deriving the one or more frequency groups from the one or more latent representations; and At least a portion of an image is combined from the one or more frequency groups.
9. The method according to claim 8, wherein: Frequency groups are obtained from the corresponding latent representation by a sequence of convolutional layers with activation functions.
10. The method according to claim 8 or 9, wherein: The combining into the at least a portion of the image is performed by a plurality of frequency combining layers, and wherein each frequency combining layer performs an intra-frequency process as well as an inter-frequency process.
11. The method according to claim 10, wherein: The intra-frequency process is performed by a set of convolutional layers.
12. The method according to claim 10, wherein: The inter-frequency process is performed by a set of convolutional layers.
13. The method according to any one of claims 8 to 12, further comprising: obtaining one or more additional latent representations in the latent space; as well as One or more further frequency groups are obtained from the one or more further latent representations, wherein the at least a portion of the image is further combined based on the one or more further frequency groups.
14. The method according to any one of claims 8 to 13, further comprising: Position information of at least a region of interest of the image is received.
15. The method according to any one of claims 13, wherein: The further one or more potential representations associated with the further one or more frequency groups are set to a constant value.
16. The method according to any one of claims 8 to 15, wherein: At least one of the one or more potential representations contains only information of the region of interest.
17. The method according to claim 16, wherein: The latent variables outside the region of interest are set to zero in the at least one latent representation of the one or more latent representations.
18. The method according to any one of claims 8 to 13, further comprising: obtaining information about the region of interest based on the combined image or the one or more latent variables; as well as Latent variables associated with the region of interest are requested, wherein the images are further combined in response to the requested latent variables.
19. A device comprising one or more processors and at least one memory, the at least one memory being coupled to the one or more processors, wherein the one or more processors are configured to perform the method according to any one of claims 1 to 18.
20. A signal comprising video data formed by performing a method according to any one of claims 1 to 7.
21. A computer-readable storage medium storing instructions for encoding or decoding a video according to the method according to any one of claims 1 to 18.
Citation Information
Cited By
Multi-granularity generation type video compression method
CN120343255A
Multi-granularity generative video compression method
CN120343255B
