Method and apparatus for image encoding and decoding

Abstract

The invention relates to an image decoding method and apparatus for a system using a neural network. The first decoding parameter is used for controlling image compression quality during image coding, and the value of the first decoding parameter is smaller than a preset minimum value or larger than a preset maximum value. Specifically, a target gain vector is obtained based on the first decoding parameter, and then the target gain vector is used for encoding the image to obtain a code stream. The first decoding parameter is indicated in the code stream and is transmitted to a decoding side, so that the code stream can be correctly decoded. Therefore, according to the method, the compression quality or the code stream size can be flexibly set, and a new decoding model does not need to be trained.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of encoding and decoding data based on neural network architectures. In particular, some embodiments relate to methods and apparatus for encoding and decoding images and / or videos from a bitstream using multiple processing layers. Background Art

[0002] Hybrid image and video codecs have been used for decades to compress image and video data. In these codecs, the signal is usually encoded block by block by predicting the block and further decoding only the difference between the original block and its prediction. In particular, this decoding may include transformation, quantization and generation of a bitstream, usually including some entropy decoding. Usually, the three components of the hybrid decoding method (i.e., transformation, quantization and entropy decoding) are optimized separately. Modern video compression standards (such as high-efficiency video coding (HEVC), versatile video coding (VVC) and essential video coding (EVC)) also use the transformed representation to encode the predicted residual signal.

[0003] Recently, neural network architectures have been applied to image and / or video decoding. In general, these neural network (NN) based methods can be applied to image and video decoding in various different ways. For example, some end-to-end optimized image or video decoding frameworks have been discussed. In addition, deep learning has been used to determine or optimize some parts of the end-to-end decoding framework, such as the selection or compression of prediction parameters. In addition, some neural network based methods have been discussed for hybrid image and video decoding frameworks, for example, for implementation as a trained deep learning model for intra-frame or inter-frame prediction in image or video decoding.

[0004] What the end-to-end optimized image or video decoding applications discussed above have in common is that they generate some feature map data that will be transmitted between the encoder and the decoder.

[0005] Neural networks are machine learning models that use one or more layers of nonlinear units based on which they can predict outputs for received inputs. Some neural networks include one or more hidden layers in addition to an output layer. A corresponding feature map can be provided as an output for each hidden layer. Such corresponding feature maps for each hidden layer can be used as inputs for subsequent layers in the network (i.e., subsequent hidden layers or output layers). Each layer of the network generates an output from the received input based on the current values ​​of the corresponding set of parameters. In a neural network that is partitioned between devices (e.g., between an encoder and a decoder, between a device and the cloud, or between different devices), the feature maps on the output side of the partition location (e.g., a first device) are compressed and transmitted to the remaining layers of the neural network (e.g., a second device).

[0006] Typically, models are trained to achieve a desired compression quality or bitstream size, and if a different compression quality is required, a new model needs to be trained, which takes a lot of time and computational cost. In addition, the amount of storage required increases with the number of models.

[0007] It is expected that further improvements in encoding and decoding can be made using the trained network architecture. Summary of the invention

[0008] The present invention provides a method and apparatus for improving the flexibility of a pre-trained image or video decoding model, thereby reducing the storage and transmission costs of the encoded image or video without sacrificing image quality, or reducing the distortion of the reconstructed image without increasing bits.

[0009] The above and other objects are achieved by the subject matter claimed in the independent claims. Other implementations are apparent from the dependent claims, the description and the drawings.

[0010] Particular embodiments are outlined in the accompanying independent claims, further embodiments are outlined in the dependent claims.

[0011] According to a first aspect, the present invention relates to a method for encoding an image using a neural network. The method may be performed by an encoding device. The method comprises: obtaining an image; obtaining a first decoding parameter for the image, wherein the value of the first decoding parameter is less than a preset minimum value or greater than a preset maximum value; obtaining a target gain vector based on the first decoding parameter; encoding the image based on the target gain vector. The first decoding parameter (denoted as β) is used to select compression quality. The larger β is, the larger the code stream size is, and the higher the quality of the reconstructed data is.

[0012] This method can provide more flexibility for the pre-trained encoding model because it can encode the image with any desired compression quality or bitstream size without training a new encoding model, especially when the compression quality indicated by the first decoding parameter is outside the pre-trained range. By using this method, the pre-trained model can be used to encode the image with any desired quality and can be flexibly deployed in various scenarios.

[0013] The above and other embodiments may optionally include one or more of the following features alone or in combination:

[0014] In a possible implementation, the minimum value β is preset s and the preset maximum value β t Stored on the encoding device. During the training phase of the encoding model (e.g. Figure 3A The encoder 101 in [β s , β t ] are input into the model and trained to obtain the corresponding gain vector. Then, the pre-trained possible value set of β and the corresponding gain vector are obtained and stored in the encoding device and the decoding device. The preset minimum value β s is the lower boundary of the range of β, and the preset maximum value β t is the upper boundary of the range of β.

[0015] In a possible implementation, when the value of the first decoding parameter is less than the preset minimum value, obtaining the target gain vector based on the first decoding parameter includes: obtaining the target gain vector based on the first decoding parameter, the preset minimum value and a first gain vector corresponding to the preset minimum value.

[0016] This method can encode images with a smaller code stream size and lower compression quality, which can meet the decoding requirements under poor communication network conditions.

[0017] In a possible implementation manner, the preset minimum value and the first gain vector corresponding to the preset minimum value are stored in the encoding device or the remote device in the form of a table.

[0018] In a possible implementation, the preset minimum value and its relationship with the first gain vector are stored in the encoding device or the remote device. The relationship may be in the form of a function or a network model.

[0019] In a possible implementation, obtaining the target gain vector based on the first decoding parameter, the preset minimum value and the first gain vector includes: obtaining a first ratio of the first decoding parameter to the preset minimum value; and obtaining the target gain vector based on the first ratio and the first gain vector.

[0020] In a possible implementation, obtaining the target gain vector based on the first ratio and the first gain vector includes: multiplying the first ratio by the first gain vector to obtain the target gain vector. The multiplication may be an element-by-element multiplication operation.

[0021] In a possible implementation manner, the target gain vector satisfies the following conditions:

[0022]

[0023] Among them, m v is the target gain vector, β s is the preset minimum value, β v is the first decoding parameter, m s is the first gain vector corresponding to the preset minimum value, K is a hyperparameter and may have a preset value. The default value of K may be 1.

[0024] In a possible implementation, when the value of the first decoding parameter is greater than the preset maximum value, obtaining the target gain vector based on the first decoding parameter includes: obtaining the target gain vector based on the first decoding parameter, the preset maximum value and a second gain vector corresponding to the preset maximum value.

[0025] This method can encode images with higher compression quality, which can meet the decoding requirements of some high-quality decoding situations, such as decoding high-definition movies or real-time transmission of sports events.

[0026] In a possible implementation manner, the preset maximum value and the second gain vector corresponding to the preset maximum value are stored in a table on the encoding device or the remote device.

[0027] In a possible implementation, the preset maximum value and its relationship with the second gain vector are stored in the encoding device or the remote device. The relationship may be in the form of a function or a network model.

[0028] In a possible implementation, obtaining the target gain vector based on the first decoding parameter, the preset maximum value and the second gain vector includes: obtaining a second ratio of the first decoding parameter to the preset maximum value; and obtaining the target gain vector based on the second ratio and the second gain vector.

[0029] In a possible implementation, obtaining the target gain vector based on the second ratio and the second gain vector includes: multiplying the second ratio by the second gain vector to obtain the target gain vector. The multiplication may be an element-by-element multiplication operation.

[0030] In a possible implementation manner, the target gain vector satisfies the following conditions:

[0031]

[0032] Among them, m v is the target gain vector, β t is the preset maximum value, β v is the first decoding parameter, m t is the second gain vector corresponding to the preset maximum value, K is a hyperparameter, and its value can be preset. The default value of K can be 1. In a possible design, the value of K can depend on β v .

[0033] In a possible implementation, when the value of the first decoding parameter is less than the preset minimum value, obtaining the target gain vector based on the first decoding parameter includes: obtaining the target gain vector based on the first decoding parameter, the preset minimum value, a first gain vector corresponding to the preset minimum value, a third preset value closest to the preset minimum value, and a third gain vector corresponding to the third preset value. The third preset value is a pre-trained value closest to the preset minimum value among several pre-trained β value sets.

[0034] This method can encode an image with a smaller bitstream size and relatively good compression quality by using two closest preset values ​​of the decoding parameters and their corresponding gain vectors.

[0035] In a possible implementation, when the value of the first decoding parameter is greater than the preset maximum value, obtaining the target gain vector based on the first decoding parameter includes: obtaining the target gain vector based on the first decoding parameter, the preset maximum value, a second gain vector corresponding to the preset maximum value, a fourth preset value closest to the preset maximum value, and a fourth gain vector corresponding to the fourth preset value. The fourth preset value is a pre-trained value closest to the preset maximum value among several pre-trained β value sets.

[0036] This method can encode an image with higher compression quality by using two closest preset values ​​of the decoding parameters and their corresponding gain vectors.

[0037] In a possible implementation, obtaining the target gain vector based on the first decoding parameter includes: obtaining the target gain vector based on the first decoding parameter, N preset values ​​and N gain vectors corresponding to the N preset values, wherein N is an integer greater than 2, and the N preset values ​​include the preset minimum value and / or the preset maximum value.

[0038] This approach can provide higher coding performance by using more pre-trained β values ​​and their corresponding gain vectors.

[0039] According to the first aspect and any possible implementation manner thereof, in a possible implementation manner, encoding the image based on the target gain vector includes: obtaining a first feature map of the image using a neural network; obtaining a second feature map based on the first feature map and the target gain vector; quantizing the second feature map to obtain a quantized second feature map; encoding the quantized second feature map to obtain a code stream. For example, the quantized second feature map can be encoded using entropy coding.

[0040] In a possible implementation, obtaining the second feature map based on the first feature map and the target gain vector includes: multiplying the target gain vector by the first feature map. The multiplication may be an element-by-element multiplication operation.

[0041] In one possible implementation, the first feature map is a tensor of shape w×h×d, the target gain vector is a vector of dimension 1×d, w and h represent the width and height of the first feature map, and d represents the number of channels of the first feature map. The gain vector can be regarded as part of the model weight.

[0042] In a possible implementation manner, when the first feature map is a feature map of brightness samples of the image, d is equal to 128.

[0043] In a possible implementation, when the first feature map is a feature map of chrominance samples of the image, d is equal to 64.

[0044] In a possible implementation, the method further includes obtaining a second decoding parameter for the image; when the first decoding parameter is used to encode a luma sample of the image, the second decoding parameter is used to encode a chroma sample of the image; or when the first decoding parameter is used to encode a chroma sample of the image, the second decoding parameter is used to encode a luma sample of the image. At least one of the value of the first decoding parameter and the value of the second decoding parameter is less than a preset minimum value or greater than a preset maximum value.

[0045] In one possible implementation, the method further includes: encoding the first decoding parameter into a bitstream. In one possible design, the first decoding parameter is directly encoded into the bitstream. In another possible design, the first decoding parameter is encoded into the bitstream as a first flag (e.g., a base-2 logarithm of the first decoding parameter) to save bits, wherein the first flag can be used to derive the first decoding parameter.

[0046] In a possible implementation manner, the first decoding parameter is encoded in a picture parameter set (PPS) of the code stream.

[0047] In a possible implementation, the number of bits in the code stream used to signal the first decoding parameter is less than or equal to 16. When the first decoding parameter is derived using the first flag, the number of bits in the code stream used to signal the first decoding parameter can even be reduced to less than or equal to 4.

[0048] In a possible implementation, the image includes luma samples and chroma samples, and the second decoding parameter is also indicated in the bitstream, when the first decoding parameter is used to encode the luma samples of the image, the second decoding parameter is used to encode the chroma samples of the image; or when the first decoding parameter is used to encode the chroma samples of the image, the second decoding parameter is used to encode the luma samples of the image. At least one of the values ​​of the first decoding parameter and the second decoding parameter is less than a preset minimum value or greater than a preset maximum value.

[0049] According to a second aspect, the present invention relates to a method for decoding a code stream to obtain an image. The method is performed by a decoding device. The method comprises: obtaining a code stream including decoded image data; parsing the code stream to obtain a first decoding parameter, wherein the value of the first decoding parameter is less than a preset minimum value or greater than a preset maximum value; obtaining a target inverse gain vector based on the first decoding parameter; and obtaining an image based on the target inverse gain vector.

[0050] This method can provide more flexibility for the pre-trained decoding model because it can decode the image at any desired compression quality or bitstream size without training a new decoding model, especially when the compression quality indicated by the first decoding parameter is outside the pre-trained range. By using this method, the pre-trained model can be used to encode or decode the image at any desired quality and can be flexibly deployed in various scenarios.

[0051] The above and other embodiments may optionally include one or more of the following features alone or in combination:

[0052] In a possible implementation, the minimum value β is preset s and the preset maximum value β t Stored on the decoding device. During the training phase of the decoding model (e.g. Figure 3A The decoder 104 in [β s , β t ] are input into the model and trained to obtain the corresponding gain vector. Then, the pre-trained possible value set of β and the corresponding gain vector are obtained and stored in the encoding device and the decoding device. The preset minimum value β s is the lower boundary of the range of β, and the preset maximum value β t is the upper boundary of the range of β.

[0053] In a possible implementation, when the value of the first decoding parameter is less than the preset minimum value, obtaining the target inverse gain vector based on the first decoding parameter includes: obtaining the target inverse gain vector based on the first decoding parameter, the preset minimum value and a first gain vector corresponding to the preset minimum value.

[0054] This method can decode images with smaller bitstream size and lower compression quality. This can meet the decoding requirements under poor communication network conditions. By using this method, the pre-trained model can be used to encode or decode images with any desired quality and can be flexibly deployed in various scenarios.

[0055] In a possible implementation manner, the preset minimum value and the first gain vector corresponding to the preset minimum value are stored in the decoding device or the remote device in the form of a table.

[0056] In a possible implementation, the preset minimum value and its relationship with the first gain vector are stored in the decoding device or the remote device. The relationship may be in the form of a function or a network model.

[0057] In a possible implementation, obtaining the target inverse gain vector based on the first decoding parameter, the preset minimum value and the first gain vector includes: obtaining a first ratio of the first decoding parameter to the preset minimum value; and obtaining the target inverse gain vector based on the first ratio and the first gain vector.

[0058] In a possible implementation manner, obtaining the target inverse gain vector based on the first ratio and the first gain vector includes: multiplying the first ratio by the first gain vector to obtain a target gain vector; and obtaining the target inverse gain vector based on the target gain vector.

[0059] In a possible implementation manner, the target gain vector satisfies the following conditions:

[0060]

[0061] Among them, m v is the target gain vector, β s is the preset minimum value, β v is the first decoding parameter, m s is the first gain vector corresponding to the preset minimum value, K is a hyperparameter, and the value of K can be preset. The default value of K can be 1.

[0062] In a possible implementation, the value of the first decoding parameter is greater than a preset maximum value, and obtaining the target inverse gain vector based on the first decoding parameter includes: obtaining the target inverse gain vector based on the first decoding parameter, the preset maximum value and a second gain vector corresponding to the preset maximum value.

[0063] This approach is able to decode images that were encoded with even higher compression quality by using the two closest preset values ​​of the decoding parameters and their corresponding gain vectors. By using this approach, the pre-trained model can be used to encode or decode images with any desired quality and can be flexibly deployed in various scenarios.

[0064] In a possible implementation, obtaining the target inverse gain vector based on the first decoding parameter, the preset maximum value and the second gain vector includes: obtaining a second ratio of the first decoding parameter to the preset maximum value; and obtaining the target inverse gain vector based on the second ratio and the second gain vector.

[0065] In a possible implementation, obtaining the target inverse gain vector based on the second ratio and the second gain vector includes: multiplying the second ratio by the second gain vector to obtain a target gain vector; and obtaining the target inverse gain vector based on the target gain vector.

[0066] In a possible implementation manner, the target gain vector satisfies the following conditions:

[0067]

[0068] Among them, m v is the target gain vector, β t is the preset maximum value, β v is the first decoding parameter, m t is the second gain vector corresponding to the preset maximum value, K is a hyperparameter, which may have a preset value. The default value of K may be 1.

[0069] In a possible implementation manner, the target inverse gain vector satisfies the following conditions:

[0070] m′ v *m v =C,

[0071] Among them, m' v is the target inverse gain vector, n v is the target gain vector, C is a vector whose elements are all constants, and * represents an element-by-element multiplication operation. In a possible design, C is a constant, and * represents a point multiplication operation.

[0072] In a possible implementation, the value of the first decoding parameter is less than the preset minimum value, and obtaining the target inverse gain vector based on the first decoding parameter includes: obtaining the target inverse gain vector based on the first decoding parameter, the preset minimum value, a first gain vector corresponding to the preset minimum value, a third preset value closest to the preset minimum value, and a third gain vector corresponding to the third preset value.

[0073] In a possible implementation, when the value of the first decoding parameter is greater than a preset maximum value, obtaining the target inverse gain vector based on the first decoding parameter includes: obtaining the target inverse gain vector based on the first decoding parameter, the preset maximum value, a second gain vector corresponding to the preset maximum value, a fourth preset value closest to the preset maximum value, and a fourth gain vector corresponding to the fourth preset value.

[0074] In one possible implementation, obtaining the target inverse gain vector based on the first decoding parameter includes: obtaining the target inverse gain vector based on the first decoding parameter, N preset values ​​and N gain vectors corresponding to the N preset values, wherein N is an integer greater than 2, and the N preset values ​​include the preset minimum value and / or the preset maximum value.

[0075] In one possible implementation, decoding the bitstream based on the target inverse gain vector to obtain an image includes: parsing the bitstream using entropy decoding to obtain a first potential representation of the image; obtaining a second potential representation based on the first potential representation and the target inverse gain vector; and decoding the second potential representation using a neural network to obtain the image.

[0076] In a possible implementation, obtaining the second potential representation based on the first potential representation and the target inverse gain vector includes: multiplying the target inverse gain vector by the first potential representation. The multiplication may be an element-by-element multiplication operation.

[0077] In one possible implementation, the first potential representation is a tensor of shape w×h×d, the target inverse gain vector is a vector of dimension 1×d, w and h represent the width and height of the first feature map, and d represents the number of channels of the first feature map.

[0078] In one possible implementation, when the first latent representation is a feature map of brightness samples of the image, d is equal to 128.

[0079] In one possible implementation, when the first latent representation is a feature map of chrominance samples of the image, d is equal to 64.

[0080] In a possible implementation, the method further includes parsing the code stream to obtain a second decoding parameter; when the first decoding parameter is used to decode the luminance sample of the image, the second decoding parameter is used to decode the chrominance sample of the image; or, when the first decoding parameter is used to decode the chrominance sample of the image, the second decoding parameter is used to decode the luminance sample of the image.

[0081] The proposed method can decode images with luma samples and chroma samples compressed with different qualities.

[0082] According to a third aspect, the present invention relates to a device / equipment for decoding an image or video. This device for decoding can refer to the same advantageous effects as the method for decoding according to the second aspect. It will not be repeated here. The decoding device provides a technical means for implementing the operations in the method according to the second aspect. The function can be implemented by hardware, or by hardware executing the corresponding software. In one possible implementation, the decoding device / equipment includes: an entropy decoding module for parsing a code stream using entropy decoding to obtain a first potential representation of an image; an inverse gain unit for obtaining a second potential representation based on the first potential representation and a target inverse gain vector; an image reconstruction module for decoding the second potential representation using a neural network to obtain the image. These modules can be used to provide corresponding functions corresponding to the method example according to the second aspect. Specific reference is made to the detailed description in the method example. It will not be repeated here.

[0083] In a possible implementation manner, the decoding device further includes an inverse gain vector acquisition module, configured to obtain the target inverse gain vector based on the first decoding parameter.

[0084] According to a fourth aspect, the present invention relates to a device / equipment for encoding an image or video. This device for encoding can refer to the same advantageous effects as the method for encoding according to the first aspect. It will not be repeated here. The encoding device provides a technical means for implementing the operations in the method according to the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. In one possible implementation, the encoding device includes: a feature map acquisition module for obtaining a first feature map from an input image; a gain unit for transforming the first feature map based on a target gain vector to obtain a second feature map; a quantization module for quantizing the second feature map to obtain a quantized second feature map; an entropy coding module for encoding (for example, by entropy coding) the quantized second feature map to obtain a code stream. These modules can be used to provide corresponding functions corresponding to the method example according to the first aspect. Specific reference is made to the detailed description in the method example. It will not be repeated here.

[0085] In a possible implementation manner, the encoding device may further include a gain vector acquisition module, configured to obtain the target gain vector based on the first decoding parameter.

[0086] The method according to the first aspect of the present invention may be performed by the apparatus according to the fourth aspect of the present invention. Other features and implementations of the method according to the first aspect of the present invention correspond to the corresponding features and implementations of the apparatus according to the fourth aspect of the present invention. The advantages of the method according to the first aspect may be the same as the advantages of the corresponding implementations of the apparatus according to the fourth aspect.

[0087] The method according to the second aspect of the present invention may be performed by the apparatus according to the first aspect of the present invention. Other features and implementations of the method according to the second aspect of the present invention correspond to the corresponding features and implementations of the apparatus according to the third aspect of the present invention. The advantages of the method according to the second aspect may be the same as the advantages of the corresponding implementations of the apparatus according to the third aspect.

[0088] According to a fifth aspect, the present invention relates to a video stream or image decoding device, comprising a processor and a memory, wherein the memory stores instructions, and the instructions enable the processor to execute the method according to the second aspect.

[0089] According to a sixth aspect, the present invention relates to a video stream or image encoding device, comprising a processor and a memory, wherein the memory stores instructions, and the instructions enable the processor to execute the method according to the first aspect.

[0090] According to a seventh aspect, there is provided a computer-readable storage medium storing instructions, wherein when executed, the instructions cause one or more processors to encode video or image data. The instructions cause the one or more processors to perform the method according to the first aspect or the second aspect or any possible embodiment of the first aspect or the second aspect.

[0091] According to an eighth aspect, the present invention relates to a computer program, comprising a program code, wherein the program code is used to perform the method according to the first aspect or the second aspect or any possible embodiment of the first aspect or the second aspect when executed in a computer.

[0092] According to a ninth aspect, the present invention relates to a decoder, comprising a processing circuit, wherein the processing circuit is used to execute the method according to the first aspect or the second aspect or any possible embodiment of the first aspect or the second aspect.

[0093] According to a tenth aspect, the present invention relates to a storage medium, wherein the storage medium stores a code stream obtained by using the method according to the first aspect or any possible embodiment of the first aspect.

[0094] According to an eleventh aspect, the present invention relates to a storage medium, wherein the storage medium stores a code stream, and the code stream can be decoded by using the method according to the second aspect or any possible embodiment of the second aspect.

[0095] According to a twelfth aspect, the present invention relates to an encoded code stream, the encoded code stream comprising decoded image data and a plurality of syntax elements, wherein the plurality of syntax elements comprises a first flag (e.g., compression_quality_level), the first flag indicating the compression quality of the decoded image data. The encoded code stream can be obtained by executing the method according to the first aspect of the present invention or any possible embodiment of the first aspect. The encoded code stream can be decoded by executing the method according to the second aspect of the present invention or any possible embodiment of the second aspect.

[0096] According to a thirteenth aspect, the present invention relates to an encoded code stream, the encoded code stream comprising decoded image data and a plurality of syntax elements, wherein the plurality of syntax elements comprises a first flag (e.g., compression_quality_level_luma) and a second flag (e.g., compression_quality_level_chroma), wherein the first flag indicates the compression quality of the luminance samples of the decoded image data, and the second flag indicates the compression quality of the chrominance samples of the decoded image data. The encoded code stream can be obtained by executing the method according to the first aspect of the present invention or any possible embodiment of the first aspect. The encoded code stream can be decoded by executing the method according to the second aspect of the present invention or any possible embodiment of the second aspect.

[0097] According to the fourteenth aspect, the present invention relates to a decoding device, comprising: a receiving unit, used to receive an image for encoding or receive a code stream for decoding; a sending unit, coupled to the receiving unit, the sending unit is used to send the code stream to a decoder or send the decoded image to a display; a memory, coupled to at least one of the receiving unit or the sending unit, the memory is used to store instructions; a processor, coupled to the memory, the processor is used to execute the instructions stored in the memory to execute the method according to the first aspect or the second aspect or any possible embodiment of the first aspect or the second aspect.

[0098] According to the fifteenth aspect, the present invention relates to a decoding system, comprising: an encoder; a decoder, communicating with the encoder, wherein the encoder or the decoder comprises the decoding device according to the third aspect or the fifth aspect of the present invention, the encoding device according to the fourth aspect or the sixth aspect of the present invention, or the decoding device according to the fourteenth aspect of the present invention.

[0099] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages are apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, in which:

[0101] Figure 1 is a schematic diagram of the channels processed by the layers of a neural network;

[0102] Figure 2 is a diagram of the autoencoder type of neural network;

[0103] Figure 3A is a schematic diagram of an exemplary network architecture including an encoding end and a decoding end of a hyper-prior model;

[0104] Figure 3B is a schematic diagram of a general network architecture including the encoding side of the hyper-prior model;

[0105] Figure 3C is a schematic diagram of a general network architecture including a decoding end of a hyper-prior model;

[0106] Figure 4 is a schematic diagram of an exemplary network architecture including an encoding end and a decoding end of a hyper-prior model;

[0107] Figure 5 is a block diagram of the structure of a cloud-based solution for machine-based tasks such as machine vision tasks;

[0108] Fig. 6A is a block diagram of a neural network-based end-to-end video compression framework;

[0109] Figure 6B is a block diagram of some exemplary details of the application of a neural network for motion field compression;

[0110] Figure 6C is a block diagram of some exemplary details of the application of a neural network for motion compression;

[0111] Figure 7 is a flow chart of an exemplary method for encoding;

[0112] Figure 8 is a schematic diagram of the relationship between the first decoding parameter and the gain vector;

[0113] Fig. 9 is a flow chart of an exemplary method for encoding an image based on a target gain vector;

[0114] Fig.10 is a schematic diagram of an exemplary network architecture including an encoding end and a decoding end of a gain unit;

[0115] Fig.11 is a flow chart of an exemplary method for decoding;

[0116] Fig.12 is a flow chart of an exemplary method for decoding a bitstream based on a target gain vector;

[0117] Fig.13 is a block diagram of an example of a video encoding device for implementing an encoding embodiment of the present invention;

[0118] Fig.14 is a block diagram of an example of a video decoding device for implementing a decoding embodiment of the present invention;

[0119] Fig.15 is a block diagram of an example of a video decoding system for implementing an embodiment of the present invention;

[0120] Fig.16 is a block diagram of another example of a video decoding system for implementing an embodiment of the present invention;

[0121] Fig.17 is a block diagram of an example of an encoding device or a decoding device;

[0122] Fig.18 is a block diagram of another example of an encoding device or a decoding device.

[0123] Similar reference numbers and designations in different drawings may indicate similar elements. DETAILED DESCRIPTION

[0124] In the following description, reference is made to the accompanying drawings that form a part of the present invention, which illustrate specific aspects of embodiments of the present invention or specific aspects in which embodiments of the present invention may be used by way of illustration. It should be understood that embodiments of the present invention may be used in other aspects and include structural or logical changes not described in the accompanying drawings. Therefore, the following specific embodiments should not be understood in a restrictive sense, and the scope of the present invention is defined by the appended claims.

[0125] For example, it should be understood that the disclosure related to the described method may also be true for the corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units (e.g., functional units) to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units each perform one or more steps in multiple steps), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, if a specific device is described based on one or more units (e.g., functional units), the corresponding method may include a step to perform the function of one or more units (e.g., one step performs the function of one or more units, or multiple steps respectively perform the function of one or more units in multiple units), even if the one or more steps are not explicitly described or shown in the drawings. In addition, it should be understood that, unless otherwise explicitly stated, the features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0126] In the specification, claims and drawings of the present application, the terms "first", "second", etc. are intended to distinguish similar objects, but do not necessarily indicate a specific order or sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, which is merely a way of distinguishing used when describing objects with the same attributes in the embodiments of the present application. In addition, the terms "including", "having" and any other variants thereof represent non-exclusive inclusion, so that a process, method, system, product or device including a series of units is not necessarily limited to these units, but may include other units that are not explicitly listed or inherent to such processes, methods, products or devices.

[0127] In this specification, claims and drawings, the term "and / or" is only used to describe the association relationship of associated objects. The term "and / or" indicates that there may be three relationships. For example, A and / or B can represent three situations: only A exists, both A and B exist, and only B exists.

[0128] In the following, an overview of some used technical terms and frameworks in which embodiments of the present invention may be employed is provided.

[0129] Artificial Neural Networks

[0130] An artificial neural network (ANN), or connectionist system, is a computing system inspired by the biological neural networks that make up animal brains. These systems "learn" to perform tasks by considering examples and are usually not programmed with task-specific rules. For example, in image recognition, an ANN might learn to recognize images containing cats by analyzing example images that have been manually labeled as "cat" or "no cat" and using the results to recognize cats in other images. The ANN does this without any prior knowledge about cats, such as that they have fur, tails, whiskers, and cat-like faces. Instead, the ANN automatically generates identifying features from the examples they process.

[0131] ANNs are based on a collection of connected units or nodes called artificial neurons, which roughly model neurons in biological brains. Each connection, like a synapse in a biological brain, can send signals to other neurons. The artificial neuron that receives the signal then processes it and can send signals to the neurons it is connected to.

[0132] In an ANN implementation, the "signals" at the connections are real numbers, and the output of each neuron is computed as some nonlinear function of the sum of its inputs. These connections are called edges. Neurons and edges typically have weights that are adjusted as learning proceeds. Weights increase or decrease the strength of the signal at a connection. Neurons can have thresholds so that they are sent only when the aggregate signal exceeds that threshold. Typically, neurons are clustered into layers. Different layers can perform different transformations on their inputs. Signals travel from the first layer (the input layer) to the last layer (the output layer), possibly after traversing the layers multiple times.

[0133] The original goal of the ANN approach was to solve problems in the same way as the human brain. Over time, the focus shifted to performing specific tasks, leading to a departure from the biological domain. ANNs have been used for a variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, board and video games, medical diagnosis, and even in activities traditionally thought of as performed by humans, such as painting.

[0134] The name "convolutional neural network" (CNN) indicates that the network uses a mathematical operation called convolution. Convolution is a specialized linear operation. A convolutional network is a neural network that uses convolution instead of general matrix multiplication in at least one of its layers.

[0135] Figure 1Schematic diagram showing the general concept of processing by a neural network such as CNN. A convolutional neural network consists of an input layer and an output layer and multiple hidden layers. The input layer is where the input (e.g. Figure 1 The hidden layers of a CNN usually consist of a series of convolutional layers that are convolutional with multiplication or other dot convolutions. The result of the layer is one or more feature maps ( Figure 1 Some or all layers may involve subsampling. As a result, feature maps may become smaller, such as Figure 1 As shown. The activation function in a CNN is usually a rectified linear unit (ReLU) layer, followed by additional convolutional layers such as pooling layers, fully connected layers, and normalization layers, which are called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. Although these layers are colloquially called convolutional layers, this is just by convention. Mathematically speaking, it is technically a sliding dot product or cross correlation. This has important implications for the indexing in the matrix because it affects how the weights are determined at a specific index point.

[0136] When programming a CNN to process an image, such as Figure 1 As shown, the input is a tensor of shape (number of images) × (image width) × (image height) × (image depth). It should be known that the image depth can be composed of the channels of the image. After passing through the convolution layer, the image is abstracted into a feature map with the shape of (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolution layer in the neural network should have the following properties. The convolution kernel defined by width and height (hyperparameters). The number of input channels and output channels (hyperparameters). The depth of the convolution filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0137] In the past, traditional multilayer perceptron (MLP) models have been used for image recognition. However, due to the full connectivity between nodes, they suffer from high dimensionality and do not scale well to high-resolution images. A 1000×1000 pixel image with RGB color channels has 3 million weights, which are too high to be efficiently processed at scale with full connectivity. Furthermore, this network architecture does not account for the spatial structure of the data, treating input pixels that are far apart in the same way as pixels that are close together. This ignores the locality of reference in the image data, both computationally and semantically. Therefore, for purposes such as image recognition, which are dominated by spatially local input patterns, full connectivity of neurons is wasteful.

[0138] Convolutional Neural Networks are biologically inspired variants of Multilayer Perceptrons specifically designed to mimic the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by exploiting the strong spatially local correlations present in natural images. Convolutional layers are the core building blocks of CNNs. The parameters of a layer consist of a set of learnable filters (the aforementioned kernels) that have small receptive fields but extend to the full depth of the input volume. During the forward pass, each filter is convolved with the width and height of the input volume, the dot product between the filter entries and the input is computed, and a 2D activation map of that filter is generated. Thus, the network learns filters that activate when it detects certain specific types of features at a certain spatial location in the input.

[0139] Stacking the activation maps of all filters along the depth dimension forms the complete output volume of the convolutional layer. Therefore, each entry in the output volume can also be interpreted as the output of a neuron that looks at a small region in the input and shares parameters with neurons in the same activation map. A feature map or activation map is the output activation of a given filter. Feature map and activation have the same meaning. In some papers, it is called an activation map because it is a map of activations corresponding to different parts of the image, and it is also a feature map because it is also a map of some kind of feature found in the image. High activation means that some kind of feature was found.

[0140] Another important concept of CNN is pooling, which is a form of nonlinear downsampling. There are several nonlinear functions to implement pooling, of which max pooling is the most common. It divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such sub-region.

[0141] Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers are used to progressively reduce the spatial size of the representation in order to reduce the number of parameters, memory footprint, and computation in the network, and therefore also to control overfitting. In CNN architectures, it is common to insert pooling layers regularly between consecutive convolutional layers. The pooling operation provides another form of translation invariance.

[0142] The pooling layer operates independently on each depth strip of the input and resizes it spatially. The most common form is a pooling layer with a filter size of 2×2, which applies 2 samples in width and height on each depth strip in the input and discards 75% of the activations. In this case, each max operation is over 4 numbers. The depth dimension remains unchanged. In addition to max pooling, pooling units can also use other functions such as average pooling or Pooling. Average pooling was often used in the past, but has become less common recently compared to max pooling, which often works better. Due to the large reduction in the size of the representation, the recent trend is to use smaller filters or to drop the pooling layer entirely. "Region of interest" pooling (also known as ROI pooling) is a variant of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is an important component of convolutional neural networks, used for object detection based on the Fast R-CNN architecture.

[0143] The above ReLU is short for Rectified Linear Unit, which applies a non-saturated activation function. It effectively removes negative values ​​from the activation map by setting them to zero. It increases the nonlinear nature of the decision function and the overall network without affecting the receptive field of the convolutional layer. Other functions are also used to increase nonlinearity, such as the saturated hyperbolic tangent function and the sigmoid function or the LeakyReLU function. ReLU is often preferred over other functions because it trains neural networks several times faster without significantly affecting the generalization accuracy.

[0144] After several convolutional and max pooling layers, high-level reasoning in a neural network is done with fully connected layers. Neurons in a fully connected layer have connections to all activations in the previous layer, as in a regular (non-convolutional) artificial neural network. Therefore, these activations can be calculated as an affine transformation, with matrix multiplication followed by a bias shift (vector addition of a learned or fixed bias term).

[0145] The "loss layer" (including the calculation of the loss function) indicates how the training penalizes the deviation between the prediction (output) and the true label, and is usually the last layer of the neural network. Various loss functions suitable for different tasks can be used. Softmax loss is used to predict a single class from K mutually exclusive classes. Sigmoid cross entropy loss is used to predict K independent probability values ​​in [0, 1]. Euclidean loss is used to regress to real-valued labels.

[0146] In conclusion, Figure 1The data flow in a typical convolutional neural network is shown. First, the input image is passed through a convolutional layer and abstracted into a feature map, which includes several channels corresponding to the number of filters in a set of learnable filters of the layer. The feature map is then subsampled using, for example, a pooling layer, thereby reducing the dimension of each channel in the feature map. Next, the data reaches another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input channels and output channels are hyperparameters of the layer. In order to establish the connection of the network, these parameters need to be synchronized between the two connected layers, for example, the number of input channels of the current layer should be equal to the number of output channels of the previous layer. For the first layer that processes the input data (such as an image), the number of input channels is usually equal to the number of channels of the data representation, such as 3 channels for RGB or YUV representation of images or videos, or 1 channel for grayscale image or video representation.

[0147] Autoencoders and Unsupervised Learning

[0148] An autoencoder is a type of artificial neural network that is used to learn efficient data encoding in an unsupervised manner. Figure 2 As shown. The purpose of an autoencoder is to learn a representation (encoding) of a set of data by training the network to ignore the signal "noise", usually to reduce the dimensionality. Together with the dimensionality reduction side, a reconstruction side is learned, in which the autoencoder tries to generate a representation as close to its original input as possible from the simplified encoding, hence the name. In the simplest case, given one hidden layer, the encoder stage of the autoencoder takes the input x and maps it to h

[0149] h=σ(Wx+b).

[0150] This image h is often called a code, latent variable, or latent representation. Here, σ is an element-wise activation function, such as a sigmoid function or a rectified linear unit. W is the weight matrix and b is the bias vector. The weights and biases are usually initialized randomly and then iteratively updated via backpropagation during training. Afterwards, the decoder stage of the autoencoder maps h to a reconstruction x' of the same shape as x:

[0151] x'=σ'(W′h'+b')

[0152] Here, σ', W', and b' of the decoder may be independent of the corresponding σ, W, and b of the encoder.

[0153] Variational autoencoder models make strong assumptions about the distribution of latent variables. They use a variational approach to latent representation learning, which results in an additional loss component, and the algorithm is trained using a specific estimator called the Stochastic Gradient Variational Bayes (SGVB) estimator. The data is assumed to be a directed graphical model p θ (x|h) is generated, and the encoder is learning the posterior distribution p θ The approximation of (h|x) is qφ(h|x), where φ and θ represent the parameters of the encoder (recognition model) and decoder (generation model), respectively. The probability distribution of the latent vector of a VAE is usually closer to the probability distribution of the training data than that of a standard autoencoder. The objective of a VAE has the following form:

[0154]

[0155] Here, D KL represents the Kullback–Leibler divergence. The priors on the latent variables are usually set to centered isotropic multivariate Gaussian Typically, the shapes of the variational and likelihood distributions are chosen so that they are factorized Gaussian distributions:

[0156]

[0157] Among them, ρ(x) and ω 2 (x) is the encoder output, and μ(h) and σ 2 (h) is the decoder output.

[0158] Recent advances in the field of artificial neural networks, especially convolutional neural networks, have made researchers interested in applying neural network-based techniques to image and video compression tasks. For example, end-to-end optimized image compression was proposed, which uses a variational autoencoder-based network.

[0159] Data compression is therefore considered a fundamental and well-studied problem in engineering, and is typically addressed by designing a code for a given discrete data set with minimum entropy. The scheme relies heavily on knowledge of the probabilistic structure of the data, and the problem is therefore closely related to probabilistic source modeling. However, since all practical codes must have finite entropy, continuous-valued data (such as a vector of image pixel intensities) must be quantized to a finite set of discrete values, which introduces errors.

[0160] In this case, the lossy compression problem, two competing costs must be balanced: the entropy of the discrete representation (rate) and the error introduced by quantization (distortion). Different compression applications, such as data storage or transmission over channels of limited capacity, require different balances between rate and distortion.

[0161] It is difficult to jointly optimize rate and distortion. The general problem of optimal quantization in high-dimensional space is intractable without other constraints. Therefore, most existing image compression methods operate by linearly transforming the data vector into a suitable continuous-valued representation, independently quantizing its elements, and then encoding the resulting discrete representation using lossless entropy coding. Due to the central role of the transform, this scheme is called transform coding.

[0162] For example, JPEG uses discrete cosine transform on pixel blocks, and JPEG 2000 uses multi-scale orthogonal wavelet decomposition. Typically, the three components of the transform coding method (i.e., transform, quantization, and entropy coding) are optimized separately (usually through manual parameter tuning). Modern video compression standards (such as HEVC, VVC, and EVC) also use the transformed representation to predict the coded residual signal. Several transforms are used for this purpose, such as discrete cosine transform (DCT) and discrete sine transform (DST), as well as lowfrequency non-separable manually optimized transform (LFNST).

[0163] Variational Image Compression

[0164] The variational auto-encoder (VAE) framework can be considered as a nonlinear transformation decoding model. The transformation process can be divided into four main parts. Figure 3A Examples are given in the following sections.

[0165] The transformation process can be divided into four parts: Figure 3A The VAE framework is illustrated in Figure 3A In the example, encoder 101 maps an input image x to a latent representation (represented by y) by a function y=f(x). Hereinafter, this latent representation may also be referred to as a portion of a "latent space" or a point in a "latent space". Function f() is a transformation function that transforms the input signal x into a more compressible representation y. Quantizer 102 Transform the latent representation y into a quantized latent representation with (discrete) values Where Q represents the quantizer function. The entropy model or super encoder / decoder (also called super prior) 103 estimates the quantized latent representation distribution to obtain the minimum rate achievable by lossless entropy source decoding.

[0166] The latent space can be understood as a representation of compressed data, where similar data points are closer in the latent space. The latent space is very useful for learning data features and finding simpler data representations for analysis. The quantized latent representation of the hyperprior 3 is converted into and side information Included into the code stream 2 (binarization). In addition, a decoder 104 is provided, which transforms the quantized latent representation into a reconstructed image Signal is an estimate of the input image x. We want x to be as close as possible to In other words, the reconstruction quality is as high as possible. However, The higher the similarity between x and x, the higher the amount of side information required to be transmitted. Side information includes Figure 3A The bitstream 1 and bitstream 2 shown in the figure are generated by the encoder and transmitted to the decoder. Generally speaking, the higher the amount of side information, the higher the reconstruction quality. However, a large amount of side information means a low compression ratio. Therefore, Figure 3A One goal of the system described in is to balance the quality of reconstruction with the amount of side information transmitted in the codestream.

[0167] exist Figure 3A , component AE 105 is an arithmetic encoding (AE) module, which converts the quantized latent representation and side information The samples are converted into binary representation code stream 1. and Examples of may include integers or floating point numbers, for example. One purpose of the arithmetic coding module is to convert the sample values ​​(through a binarization process) into a binary digital string (which is then included in the codestream, which may include other parts corresponding to the encoded image or other side information).

[0168] Arithmetic decoding (AD) 106 is a process of recovering the binarization process, where the binarized numbers are converted back to sample values. Arithmetic decoding is provided by the arithmetic decoding module 106 .

[0169] It should be noted that the present invention is not limited to this particular framework. In addition, the present invention is not limited to image or video compression, and can also be applied to object detection, image generation and recognition systems.

[0170] exist Figure 3A In this context, a subnet is a logical division between parts of the overall network. For example, in Figure 3A In FIG. 1 , modules 101 , 102 , 104 , 105 and 106 are referred to as the “encoder / decoder” subnet. The “encoder / decoder” subnet is responsible for encoding (generating) and decoding (parsing) the first bitstream “bitstream 1”. Figure 3A The second network in the example includes modules 103, 108, 109, 110 and 107, and is called the "super encoder / decoder" subnet. The second subnet is responsible for generating the second bitstream "bitstream 2". The two subnets have different purposes.

[0171] The first subnet is responsible for:

[0172] Transform (101) the input image x to its latent representation y (which makes it easier to compress x),

[0173] Quantize (102) the latent representation y into a quantized latent representation

[0174] Arithmetic coding module 105 uses AE to compress the quantized latent representation To obtain the code stream "code stream 1".

[0175] · Use the arithmetic decoding module 106 to parse the code stream 1 through AD,

[0176] Reconstruct the image using the parsed data (104)

[0177] The purpose of the second subnet is to obtain the statistical characteristics of the "code stream 1" samples (such as the average value, variance and correlation between the samples of code stream 1) so that the first subnet can compress code stream 1 more efficiently. The second subnet generates a second code stream "code stream 2", which includes the information (such as the average value, variance and correlation between the samples of code stream 1).

[0178] The second network includes an encoding part, which includes converting the quantized latent representation Transform (103) into side information z, and quantize the side information z into quantized side information And the quantized side information The encoding (e.g., binarization) (109) is a code stream 2. In this example, the binarization is performed by arithmetic encoding (AE). The decoding part of the second network includes arithmetic decoding (AD) 110, which transforms the input code stream 2 into decoded quantized side information Can be used with The same as the arithmetic coding side decoding operation is a lossless compression method. Then, the decoded quantized side information is transformed (107) into decoded side information express The statistical properties of The average value of samples, or the variance of sample values, etc.). Then, the decoded latent representation is provided to the arithmetic encoder 105 and the arithmetic decoder 106 to control The probability model of .

[0179] Figure 3A An example of a variational auto encoder (VAE) is described, the details of which may vary in different implementations. For example, in a particular implementation, there may be additional components to more efficiently obtain the statistical characteristics of samples of bitstream 1. In one such implementation, there may be a context modeler whose goal is to extract cross-correlation information for bitstream 1. The statistical information provided by the second subnet can be used by the arithmetic encoder (AE) 105 and arithmetic decoder (AD) 106 components.

[0180] Figure 3A The encoder and decoder are depicted in a single figure. As is clear to those skilled in the art, the encoder and decoder can be and often are embedded in mutually different devices.

[0181] Figure 3B describes the encoder, Figure 3C The decoder component of the VAE framework is described separately. According to some embodiments, the encoder receives an image as input. The input image may include one or more channels, such as color channels or other types of channels, such as depth channels or motion information channels. The output of the encoder (e.g. Figure 3B ) are code stream 1 and code stream 2. Code stream 1 is the output of the first subnet of the encoder, and code stream 2 is the output of the second subnet of the encoder.

[0182] Similarly, in Figure 3C In this example, two bitstreams (bitstream 1 and bitstream 2) are received as input and generated at the output Reconstruct the (decoded) image. As mentioned above, the VAE can be divided into different logical units that perform different operations. Figure 3B and Figure 3C In the example, Figure 3B Describes the components involved in encoding a signal, such as a video, and provides the encoded information. The encoded information is then Figure 3C It should be noted that the components of the encoder and decoder denoted by the numbers 12x and 14x may correspond in their functions to those in the above Figure 3A Components mentioned in and denoted by the number 10x.

[0183] Specifically, Figure 3B As shown in FIG. 1 , the encoder includes an encoder 121, which transforms an input x into a signal y and then provides the signal y to a quantizer 322. The quantizer 122 provides information to an arithmetic coding module 125 and a super encoder 123. The super encoder 123 provides the code stream 2 discussed above to a super decoder 147, which in turn provides information to the arithmetic coding module 105 (125).

[0184] The output of the arithmetic coding module is code stream 1. Code stream 1 and code stream 2 are the outputs of signal encoding, which are then provided (transmitted) to the decoding process. Although unit 101 (121) is called an "encoder", it can also be called Figure 3B The complete subnetwork described in is called an "encoder". The encoding process generally refers to a unit (module) that converts an input into an encoded (e.g., compressed) output. Figure 3B As can be seen, unit 121 can actually be considered the core of the entire subnetwork, since it performs a transformation of the input x to y, which is a compressed version of x. Compression in encoder 121 can be achieved, for example, by applying a neural network or any processing network that generally has one or more layers. In such a network, compression can be performed by a cascade of processes including downsampling, which reduces the size and / or number of channels of the input. Therefore, the encoder can be referred to as a neural network (NN) based encoder, etc.

[0185] The remaining parts in the figure (quantization unit, super encoder, super decoder, arithmetic encoder / decoder) are all parts that improve the efficiency of the encoding process or are responsible for converting the compressed output y into a series of bits (bitstream). Quantization can be provided to further compress the output of the NN encoder 121 through lossy compression. The AE 125 in conjunction with the super encoder 123 and super decoder 127 used to configure the AE 125 can perform binarization, which can further compress the quantized signal through lossless compression. Therefore, it can also be Figure 3B The entire subnetwork in is called an "encoder".

[0186] Most deep learning (DL)-based image / video compression systems reduce the dimensionality of the signal before converting it into binary numbers (bits). For example, in the VAE framework, the encoder is a nonlinear transformation that maps the input image x to y, where the width and height of y are smaller than the width and height of x. Since y has a smaller width and height, it is smaller in size, which reduces the dimensionality (size) of the signal, so it is easier to compress the signal y. It should be noted that, in general, the encoder does not necessarily need to reduce the size in two (or usually all) dimensions. On the contrary, some exemplary implementations may provide an encoder that reduces the size in only one dimension (or usually a subset of dimensions).

[0187] In "Density Modeling of Images Using a Generalized Normalization Transformation", J. Balle, L. Valero Laparra, and EPSimoncelli (2015), In: arXiv e-prints, Presented at the 4th Int. Conf. for Learning Representations, 2016 (hereafter "Balle"), the authors propose an end-to-end optimization framework for image compression models based on nonlinear transformations. The authors optimize for mean squared error (MSE), but use a more flexible transformation built from a cascade of linear convolutions and nonlinearities. Specifically, the authors use a generalized divisive normalization (GDN) joint nonlinearity inspired by neuron models in biological vision systems and has been shown to be effective in Gaussianizing image density. This cascade of transformations is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), which effectively implements a parametric form of vector quantization on the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric inverse nonlinear transform.

[0188] Such an example of a VAE framework is Figure 4 As shown, it utilizes 6 downsampling layers, which are labeled 401 to 406. The network architecture includes a hyper-prior model. a , g s ) shows the image autoencoder architecture, the right side (h a ,h s ) corresponds to an autoencoder that implements a hyper-prior. The factored prior models the analysis and synthesis transformations g a and gs The same architecture is used. Q represents quantization, AE and AD represent arithmetic encoder and arithmetic decoder respectively. The encoder performs g on the input image x. a Processing, resulting in a response y (latent representation) with a spatially varying standard deviation. Encoding g a It includes multiple convolutional layers with subsampling and generalized divisive normalization (GDN) as activation function.

[0189] Feed the response to h a to summarize the distribution of standard deviations in z. z is then quantized, compressed, and sent as side information. The encoder then uses the quantized vector To estimate Used to obtain the spatial distribution of the standard deviation of probability values ​​(or frequency values) for arithmetic encoding (AE) and use it to compress and transmit the quantized image representation (or potential representation). The decoder first recovers the The decoder then uses h s Get This provides it with the correct probability estimate to successfully recover The decoder will then Feed to g s to obtain the reconstructed image.

[0190] Layers that include downsampling are indicated by a downward arrow in the layer description. The layer description "Conv N×5×5 / 2↓" indicates that the layer is a convolutional layer with N channels and a kernel size of 5×5. As mentioned above, 2↓ indicates that downsampling by a factor of 2 is performed in this layer. Downsampling by a factor of 2 reduces one dimension of the input signal by half at the output. Figure 4 In the example, 2↓ indicates that both the width and height of the input image are reduced by 1 / 2. Since there are 6 downsampling layers, if the width and height of the input image 414 (also represented by x) are represented by w and h, the output signal The width and height of are equal to w / 64 and h / 64 respectively. The modules denoted by AE and AD are the arithmetic encoder and arithmetic decoder, which will be referred to as FIG. 3A to FIG. 3C Explain. Arithmetic encoders and decoders are specific implementations of entropy decoding. AE and AD can be replaced by other entropy decoding methods. In information theory, entropy coding is a lossless data compression scheme used to convert the value of a symbol into a binary representation, which is a recoverable process. In addition, the "Q" in the figure corresponds to the above Figure 4A quantization operation is also mentioned and further explained in the "Quantization" section above. Furthermore, a quantization operation and a corresponding quantization unit as part of component 413 or 415 need not necessarily be present and / or may be replaced by another unit.

[0191] exist Figure 4 , a decoder including upsampling layers 407 to 412 is also shown. Another layer 420 is provided between the upsampling layers 411 and 410 in the order in which the inputs are processed, which layer is implemented as a convolutional layer, but does not provide upsampling of the received input. A corresponding convolutional layer 430 for the decoder is also shown. Such a layer may be provided in the NN for performing operations on the input that do not change the size of the input but change certain characteristics. However, it is not necessary to provide such a layer.

[0192] When viewed from the processing order of the code stream 2 through the decoder, the upsampling layers run in the reverse order, i.e., from upsampling layer 412 to upsampling layer 407. Each upsampling layer is shown here to provide an upsampling with an upsampling ratio of 2, indicated by ↑. Of course, not all upsampling layers necessarily have the same upsampling ratio, and other upsampling ratios such as 3, 4, 8, etc. may also be used. Layers 407 to 412 are implemented as convolutional layers (conv). Specifically, since they may be intended to provide an operation opposite to that of the encoder on the input, the upsampling layers may apply a deconvolution operation to the received input so that its size is increased by a factor corresponding to the upsampling ratio. However, the present invention is generally not limited to deconvolution, and upsampling may be performed in any other manner, such as by bilinear interpolation between two adjacent samples, or by nearest neighbor sample replication, etc.

[0193] In the first subnet, some convolutional layers (401 to 403) use generalized divisive normalization (GDN) at the encoding end and inverse GDN (IGDN) at the decoding end. In the second subnet, the activation function applied is ReLu. It should be noted that the present invention is not limited to this implementation method. Generally, other activation functions can be used instead of GDN or ReLu.

[0194] Cloud solutions for machine tasks

[0195] Video coding for machine (VCM) is another popular direction in computer science today. The main idea behind this approach is to send a coded representation of image or video information for further processing by computer vision (CV) algorithms, such as object segmentation, detection, and recognition. In contrast to traditional image and video decoding for human perception, the quality features are the performance of computer vision tasks, such as correspondence detection accuracy, rather than reconstruction quality. This is particularly true in Figure 5 Shown in.

[0196] Machine video coding, also known as collaborative intelligence, is a relatively new paradigm for the efficient deployment of deep neural networks in mobile cloud infrastructure. By dividing the network between mobile devices and the cloud, the computing workload can be distributed so that the overall energy and / or latency of the system is minimized. In general, collaborative intelligence is a paradigm in which the processing of a neural network is distributed between two or more different computing nodes; for example, devices, but generally any functionally defined nodes. Here, the term "node" does not refer to the neural network nodes mentioned above. Instead, the (computing) nodes here refer to (physically or at least logically) separate devices / modules that implement parts of the neural network. These devices can be different servers, different end-user devices, a combination of servers and / or user devices and / or clouds and / or processors, and so on. In other words, computing nodes can be considered to be nodes that belong to the same neural network and communicate with each other to transmit decoded data within / for the neural network. For example, in order to be able to perform complex calculations, one or more layers can be executed on a first device and one or more layers can be executed on another device. However, the distribution can also be more refined, and a single layer can be executed on multiple devices. In the present invention, the term "multiple" refers to two or more. In some prior art solutions, part of the neural network function is executed in a device (user device or edge device, etc.) or multiple such devices, and then the output (feature map) is passed to the cloud. The cloud is a collection of processing or computing systems located outside the device that is operating the part of the neural network. The concept of collaborative intelligence is also extended to model training. In this case, data flows in both directions: from the cloud to the mobile device during the back propagation of training, and from the mobile device to the cloud during the forward pass of training and inference.

[0197] Some works present semantic image compression by encoding deep features and then reconstructing the input image from these features. Compression based on uniform quantization is shown, followed by context-based adaptive arithmetic coding (CABAC) of H.264. In some scenarios, it may be more efficient to send the output of the hidden layer (deep feature map) from the mobile part to the cloud instead of sending compressed natural image data to the cloud and performing object detection using the reconstructed image. Efficient compression of feature maps is beneficial for image and video compression and reconstruction for human perception and machine vision. Entropy coding methods (such as arithmetic coding) are popular methods for compression of deep features (i.e., feature maps).

[0198] Nowadays, video content contributes more than 80% of Internet traffic, and this proportion is expected to rise further. Therefore, it is crucial to build an efficient video compression system and generate higher quality frames under a given bandwidth budget. In addition, most video-related computer vision tasks, such as video object detection or video object tracking, are sensitive to the quality of compressed videos, and efficient video compression may benefit other computer vision tasks. At the same time, video compression technology also helps action recognition and model compression. However, in the past few decades, video compression algorithms have relied on hand-crafted modules, such as block-based motion estimation and discrete cosine transform (DCT) to reduce redundancy in video sequences, as mentioned above. Although each module has been carefully designed, the entire compression system has not been optimized end-to-end. It is hoped that the video compression performance can be further improved by jointly optimizing the entire compression system.

[0199] End-to-end image or video compression

[0200] DNN-based image compression methods can take advantage of large-scale end-to-end training and highly nonlinear transformations, which are not used in traditional methods. However, it is non-trivial to directly apply these techniques to build an end-to-end learning system for video compression. First, learning how to generate and compress motion information tailored for video compression remains an open problem. Video compression methods heavily rely on motion information to reduce temporal redundancy in video sequences.

[0201] A simple solution is to use learning-based optical flow to represent motion information. However, current learning-based optical flow methods aim to generate flow fields as accurately as possible. Accurate optical flow is often not the best choice for a specific video task. In addition, the amount of optical flow data increases significantly compared to motion information in traditional compression systems, and directly applying existing compression methods to compress optical flow values ​​will significantly increase the number of bits required to store motion information. Second, it is not clear how to build a DNN-based video compression system by minimizing the rate-distortion-based objective of residual and motion information. The goal of rate-distortion optimization (RDO) is to achieve higher quality (i.e., less distortion) of reconstructed frames given the number of compressed bits (or bitrate). RDO is very important for video compression performance. In order to leverage the power of end-to-end training of learning-based compression systems, RDO strategies are needed to optimize the entire system.

[0202] In Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao, “DVC: ​​An End-to-end Deep Video Compression Framework”, Proceedings of the IEEE / CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11006–11015, the authors propose an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual coding.

[0203] This encoder Fig. 6A shown in . Specifically, Fig. 6A The overall structure of the end-to-end trainable video compression framework is shown. In order to compress motion information, a CNN is specified to transform the optical flow into a corresponding representation suitable for better compression. Specifically, an autoencoder style network is used to compress the optical flow. The motion vector (MV) compression network is shown in Figure 2. Figure 6B As shown in Figure 2. Figure 4 g in a / g sSomewhat similar. Specifically, the optical flow is fed into a series of convolution operations and nonlinear transformations including GDN and IGDN. The number of output channels of the convolution (deconvolution) is 128, except for the last deconvolution layer, whose number of output channels is equal to 2. Given an optical flow of size M×N×2, the MV encoder will generate a motion representation of size M / 16×N / 16×128. The motion representation is then quantized, entropy encoded and sent to the bitstream. The MV decoder receives the quantized representation and reconstructs the motion information using the MV encoder.

[0204] Figure 6C Here, the previously reconstructed frame x is used. t–1 The warping unit generates the warped frame (usually with the help of an interpolation filter, such as a bilinear interpolation filter) by reconstructing the motion information. Then, a separate CNN with three inputs generates the predicted image. The architecture of the motion compensated CNN is also as Figure 6C shown.

[0205] The residual information between the original frame and the predicted frame is encoded by the residual encoder network. A highly nonlinear neural network is used to transform the residual into the corresponding latent representation. Compared with the discrete cosine transform in traditional video compression systems, this method can better utilize the power of nonlinear transformation and achieve higher compression efficiency.

[0206] From the above overview, it can be seen that CNN-based architectures can be applied to image and video compression, taking into account different parts of the video framework, including motion estimation, motion compensation, and residual coding. Entropy decoding is a popular method for data compression, widely adopted by the industry, and is also suitable for feature map compression for human perception or computer vision tasks.

[0207] Machine video encoding

[0208] Video coding for machine (VCM) is another popular direction in computer science today. The main idea behind this approach is to send a coded representation of image or video information for further processing by computer vision (CV) algorithms, such as object segmentation, detection, and recognition. In contrast to traditional image and video decoding aimed at human perception, the quality features are the performance of computer vision tasks, such as correspondence detection accuracy, rather than reconstruction quality.

[0209] A recent study proposed a new deployment paradigm, called collaborative intelligence, in which deep models are partitioned between mobile and cloud. Extensive experiments on various hardware configurations and wireless connectivity modes show that the optimal operating point in terms of energy consumption and / or collaborative intelligence involves partitioning the model, typically at a point deep in the network. It was found that today's common schemes (i.e., models are either entirely in the cloud or entirely in mobile) are rarely, if ever, optimal. The concept of collaborative intelligence is also extended to model training. In this case, data flows bidirectionally: from the cloud to the mobile device during backpropagation of training, and from the mobile device to the cloud during the forward pass of training and during inference.

[0210] In the context of recent deep models for object detection, lossy compression of deep feature data has been studied based on HEVC intra-coding. It should be noted that the detection performance degrades with increasing compression levels, and compression-augmented training has been proposed to minimize this loss by generating models that are more robust to quantization noise in the feature values. However, this is still a suboptimal solution because the codecs used are highly complex and optimized for natural scene compression rather than deep feature compression.

[0211] The method of using the popular YOLOv2 network for object detection tasks addresses the problem of deep feature compression for collaborative intelligence to study the balance between compression efficiency and recognition accuracy. Here, the term deep feature has the same meaning as feature map. The word "deep" comes from the collaborative intelligence idea, that is, capturing the output feature map of some hidden (deep) layers and transmitting it to the cloud to perform inference. This seems to be more efficient than sending compressed natural image data to the cloud and using the reconstructed image to perform object detection.

[0212] Efficient compression of feature maps is beneficial for image and video compression and reconstruction for both human perception and machine vision. The shortcomings of state-of-the-art autoencoder-based compression methods are also applicable to machine vision tasks.

[0213] …

[0214] Improve decoding efficiency

[0215] As mentioned above, variational autoencoders have become the state-of-the-art approach for learnable image compression. Appearing in 2017, they only supported a single quality mode (a balance between rate and distortion), until variable rate methods emerged, which supported the selection of compression quality within a predefined specific range. The core of variable rate models is the gain unit, which is a part of the neural network-based codec that provides the model with the ability to compress input data of different compression ranges. However, traditional gain units are designed to work within some predefined range of compression quality. Traditional gain units cannot match some bitstream sizes outside the predefined compression quality range.

[0216] Therefore, some embodiments of the present invention introduce a gain vector extrapolation scheme for an autoencoder to enable flexible content representation and transmission without being restricted to a predefined or pre-trained range of compression quality or bitstream size.

[0217] Some detailed embodiments and examples related to the encoding end and the decoding end are provided below.

[0218] Encoding method

[0219] Figure 7 It is a flowchart of an exemplary method for encoding, the method comprising: step 701, receiving an image; step 702, receiving a first decoding parameter for the image, wherein the value of the first decoding parameter is less than a preset minimum value or greater than a preset maximum value; step 703, obtaining a target gain vector based on the first decoding parameter; step 704, encoding the image based on the target gain vector.

[0220] In step 701, an image is received. In an embodiment of the present invention, the image is an image to be compressed, which may be an image captured by a camera of an encoding device (the encoding device executes method 700 to encode an image), or the image may be an image obtained from inside the encoding device (for example, an image stored in an album of the encoding device, or an image obtained by the encoding device from a cloud or other device). The image may be a single image or a frame of a video. In addition, the image may be a part of an image, i.e., an image block. It should be understood that the above-mentioned image may be an image or an image block with image compression requirements, and the present invention does not impose any restrictions on the source of the image to be processed. The image includes brightness (Y) samples and / or chrominance (UV) samples.

[0221] In step 702, a first decoding parameter is received, wherein the value of the first decoding parameter is less than a preset minimum value or greater than a preset maximum value. The first decoding parameter (e.g., denoted as β) is a scalar input to the encoding process and is used to select the compression quality. The larger β is, the larger the code stream size is, and the higher the quality of the reconstructed data is. The preset minimum value (e.g., denoted as β) is s ) and a preset maximum value (for example, denoted as β t ) and the pre-trained value range of the first decoding parameter [β s , β t ] related. Preset maximum value β t Greater than or equal to the preset minimum value β s In some implementations, the pre-trained value range of the first decoding parameter β may be discontinuous, and the value of the first parameter β is a finite discrete point. During the inference process, by applying interpolation, the value of the first parameter β can be expanded to β s With β t Any value in between.

[0222] In a possible design, the image includes a luminance sample and a chrominance sample, and the first decoding parameter may include a first component and a second component. At least one of the first component and the second component of the first decoding parameter satisfies the condition that the value of the first decoding parameter is less than a preset minimum value or greater than a preset maximum value. It should be noted that the preset minimum value and the preset maximum value of the luminance sample and the chrominance sample may be different. The first component may be used to encode the luminance sample, thereby controlling the compression quality of the luminance sample of the image. The second component may be used to encode the chrominance sample, thereby controlling the compression quality of the chrominance sample of the image.

[0223] In a possible design, the image includes a luminance sample and a chrominance sample, and a second decoding parameter is obtained. One of the first decoding parameter and the second decoding parameter is used to encode the luminance sample, and the other is used to encode the chrominance sample. The chrominance sample and the luminance sample are encoded using different decoding parameters β to control the compression quality respectively. It should be noted that the preset minimum value and the preset maximum value of the luminance sample and the chrominance sample can be different or the same, and the present invention is not limited thereto.

[0224] In the embodiment of the present invention, the preset minimum value β s and the preset maximum value β t It can be stored on the encoding device or received from other devices, which is not limited here.

[0225] In an embodiment of the present invention, each value of the first decoding parameter β corresponds to a gain vector, i.e., a model weight of a neural network model for encoding an image. A set of possible values ​​of β pre-trained and corresponding gain vectors are obtained in the model training phase. As described above, the possible values ​​of β are within a preset minimum value β. s With the preset maximum value β t In the present invention, the gain vector is a vector of dimension 1×d, where d is an integer greater than 1. In one possible implementation, when encoding the luminance samples of an image, d=128. In one possible implementation, when encoding the chrominance samples of an image, d=64. It should be noted that d may also be other integers, such as 32, 48, 96, 144, 160, 176, 192, 256, etc.

[0226] Therefore, in one possible embodiment, the pre-trained possible value set of β and the corresponding gain vector are stored on the encoding device and the decoding device. In another possible embodiment, the encoding device and the decoding device only store the pre-trained possible value set of β or the corresponding gain vector; the encoding device and the decoding device also store the mapping relationship between the possible values ​​of β and their corresponding gain vectors. The mapping relationship represents a one-to-one correspondence between the possible values ​​of β and the gain vector. The mapping relationship can be a preset table or a preset objective function or any other form, as long as they are the same on the encoding device and the decoding device, which is not limited here.

[0227] In one possible design, luma samples and chroma samples are trained together. Therefore, luma samples and chroma samples share a list of possible values ​​of the first decoding parameter. However, the gain vectors obtained after the training phase are different for luma samples and chroma samples. Table 1 is an exemplary table of pre-trained β and corresponding gain vectors for luma samples and chroma samples. The number of pre-trained decoding parameters is N, where N is an integer greater than 0. In one possible design, N=13. In Table 1, β0 can be the minimum value β s , β N-1 It can also be the maximum value β t .

[0228] Table 1 - Luma samples and chroma samples share the first decoding parameter β

[0229] β Brightness gain vector Chroma gain vector <![CDATA[β0]]> <![CDATA[m Y0 ]]> <![CDATA[m UV0 ]]> <![CDATA[β1]]> <![CDATA[m Y1 ]]> <![CDATA[m UV1 ]]> …… …… …… <![CDATA[β N-1 ]]> <![CDATA[m Y(N-1) ]]> <![CDATA[m UV(N-1) ]]>

[0230] In one possible design, luma samples and chroma samples are trained separately. Therefore, luma samples and chroma samples may not share a list of possible values ​​of the first decoding parameter. Therefore, two tables are used to store pre-trained [β, gain vector] pairs for luma samples and chroma samples.

[0231] Table 2 is an exemplary table of pre-trained β and corresponding gain vectors for luma samples. Table 3 is an exemplary table of pre-trained β and corresponding gain vectors for chroma samples. The number of pre-trained decoding parameters for chroma samples is N1, where N1 is an integer greater than 0. In one possible design, N1=13. The number of pre-trained decoding parameters for chroma samples is N2, where N2 is an integer greater than 0. In one possible design, N2=13. It should be noted that N1 and N2 can be other integer values, such as N1=10, N2=5, etc. It is possible that N1 is greater than, equal to, or less than N2, and the present invention is not limited thereto. In Table 2, β0 can be the minimum value β of the luma sample. s , β N1-1 It can also be the maximum value of the brightness sample β t In Table 3, β0 can be the minimum value of chrominance samples β s , βN2-1 It can also be the maximum value of chroma samples β t .

[0232] Table 2 - First decoding parameter β of luma samples and corresponding gain vector

[0233] β Brightness gain vector <![CDATA[β0]]> <![CDATA[m Y0 ]]> <![CDATA[β1]]> <![CDATA[m Y1 ]]> …… …… <![CDATA[β N1-1 ]]> <![CDATA[m Y(N1-1) ]]>

[0234] Table 3 - First decoding parameter β of chroma samples and corresponding gain vector

[0235] β Brightness gain vector <![CDATA[β0]]> <![CDATA[m UV0 ]]> <![CDATA[β1]]> <![CDATA[m UV1 <!-- 21 -->]]> …… …… <![CDATA[β N2-1 ]]> <![CDATA[m UV(N2-1) ]]>

[0236] In step 703, a target gain vector is obtained based on the first decoding parameter. As described above, in the pre-trained possible value set of β and the corresponding gain vector, the possible value of β is within the preset minimum value β s With the preset maximum value β t The disadvantage of this method is that the image compression quality of the pre-trained model is limited by some minimum and maximum values. Even if twice as many bits are spent, it is impossible to obtain better quality than the maximum quality using the existing methods. Even in some cases, lower quality than the minimum quality is acceptable, the existing methods do not support compressing images with fewer bits. Therefore, a coding method that supports encoding images with any image compression quality is proposed to overcome the shortcomings of the existing methods. When the target compression quality is provided by the first decoding parameter, even if the value of the first decoding parameter is less than the preset minimum value or greater than the preset maximum value, the corresponding target gain vector can be obtained using the method proposed by the present invention.

[0237] Figure 8 is a schematic diagram of the relationship between the first decoding parameter (ie, network input parameter) β and the gain vector. Figure 3A The encoder 101 in [β s , β t ] are input into the network and trained to obtain the corresponding gain vector. Figure 8 In the figure, three pre-trained [β, gain vector] pairs are shown, namely [β s , m s ]、[β r , m r ] and [β t , m t ], where β s is the lower boundary of the range of β values, β tis the upper boundary of the value range of β. It should be noted that the present invention is not limited to this implementation mode. Generally, other possible numbers of [β, gain vector] pairs can be pre-trained, and usually more than 3 [β, gain vector] pairs are pre-trained. For example, 13 [β, gain vector] pairs are pre-trained in a possible design.

[0238] In the present invention, if Figure 8 As shown in FIG. 1 , an extrapolation method for obtaining a target gain vector based on a first decoding parameter is proposed. In the inference stage, given a specific value of the first decoding parameter (denoted as β v ), the target gain vector (expressed as m) can be derived according to the objective function v ), that is, m v =f(β v ,β r ,β t ,β s ,m r ,m s ,m t ,…).

[0239] In one possible embodiment, only the most recent pre-trained input parameters and their corresponding gain vectors are used to derive β v The new gain vector m v For example, when β v Greater than the preset maximum value β t When , only the maximum value β is used t and its corresponding gain vector m t Derivation m v In this case, m v =f(β v ,β t ,m t ). This function m v =f(β v ,β t ,m t ) is:

[0240]

[0241] Among them, K is a hyperparameter used to control m v The relationship between its nearest pre-trained gain vector. The value of K can be preset or depend on β v .

[0242] Similarly, when β v Less than the preset minimum value β s When , only the minimum value β is used s and its corresponding gain vector m s To derive m vIn this case, m v =f(β v ,β s ,m s ). Similarly, the function m v =f(β v ,β s ,m s ) is:

[0243]

[0244] Similar to equation (1), where K is a hyperparameter used to control m v The relationship between its nearest pre-trained gain vector. The value of K can be preset or depend on β v .

[0245] In equations (1) and (2), the value of K can be preset and stored on the encoding device and the decoding device, and K does not need to be transmitted, so the size of the code stream can be smaller. The default value of K can be 1. In another possible implementation, the value of K can depend on β v Different values ​​of K may be used in equation (1) and equation (2), for example, in equation (1), K=1.5, and in equation (2), K=0.5.

[0246] The above embodiment proposes to derive the target gain vector outside the predefined range, so that any desired image compression quality can be achieved. In the case where lower distortion is required, higher compression quality can be achieved by extrapolating the predefined maximum gain vector. In the case where a lower bitstream size is required, lower compression quality can be achieved by extrapolating the predefined minimum gain vector.

[0247] In one possible embodiment, the two closest pre-trained values ​​of β and their corresponding gain vectors are used to obtain β v The target gain vector m v For example, when β v Greater than the preset maximum value β t When t and its corresponding gain vector m t , and β to β v The next closest pre-trained value of Figure 8 Indicated as β r ) and its corresponding gain vector m r To derive β v m v In this case, m v =f(β v ,β r ,β t,m r ,m t ). This function m v =f(β v ,β r ,β t ,m r ,m t ) is:

[0248]

[0249] Among them, K is a hyperparameter used to control m v The relationship between β and its next closest pre-trained gain vector. The value of K can be preset and stored on the encoding device and the decoding device. The value of K can be preset and stored on the encoding device and the decoding device. The default value of K can be 1. In another possible implementation, the value of K can depend on β v ,For example,

[0250] Similarly, when β v Less than the preset minimum value β s When s and its corresponding gain vector m s , and β to β v The next closest pre-trained value of Figure 8 Indicated as β r ) and its corresponding gain vector m r To derive β v m v In this case, m v =f(β v ,β r ,β s ,m r ,m s ). This function m v =f(β v ,β r ,β s ,m r ,m s ) is:

[0251]

[0252] Similar to equation (3), where K is a hyperparameter used to control m v The relationship between β and its next closest pre-trained gain vector. The value of K can be preset and stored on the encoding device and the decoding device. The default value of K can be 1. In another possible implementation, the value of K can depend on β v ,For example,

[0253] It should be noted that in Figure 8 In the example, only three pre-trained [β, gain vector] pairs are shown, so β ​​to β v The next closest pre-trained value of is denoted as β in both Equation (3) and Equation (4). r . Figure 8 is a schematic diagram. In other possible embodiments, more than 3 [β, gain vector] pairs are pre-trained. v Less than β s And β v Greater than β t When β to β v The next closest pretrained value of should be different.

[0254] In equation (3) and equation (4), when K is set to the default value, K does not need to be transmitted, so the size of the code stream can be smaller. The default value of K can be 1. In another possible implementation, the value of K can depend on β v Different values ​​of K can be used in equations (3) and (4), for example, for large β v , K = 1, for small β v ,

[0255] Using more pre-trained [β, gain vector] pairs leads to better performance.

[0256] In one possible embodiment, more than one [β, gain vector] pair is used to derive β v The new gain vector m v In one possible implementation of this embodiment, N (N>2) closest pre-trained values ​​of β to v and their corresponding gain vectors are used to derive β by polynomial extrapolation. v The target gain vector m v For example, when β v Greater than the preset maximum value β t When , the target gain vector satisfies the following conditions:

[0257]

[0258] In one possible design, the specific form of equation (5) is:

[0259]

[0260] Similarly, when β v Less than the preset minimum value β s When , the target gain vector satisfies the following conditions:

[0261]

[0262] In one possible design, the specific form of equation (6) is:

[0263]

[0264] In one possible embodiment, all pre-trained [β, gain vector] pairs are used to derive

[0265] β v The new gain vector m v A weight is provided for each pre-trained value of β to control the influence of different pre-trained values ​​of β and their corresponding gain vectors.

[0266] It should be noted that in the above possible forms of the function, an offset or shift may also be added. The present invention does not limit the form of the function. All operations related to the gain vector are element-by-element operations, which means that the operation is performed on a single element of the gain vector.

[0267] In one possible design, when the image includes only chroma samples or luma samples, a target gain vector is obtained in step 703; when the image includes both chroma samples and luma samples, a target gain vector of luma samples is obtained in step 703, and a target gain vector of chroma samples is also obtained in step 703. Step 703 may be performed twice. Step 704 encodes the image based on the target gain vector. After the target gain vector is obtained in step 703, the target gain vector is used to encode the image. Fig. 9 is a flow chart of an exemplary method for encoding an image based on a target gain vector, Fig.10 Is executable Fig. 9 An exemplary VAE framework of method 900 in FIG. As is clear to those skilled in the art, this embodiment can be combined with any of the above embodiments and any possible designs or possible implementations thereof.

[0268] Fig. 9 The encoding method 900 shown includes: step 901, obtaining a first feature map of an image through a neural network; step 902, obtaining a second feature map based on the first feature map and a target gain vector; step 903, quantizing the second feature map to obtain a quantized second feature map; step 904, encoding the quantized second feature map to obtain a code stream.

[0269] In step 901, a neural network is used to obtain a potential representation of the image (i.e., a first feature map). In the VAE framework, the function of the neural network can be expressed as Fig.10The encoder 1001 shown is implemented. The encoder 1001 maps the input image x (i.e., image) to a potential representation (represented by y) through the function y=f(x). The function f() is a transformation function that converts the input signal x into a more compressible representation y. Typically, y is a tensor of shape w×h×d (e.g., w×h×128 or w×h×64), where w represents the feature map width, h represents the feature map height, and d represents the number of feature map channels. It should be noted that the feature map width and the feature map height may be equal or unequal, and are not limited here. The feature map channels may also be other integer values, such as 32, 48, 96, 144, 160, 176, 192, 256, etc., which are not limited in the present invention. In one possible design, the number of feature map channels for luminance samples and chrominance samples may be different, such as 128 luminance samples and 64 chrominance samples. Although unit 1001 is referred to as an "encoder", it may also be Fig.10 The complete encoding network described in is called an “encoder”. The encoding process generally refers to a unit (module) that converts an input into an encoded (eg, compressed) output.

[0270] In step 902, after obtaining the first feature map of the image in step 901, the first feature map can be further calculated based on the target gain vector. The target gain vector can be the target gain vector obtained in step 703 of method 700. In one possible implementation, the function of step 902 can be performed by a gain unit 1011 after the encoder 1001. The input of the gain unit 1011 (represented by y) is the output of the encoder 1001. The gain unit 1011 also transforms the first feature map with the target gain vector. As described in step 702, the gain vector is a vector with a dimension of 1×d, where d is an integer greater than 1. In the present invention, d is equal to the number of feature map channels of y. Therefore, each element of the target gain vector can be mapped to a feature map channel of y. In one possible implementation, the gain unit 1011 multiplies the target gain vector with the first feature map to obtain a second feature map. The output of the gain unit 1011 is expressed as

[0271] It should be noted that step 703 needs to be performed before step 902, and the actions described in other steps of methods 700 and 900 can be performed in different orders and still obtain the desired results. For example, the process described in the accompanying drawings does not necessarily require the specific order shown or the sequential execution to achieve the desired results. In some implementations, it may be advantageous to perform multitasking and parallel processing. For example, step 702 may be performed before step 901, or step 702 may be performed after step 901, or step 702 and step 901 may be performed in parallel.

[0272] In step 903, the second feature map obtained in step 902 is quantized. The quantization process can be performed by Fig.10 Quantizer 1002 is shown as performing the quantization. Quantizer 1002 performs the quantization by Potentially represent Transformed into a quantized latent representation with (discrete) values Here, Q represents the quantizer function.

[0273] In step 904, the quantized second feature map obtained in step 903 is further encoded into the bitstream 1 using entropy coding. The entropy coding process can be performed by Fig.10 The arithmetic encoder (AE) 1005 shown in FIG. 1 is executed by quantizing the latent representation The samples of the quantized second feature map are converted into a binary digital string (which is then included in the bitstream, which may include other parts corresponding to the encoded image or other side information). It should be noted that the arithmetic encoder 1005 is a specific implementation of entropy decoding. AE can be replaced by other entropy decoding methods. In information theory, entropy coding is a lossless data compression scheme for converting the value of a symbol into a binary representation, which is a recoverable process.

[0274] The output of step 904 is a code stream, which is then provided (transmitted) to the decoding process. As is clear to those skilled in the art, this embodiment can be combined with any of the above embodiments and any possible designs or possible implementations thereof.

[0275] In one possible design, when the image includes only chroma samples or luma samples, a target gain vector is obtained in step 703, and method 900 is performed to encode the image based on the target gain vector; when the image includes both chroma samples and luma samples, a target gain vector of the luma sample is obtained in step 703, a target gain vector of the chroma sample is also obtained in step 703, step 703 may be performed twice, and the luma sample and the chroma sample are independently encoded based on their corresponding target gain vectors using method 900. The encoded luma samples and chroma samples may be packaged into a bitstream for storage or transmission.

[0276] In one possible embodiment, the encoding method 700 further includes encoding the first decoding parameter into the bitstream. In some cases, different images may need to be compressed with different qualities, and therefore, the encoding method 700 can be used to encode different images, with the first decoding parameter set to different values. The proposed method 700 can achieve different compression qualities using a single pre-trained model, in particular, compression qualities outside the pre-trained range.

[0277] The first decoding parameter is indicated in the code stream (for example, as a first flag, such as compression_quality_level), and is transmitted to the decoding device so that the decoding device can decode the decoded image data based on the first decoding parameter. In one possible implementation, the first decoding parameter is encoded or indicated in a picture parameter set (PPS) of the code stream. In one possible implementation, the first decoding parameter is encoded or indicated in a sequence parameter set (SPS) of the code stream. In one possible implementation, the first decoding parameter is encoded or indicated in a picture header (PH) or a slice header (SH) of the code stream.

[0278] In a possible design, the number of bits used to indicate the first decoding parameter (first flag) in the code stream is less than or equal to 16. For example, the first parameter may occupy 15 bits in the code stream.

[0279] In some possible designs, the first decoding parameter is indicated in other forms to save bits for transmitting the first decoding parameter. For example, in one possible design, the first decoding parameter is indicated in its logarithmic form with base 2. In another possible design, the first decoding parameter is indicated in its logarithmic form with base 2 and has a bias. The base of the logarithm can be other values, such as 10, which is not limited here. In this way, the bits used to transmit the first decoding parameter can be reduced to less than or equal to 4 bits.

[0280] Optionally, the side information z output by the super encoder 1003 may be processed by a gain unit 1012 before quantization. The gain unit 1012 is functionally similar to the gain unit 1011, but the gain vectors used in the gain unit 1012 and the gain unit 1011 may be different. This embodiment can encode the side information in a more flexible manner.

[0281] In a possible embodiment, the image includes only luma samples, and the method 700 is used to encode the luma samples based on a first decoding parameter to obtain a bitstream, and the first decoding parameter is indicated as a first flag (eg compression_quality_level) in the bitstream as described above.

[0282] In a possible embodiment, the image includes only chrominance samples, and the method 700 is used to encode the chrominance samples based on the first decoding parameter to obtain a code stream, and the first decoding parameter is indicated as a first flag (eg compression_quality_level) in the code stream as described above.

[0283] In a possible embodiment, the image includes luma samples and chroma samples. When the luma samples are encoded based on the first decoding parameters using the method 700, the chroma samples are encoded based on the second decoding parameters using the method 700; or, when the chroma samples are encoded based on the first decoding parameters using the method 700, the luma samples are encoded based on the second decoding parameters using the method 700. It should be noted that when the luma samples and the chroma samples are used to encode the image, some steps of the method 700 may be performed only once, such as step 701. The first decoding parameter and the second decoding parameter are both indicated in the bitstream as described above, for example, the first decoding parameter is indicated as a first flag (e.g., compression_quality_level_luma), and the second decoding parameter is indicated as a second flag (e.g., compression_quality_level_chroma). Alternatively, the first decoding parameter is indicated as a first flag (e.g., compression_quality_level_chroma), and the second decoding parameter is indicated as a second flag (e.g., compression_quality_level_luma). The indication method of the second decoding parameter is the same as the indication method of the first decoding parameter. Therefore, the number of bits used to transmit the decoding parameter is doubled. At least one of the value of the first decoding parameter and the value of the second decoding parameter is smaller than a preset minimum value or larger than a preset maximum value.

[0284] In one possible embodiment, the image includes luma samples and chroma samples, and the first decoding parameter includes a first component and a second component. The luma samples are encoded based on the first component using method 700, and the chroma samples are encoded based on the second component using method 700. It should be noted that when the image is encoded using luma samples and chroma samples, some steps of method 700 may be performed only once, such as step 701. The first component and the second component are both indicated in the bitstream, as described above, for example, the first component is indicated as a first flag (e.g., compression_quality_level_luma), and the second component is indicated as a second flag (e.g., compression_quality_level_chroma). The indication method of the first component and the second component is the same as the indication method of the first decoding parameter as described above. Therefore, the number of bits used to send the first decoding parameter is doubled.

[0285] In a possible embodiment, the decoding parameters β can be used to v Different regions of the entire image are encoded, and several decoding parameters β v Indicated in the bitstream for transmission to other devices. For example, the region of interest can be indicated with a larger β v Encoding to obtain better compression quality, the background area can be encoded with a smaller β vEncoding to obtain a smaller code stream size. This method can enable different areas of a single image to obtain different compression qualities, making the encoding process more flexible and meeting different compression requirements in different situations.

[0286] according to Figure 7 The embodiments can be used to provide reference to Fig.11 The decoding method described here easily decodes the output. Figure 7 The method 700 will be described as being performed by a neural network system of one or more computers located in one or more locations. For example, a system for performing image compression, such as Figure 1 The neural network may perform method 700. Generally, the above embodiments may be combined to provide more flexibility.

[0287] Decoding method

[0288] Fig.11 It is a flowchart of an exemplary method for image decoding based on a neural network architecture, comprising: step 1101, obtaining a code stream including decoded image data; step 1102, parsing the code stream to obtain a first decoding parameter, wherein the value of the first decoding parameter is less than a preset minimum value or greater than a preset maximum value; step 1103, obtaining a target inverse gain vector based on the first decoding parameter; step 1104, obtaining an image based on the target inverse gain vector.

[0289] In step 1101, a code stream including decoded image data may be obtained from an encoding device or a distribution device. The code stream may include some side information (e.g., the mean value or variance of the encoded samples, etc.), some decoding parameter information (e.g., decoding mode, compression quality parameter, quantization parameter, etc.).

[0290] In step 1102, the code stream is parsed to obtain a first decoding parameter, wherein the value of the first decoding parameter is less than a preset minimum value or greater than a preset maximum value. The first decoding parameter (eg, denoted as β) is a scalar input of the decoding process for selecting compression quality.

[0291] The first flag (e.g., compression_quality_level) is indicated in the bitstream to specify the compression quality of the image (i.e., the first decoding parameter β). In one possible design, the first flag is the compression quality, which may require 16 bits or 15 bits to store and send. In this design, the first decoding parameter β is parsed directly from the bitstream. In another possible design, the first flag is the base-2 logarithmic form of the first decoding parameter (e.g., log2_compression_quality_level or log2_compression_quality_level_minus1), and the first decoding parameter may be derived as follows: the first decoding parameter β = 1 << first flag; or the first decoding parameter β = 1 << (first flag + offset), where "<<" indicates left shift, the first flag may be an integer from 0 to 16, and the offset may be 1, 2, 3, 4, 5, 8, etc.

[0292] In one possible embodiment, the first flag and the second flag are indicated in the bitstream to specify the compression quality of the image. When the first flag (eg, compression_quality_level_luma) is combined with the first decoding parameter (denoted as β) specifying the compression quality of the luminance samples of the image, Y ), the second flag (eg, compression_quality_level_chroma) is associated with a second decoding parameter (denoted as β) specifying the compression quality of the chrominance samples of the image UV ). When the first flag (e.g., compression_quality_level_chroma) is associated with the first decoding parameter (denoted as β) that specifies the compression quality of the chrominance samples of the image UV ), the second flag (eg, compression_quality_level_luma) is associated with a second decoding parameter (denoted as β) specifying the compression quality of the luminance samples of the image. Y ). The first decoding parameter and the second decoding parameter can both be derived using the method described in the previous paragraph. At least one of the value of the first decoding parameter and the value of the second decoding parameter is less than a preset minimum value or greater than a preset maximum value.

[0293] In one possible embodiment, a first flag is indicated in the bitstream to specify the compression quality of the image. The first flag includes a first component (e.g., compression_quality_level_luma) and a second component (e.g., compression_quality_level_chroma). The first component is associated with a first decoding parameter (denoted as β) of the compression quality of the luminance samples of the specified image. Y), the second component is related to a second decoding parameter (denoted as β) specifying the compression quality of the chrominance samples of the image UV The first decoded component and the second decoded component can both be derived using the method described in the above paragraphs.

[0294] In step 1103, after obtaining the first decoding parameter, a target inverse gain vector is obtained based on the first decoding parameter. During the network training phase, a pre-trained set of possible values ​​of β and corresponding gain vectors are obtained and stored in the encoding device and the decoding device, and the possible values ​​of β are within a preset minimum value β. s With the preset maximum value β t Therefore, the reference Figure 8 The method described above obtains the target gain vector m v Then, the target inverse gain vector m can be obtained based on the target gain vector v ', target inverse gain vector m v 'The following conditions are met:

[0295] m′ v *m v =C,

[0296] Where C is a vector whose elements are all constants, and * represents an element-by-element multiplication operation. In one possible design, C is a constant, and * represents a point multiplication operation.

[0297] In one possible design, a pre-trained set of possible values ​​of β and the corresponding inverse gain vector are obtained during the training phase and then stored on the decoding device. Figure 8 The method described above obtains the target inverse gain vector m v '.

[0298] In step 1104, an image is acquired based on the target inverse gain vector. After the target inverse gain vector is acquired in step 1103, the image is decoded using the target inverse gain vector. Fig.12 is a flow chart of an exemplary method for decoding an image based on a target inverse gain vector, Fig.10 Is executable Fig.12 An exemplary VAE framework of the method 1200 in FIG. 1200 . In general, the above embodiments and any possible designs or possible implementations thereof may be combined to provide more flexibility.

[0299] Fig.12 The decoding method 1200 shown includes: step 1201, using entropy decoding to parse the bitstream to obtain a first potential representation of the image; step 1202, obtaining a second potential representation based on the first potential representation and the target inverse gain vector; step 1203, using a neural network to decode the second potential representation to obtain the image.

[0300] In step 1201, an entropy decoding process is performed to convert the binarized numbers back into sample values ​​(i.e., the first potential representation, represented by In one possible implementation, entropy decoding is performed by Fig.10 The arithmetic decoding module 1006 shown in the figure provides. It should be noted that the arithmetic decoder (AD) 1006 is a specific implementation of entropy decoding. AD can be replaced by other entropy decoding methods. Generally, It is a tensor of shape w×h×d (for example, w×h×128 or w×h×64), where w represents the feature map width, h represents the feature map height, and d represents the number of feature map channels. It should be noted that the feature map width and the feature map height may be equal or unequal, which is not limited here. The feature map channel may also be other integer values, such as 32, 48, 96, 144, 160, 176, 192, 256, etc., which is not limited in the present invention. In one possible design, the number of feature map channels for luminance samples and chrominance samples may be different, such as 128 luminance samples and 64 chrominance samples.

[0301] In step 1202, after obtaining the first potential representation of the image in step 1201, the first potential representation may be further calculated based on the target inverse gain vector. The target inverse gain vector may be the target inverse gain vector obtained in step 1103 of method 1100. In one possible implementation, the function of step 1202 may be performed by an inverse gain unit 1013 after AD 1006. The input of the gain unit 1013 (in The inverse gain unit 1013 also transforms the first potential representation with the target inverse gain vector. As described in step 1202, the inverse gain vector is a vector of dimension 1×d, where d is an integer greater than 1. In the present invention, d is equal to Therefore, each element of the target inverse gain vector can be mapped to In one possible implementation, the inverse gain unit 1013 multiplies the target inverse gain vector by the first potential representation to obtain a second potential representation. The output of the inverse gain unit 1013 is represented as y'.

[0302] It should be noted that step 1103 needs to be performed before step 1202, and the actions described in other steps of methods 1100 and 1200 can be performed in different orders and still obtain the desired results. For example, the process described in the accompanying drawings does not necessarily require the specific order shown or the sequential execution to achieve the desired results. In some implementations, it may be advantageous to perform multitasking and parallel processing. For example, step 1102 may be performed before step 1201, or step 1102 may be performed after step 1201, or step 1102 and step 1201 may be performed in parallel.

[0303] In one possible embodiment, in step 1103'( Fig.11 (not shown) to obtain the target gain vector, and then in step 1104' ( Fig.11 (not shown) decodes the bitstream based on the target gain vector to obtain an image. In this embodiment, the inverse gain unit 1013 multiplies the inverse of the target inverse gain vector by the first potential representation to obtain a second potential representation.

[0304] In step 1203, the second latent representation obtained in step 1202 is further decoded into an image using a neural network. In the VAE framework, the function of the neural network can be expressed as Fig.10 The decoder 1004 shown is implemented as shown. Although the unit 1004 is referred to as a "decoder", it can also be Fig.10 The complete decoding network described in is called the "decoder". The process of decoding generally refers to the unit (module) that converts the latent representation into an image output.

[0305] The output of step 1203 is the image The more similar it is to x, the better.

[0306] Optionally, when the side information z output by the super encoder 1003 is processed by the gain unit 1012 before quantization. The inverse gain unit 1014 is used to process Fig.10 The output of AD 1010.

[0307] As is clear to those skilled in the art, this embodiment can be combined with any of the above embodiments and any possible designs or possible implementations thereof. Side information can also be provided.

[0308] Although the drawings describe operations in a particular order, this should not be construed as requiring that such operations be performed in the particular order shown or in sequence, or that all operations shown be performed to achieve the desired result. In some cases, it may be advantageous to perform multitasking and parallel processing. In addition, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments. It should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0309] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the operations described in the claims may be performed in a different order and still achieve the desired results. For example, the processes described in the accompanying drawings do not necessarily require that they be performed in the specific order shown or in sequence to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0310] based on Fig.10 The framework shown provides a simple example to illustrate the technical effect of the provided method. Assume that the output of encoder 1001 is y=8.53,

[0311] Traditional method: do not use the gain vector to process y, then quantize After going through lossless AE 1005 and lossless AD1006, we got The loss in the latent representation space is (9–8.53) / 8.53=5.5%;

[0312] Proposed method: Use the gain vector (e.g. m = 5) to process y and obtain Afterwards quantification After going through lossless AE 1005 and lossless AD 1006, we got Then use the inverse gain vector (e.g. m'=1 / 5) to process We obtain y'=43*1 / 5=8.6, and the loss of the latent representation space is (8.6–8.53) / 8.53=0.8%.

[0313] As shown in the above simple example, the method proposed in the present invention leads to much smaller distortion, especially when the gain vector has a large value.

[0314] Encoding and decoding equipment

[0315] In addition, as mentioned above, the present invention also provides a device for executing the steps of the above method. Fig.13An encoding device 1300 for encoding processing by a neural network-based unit is shown. The device 1300 includes: a feature map acquisition module 1310, for acquiring a first feature map from an input image; a gain unit 1330, for transforming the first feature map based on a target gain vector to obtain a second feature map; a quantization module 1340, for quantizing the second feature map to obtain a quantized second feature map; an entropy coding module 1350, for encoding the quantized second feature map to obtain a code stream (e.g., by entropy coding). Optionally, the device 1300 may also include a gain vector acquisition module 1320, for acquiring a target gain vector based on a first decoding parameter.

[0316] Corresponding to the above-mentioned encoding device 1300, Fig.14 A decoding device 1400 for decoding a bitstream to reconstruct an image by a unit based on a neural network is shown in FIG. The device may include: an entropy decoding module 1410 for parsing the bitstream using entropy decoding to obtain a first potential representation of the image; an inverse gain unit 1430 for obtaining a second potential representation based on the first potential representation and a target inverse gain vector; and an image reconstruction module for decoding the second potential representation using a neural network to obtain the image. Optionally, the device 1400 may also include an inverse gain vector acquisition module 1420 for acquiring a target inverse gain vector based on the first decoding parameter.

[0317] It should be noted that these devices can also be used to perform any additional features including the above exemplary implementations. For example, a device for decoding a feature map for processing by a neural network based on a bitstream is provided, and the device includes a processing circuit for performing the steps of any of the above decoding methods. Similarly, a device for encoding a feature map for processing by a neural network into a bitstream is provided, and the device includes a processing circuit for performing the steps of any of the above encoding methods.

[0318] Other devices utilizing the devices 1300 and / or 1400 may be provided. For example, a device for image or video encoding may include the encoding device 1300. Furthermore, it may include the decoding device 1400. A device for image or video decoding may include the decoding device 1400 and / or the encoding device 1300.

[0319] In addition, a decoding system using the device 1300 and / or 1400 may be provided. For example, the decoding system may be deployed in a server. The server receives the code stream, then decodes the code stream using the device 1400 or other decoders to obtain an image, and then encodes the image using the device 1300 or other encoders. After encoding, the newly obtained code stream is stored and / or transmitted to other devices.

[0320] Some example implementations in hardware and software

[0321] Fig.15 A corresponding system is shown in which the above-described encoder-decoder processing chain may be deployed. Fig.15 1 is a schematic block diagram of an exemplary decoding system (e.g., video, image, audio and / or other decoding system) that can use the technology in the present application. The video encoder 20 (or simply referred to as encoder 20) and the video decoder 30 (or simply referred to as decoder 30) in the video decoding system 10 are examples of devices that can be used to perform various techniques according to various examples described in the present invention. For example, video encoding and decoding can use a neural network, which can be distributed, and the above-mentioned bitstream parsing and / or bitstream generation can be applied to transmit feature maps between distributed computing nodes (two or more).

[0322] like Fig.15 As shown, the decoding system 10 includes a source device 12 for providing the encoded image data 21 to a destination device 14 or the like for decoding the encoded image data 13 .

[0323] The source device 12 includes an encoder 20 and may additionally (ie, optionally) include an image source 16 , a pre-processor (or pre-processing unit) 18 (eg, an image pre-processor 18 ), and a communication interface or communication unit 22 .

[0324] The image source 16 may include or may be any type of image capture device such as a camera for capturing real-world images, and / or any type of image generation device such as a computer graphics processor for generating computer-animated images, or any type of other device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory (memory / storage) for storing any of the above images.

[0325] In order to distinguish the pre-processor 18 and the processing performed by the pre-processing unit 18 , the image or image data 17 may also be referred to as a raw image or raw image data 17 .

[0326] The preprocessor 18 is used to receive (raw) image data 17 and perform preprocessing on the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. The preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color adjustment or denoising, etc. It is understood that the preprocessing unit 18 may be an optional component. It should be noted that the preprocessing may also utilize a neural network using presence indicator signaling (e.g., in Figures 1 to 7 any one of ).

[0327] The video encoder 20 is configured to receive the pre-processed image data 19 and provide encoded image data 21 .

[0328] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or data obtained after further processing of the encoded image data 21) to another device, such as the destination device 14 or any other device, through the communication channel 13 for storage or direct reconstruction.

[0329] The destination device 14 includes a decoder 30 (eg, a video decoder 30 ), and may additionally (ie, optionally) include a communication interface or communication unit 28 , a post-processor 32 (or a post-processing unit 32 ), and a display device 34 .

[0330] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or data obtained after further processing of the encoded image data 21) directly from the source device 12 or from any other source such as a storage device (such as an encoded image data storage device), and provide the encoded image data 21 to the decoder 30.

[0331] Communication interface 22 and communication interface 28 can be used to send or receive encoded image data 21 or encoded data 13 via a direct communication link (e.g., a direct wired or wireless connection) between source device 12 and destination device 14 or via any type of network (e.g., a wired network or a wireless network or any combination thereof, or any type of private network and public network, or any type of combination thereof).

[0332] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format (eg, data packets), and / or process the encoded image data by any type of transmission coding or processing for transmission over a communication link or communication network.

[0333] For example, the communication interface 28 corresponding to the communication interface 22 may be used to receive the transmission data and process the transmission data using any type of corresponding transmission decoding or processing manner and / or decapsulation manner to obtain the encoded image data 21 .

[0334] Both the communication interface 22 and the communication interface 28 can be configured as Fig.15The decoder 30 is a unidirectional communication interface represented by an arrow pointing from the source device 12 to the destination device 14 in the communication channel 13, or a bidirectional communication interface, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission (such as encoded image data transmission), etc. The decoder 30 is used to receive the encoded image data 21 and provide decoded image data 31 or decoded image 31.

[0335] The post-processor 32 in the destination device 14 is used to post-process the decoded image data 31 (also referred to as reconstructed image data) (e.g., the decoded image 31) to obtain post-processed image data 33 (e.g., the post-processed image 33). The post-processing performed by the post-processing unit 32 may include color format conversion (e.g., from YCbCr to RGB), color adjustment, cropping or resampling, or any other processing to provide the decoded image data 31 for display by the display device 34, etc.

[0336] The display device 34 in the destination device 14 is used to receive the post-processed image data 33 so as to display the image to a user or viewer, etc. The display device 34 can be or include any type of display for representing the reconstructed image, such as an integrated or external display or display screen. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display.

[0337] although Fig.15 The source device 12 and the destination device 14 are described as separate devices, but the device embodiments may also include two devices or two functions, namely the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions. In these embodiments, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0338] According to the description, Fig.15 It will be apparent to a skilled person that the different units or functions shown and (accurately) divided in the source device 12 and / or the destination device 14 may vary depending on the actual device and application.

[0339] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30 may be implemented by processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video decoding dedicated processors, or any combination thereof. The encoder 20 may be implemented by processing circuitry 46 to embody various modules including a neural network or portions thereof. The decoder 30 may be implemented by processing circuitry 46 to embody any decoding system or subsystem described herein. The processing circuitry may be used to perform various operations discussed below. If these techniques are implemented in part in software, the device may store the software instructions in a suitable non-transitory computer-readable storage medium, and the instructions may be executed in hardware by one or more processors to perform the techniques of the present invention. Either of the video encoder 20 and the video decoder 30 may be integrated into a single device as part of a combined encoder / decoder (CODEC), such as Fig.16 shown.

[0340] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or a tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content distribution server), a broadcast receiver device, a broadcast transmitter device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0341] In some cases, Fig.15The video decoding system 10 shown is merely an example, and the techniques of the present invention may be applicable to video decoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data may be retrieved from a local memory, transmitted over a network, etc. A video encoding device may encode data and store the data in a memory, and / or a video decoding device may retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but merely encode data to a memory and / or retrieve data from a memory and decode the data.

[0342] Fig.17 Schematic diagram of a video decoding device 8000 provided in an embodiment of the present invention. The video decoding device 8000 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 8000 may be a decoder (e.g. Fig.15 A video decoder 30) or an encoder (such as Fig.15 video encoder 20).

[0343] The video decoding device 8000 includes an input port 8010 (or input port 8010) and a receiving unit (Rx) 8020 for receiving data; a processor, a logic unit or a central processing unit (CPU) 8030 for processing data; a transmitting unit (Tx) 8040 and an output port 8050 (or output port 8050) for transmitting data; and a memory 8060 for storing data. The video decoding device 8000 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the input port 8010, the receiving unit 8020, the transmitting unit 8040 and the output port 8050 for inputting and outputting signals or electrical signals.

[0344] The processor 8030 is implemented by hardware and software. The processor 8030 can be implemented as one or more CPU chips, one or more cores (e.g., multi-core processors), one or more FPGAs, one or more ASICs, and one or more DSPs. The processor 8030 communicates with the input port 8010, the receiving unit 8020, the sending unit 8040, the output port 8050, and the memory 8060. The processor 8030 includes a neural network-based codec 8070. The neural network-based codec 8070 implements the above-disclosed embodiments. For example, the neural network-based codec 8070 performs, processes, prepares, or provides various decoding operations. Therefore, the inclusion of the neural network-based codec 8070 provides substantial improvements to the functions of the video decoding device 8000 and affects the transformation of the video decoding device 8000 to different states. Optionally, the neural network-based codec 8070 is implemented with instructions stored in the memory 8060 and executed by the processor 8030.

[0345] The memory 8060 may include one or more disks, tape drives, and solid-state hard disks, and may be used as an overflow data storage device for storing such programs when such programs are selected for execution, and for storing instructions and data read during program execution. For example, the memory 8060 may be volatile and / or non-volatile, and may be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).

[0346] Fig.18 A simplified block diagram of an exemplary embodiment of an apparatus that can be used as Fig.15 Either or both of the source device 12 and the destination device 14 in .

[0347] The processor 9002 in the device 9000 may be a central processing unit. Alternatively, the processor 9002 may be any other type of device or devices that are currently available or will be developed in the future and are capable of manipulating or processing information. Although the disclosed implementations may be implemented using a single processor such as the processor 9002 shown in the figure, using more than one processor may improve speed and efficiency.

[0348] In one implementation, the memory 9004 in the apparatus 9000 may be a read only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 9004. The memory 9004 may include code and data 9006 accessed by the processor 9002 via a bus 9012. The memory 9004 may also include an operating system 9008 and an application 9010, the application 9010 including at least one program that enables the processor 9002 to perform the method described herein. For example, the application 9010 may include application 1 to N, including a video decoding application that performs the method described herein.

[0349] The apparatus 9000 may also include one or more output devices, such as a display 9018. In one example, the display 9018 may be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 9018 may be coupled to the processor 9002 via the bus 9012.

[0350] Although the bus 9012 of the device 9000 is described herein as a single bus, the bus 9012 may include multiple buses. In addition, the auxiliary memory may be directly coupled to other components in the device 9000 or may be accessed through a network, and may include a single integrated unit (e.g., one memory card) or multiple units (e.g., multiple memory cards). Therefore, the device 9000 may be implemented in a variety of configurations.

[0351] In a possible embodiment, an exemplary method for storing a code stream is provided, the method comprising:

[0352] Obtain a bit stream according to any of the encoding methods shown above;

[0353] The code stream is stored in a storage medium.

[0354] Optionally, the method further comprises:

[0355] Encrypt the code stream to obtain an encrypted code stream;

[0356] The encrypted code stream is stored in a storage medium.

[0357] It will be appreciated that any known encryption method may be used.

[0358] Optionally, the method further comprises:

[0359] The code stream is segmented to obtain a plurality of code stream segments; and the plurality of code stream segments are stored in a storage medium.

[0360] Optionally, the method further comprises:

[0361] At least one backup of the code stream is obtained, and the at least one backup is stored in a storage medium. It should be understood that the at least one backup of the code stream can be stored in a storage medium different from the storage medium storing the original code stream.

[0362] Optionally, the method further comprises:

[0363] Receiving a plurality of code streams generated according to any one of the encoding methods shown above;

[0364] individually allocating address information or identification information to the multiple code streams;

[0365] According to the address information or identification information corresponding to the plurality of code streams, the code streams are stored in corresponding positions.

[0366] Optionally, the method further comprises:

[0367] Classifying the code streams to obtain at least two code streams, wherein the at least two code streams include a first code stream and a second code stream;

[0368] The first code stream is stored in a first storage space, and the second code stream is stored in a second storage space.

[0369] Optionally, the method further comprises:

[0370] The video streaming device sends the code stream to the terminal device, wherein the video streaming device may be a content server or a content distribution server.

[0371] In a possible embodiment, an exemplary system for storing a code stream is provided, the system comprising:

[0372] A receiver, configured to receive a code stream generated by any one of the above encoding methods;

[0373] A processor, used for encrypting the code stream to obtain an encrypted code stream;

[0374] A computer-readable storage medium is used to store an encrypted code stream.

[0375] Optionally, the system includes several storage media, which can be deployed in different locations. Multiple code streams can be distributed and stored in different storage media. For example, the several storage media include: a first storage medium for storing a first code stream; a second storage medium for storing a second code stream.

[0376] Optionally, the system includes a video streaming device, wherein the video streaming device may be a content server or a content distribution server, and the video streaming device is used to obtain a code stream from one of the storage media and send the code stream to the terminal device.

[0377] In a possible embodiment, an exemplary method for converting a bitstream format is provided, the method comprising:

[0378] Receiving a code stream in a first format generated by any one of the encoding methods shown above;

[0379] Converting a bitstream in a first format into a bitstream in a second format;

[0380] The code stream in the second format is stored in a storage medium.

[0381] Optionally, the method further comprises:

[0382] In response to the access request of the terminal side device, the stored code stream in the second format is sent to the terminal side device.

[0383] In a possible embodiment, an exemplary system for converting a code stream format is provided, the system comprising:

[0384] A receiver, configured to receive a code stream in a first format generated by any one of the above encoding methods;

[0385] A processor, configured to convert a code stream in a first format into a code stream in a second format;

[0386] The processor is further configured to store the code stream in the second format into a storage medium;

[0387] The storage medium is used to store the code stream in the second format;

[0388] The transmitter is used to send the stored second format code stream to the terminal side device in response to the access request of the terminal side device.

[0389] In a possible embodiment, an exemplary method for processing a code stream is provided, the method comprising:

[0390] Receiving a transport stream including a video stream and an audio stream, wherein the video stream is generated by any one of the encoding methods shown above;

[0391] Demultiplexing the transport stream to separate the video stream and the audio stream;

[0392] Decoding the video stream using a video decoder to obtain video data;

[0393] The audio stream is decoded by using an audio decoder to obtain audio data.

[0394] Optionally, the method further comprises:

[0395] Synchronize audio data and video data;

[0396] Output the synchronization result to the player for playback.

[0397] Optionally, the method further comprises:

[0398] Decode the code stream to obtain video data or image data;

[0399] Performing at least one of brightness mapping, chrominance mapping, resolution adjustment, or format conversion on the video data or the image data;

[0400] Sends video data or image data to a display.

[0401] In a possible embodiment, an exemplary method for sending a code stream based on a user operation request is provided, the method comprising:

[0402] Receiving a first operation request from a terminal-side device, wherein the first operation request is used to request to play a target video;

[0403] In response to the first operation request, determining a code stream corresponding to the target video in the storage medium, wherein the code stream corresponding to the target video is a code stream generated according to any one of the above encoding methods;

[0404] Send the target code stream to the terminal side device.

[0405] Optionally, the method further comprises:

[0406] Encapsulating the code stream to obtain a transport stream in a first format;

[0407] Sending the transport stream in the first format to the terminal side device for display; or,

[0408] The transport stream in the first format is sent to the storage space for storage.

[0409] In a possible embodiment, an exemplary system for sending a code stream based on a user operation request is provided, the system comprising:

[0410] A storage medium, used for storing a code stream, wherein the code stream is a code stream generated according to any one of the above encoding methods;

[0411] A receiver, configured to receive a first operation request;

[0412] The processor is configured to determine a target code stream in a storage medium in response to the first operation request;

[0413] The transmitter is used to send the target code stream to the terminal side device.

[0414] Optionally, the processor is further configured to:

[0415] The code stream is encapsulated to obtain a transport stream in a first format; the system also includes a transmitter, which is used to:

[0416] Sending the transport stream in the first format to the terminal side device for display; or,

[0417] The transport stream in the first format is sent to the storage space for storage.

[0418] In a possible embodiment, an exemplary method for downloading a code stream is provided, the method comprising:

[0419] Obtaining a bitstream from a storage medium, wherein the bitstream is generated according to any one of the above encoding methods;

[0420] Decode the code stream to obtain a streaming media file;

[0421] Divide the streaming media file into multiple streaming media segments;

[0422] Download multiple streaming segments individually.

[0423] In a possible embodiment, an exemplary system for downloading a code stream is provided, the system comprising:

[0424] An acquisition unit, configured to acquire a code stream from a storage medium, wherein the code stream is generated according to any one of the above encoding methods;

[0425] A decoder is used to decode the code stream to obtain a streaming media file;

[0426] A processor, used for dividing a streaming media file into a plurality of streaming media segments;

[0427] A processor is used to download multiple streaming media segments separately. However, the present invention is not limited to any of these exemplary embodiments.

[0428] In summary, the present invention relates to a method and apparatus for encoding data (processing still or video into a bitstream). Specifically, the data is processed by a network including a gain unit. In the process, a feature map is generated by an encoder layer. Before quantization, the feature map is processed by a gain unit, which further transforms the feature map with a target gain vector. The target gain vector corresponds to a first decoding parameter, which is used to control the compression quality of the image. The user can set different values ​​for the first decoding parameter to control different compression qualities for different images. The user can set different values ​​for the first decoding parameter to control different compression qualities for luminance samples and chrominance samples. The first decoding parameter is also encoded into the bitstream. By this method, a flexible process that can operate on different bitstream sizes is provided. Therefore, according to the first decoding parameter, the data can be efficiently encoded in the bitstream, and the first decoding parameter can vary according to the content of the decoded image data.

[0429] The present invention also relates to a method and apparatus for decoding data (processing static or video into a bitstream). Specifically, the data is processed by a network including an inverse gain unit. In the process, a feature map is generated by an entropy decoder. Before further decoding, the feature map is processed by an inverse gain unit, which further transforms the feature map with a target inverse gain vector. The target inverse gain vector is obtained based on a first decoding parameter, which can be parsed from the bitstream.

[0430] The present invention provides an efficient method for decoding images or videos with different qualities and bitstream sizes without training more models.

Claims

1. An image coding method, characterized in that: include: Obtaining images; Obtaining a first decoding parameter for the image, wherein a value of the first decoding parameter is less than a preset minimum value or greater than a preset maximum value; Obtaining a target gain vector based on the first decoding parameter; The image is encoded based on the target gain vector.

2. The method according to claim 1, characterized in that When the value of the first decoding parameter is less than the preset minimum value, obtaining a target gain vector based on the first decoding parameter includes: The target gain vector is obtained based on the first decoding parameter, the preset minimum value, and a first gain vector corresponding to the preset minimum value.

3. The method according to claim 2, characterized in that The obtaining the target gain vector based on the first decoding parameter, the preset minimum value and the first gain vector comprises: Obtaining a first ratio of the first decoding parameter to the preset minimum value; The target gain vector is obtained based on the first ratio and the first gain vector.

4. The method according to claim 3, characterized in that: The obtaining the target gain vector based on the first ratio and the first gain vector comprises: The first ratio is multiplied by the first gain vector to obtain the target gain vector.

5. The method according to any one of claims 2 to 4, characterized in that The target gain vector satisfies the following conditions: Among them, m v is the target gain vector, β s is the preset minimum value, β v is the first decoding parameter, m s is the first gain vector corresponding to the preset minimum value, and K is a preset value.

6. The method according to claim 1, characterized in that When the value of the first decoding parameter is greater than the preset maximum value, obtaining a target gain vector based on the first decoding parameter includes: The target gain vector is obtained based on the first decoding parameter, the preset maximum value and a second gain vector corresponding to the preset maximum value.

7. The method according to claim 6, characterized in that The obtaining the target gain vector based on the first decoding parameter, the preset maximum value and the second gain vector comprises: Obtaining a second ratio of the first decoding parameter to the preset maximum value; The target gain vector is obtained based on the second ratio and the second gain vector.

8. The method according to claim 7, characterized in that The obtaining the target gain vector based on the second ratio and the second gain vector comprises: The second ratio is multiplied by the second gain vector to obtain the target gain vector.

9. The method according to any one of claims 6 to 8, characterized in that The target gain vector satisfies the following conditions: Among them, m v is the target gain vector, β t is the preset maximum value, β v is the first decoding parameter, m t is the second gain vector corresponding to the preset maximum value, and K is a preset value.

10. The method according to claim 1, characterized in that When the value of the first decoding parameter is less than the preset minimum value, obtaining a target gain vector based on the first decoding parameter includes: The target gain vector is obtained based on the first decoding parameter, the preset minimum value, a first gain vector corresponding to the preset minimum value, a third preset value closest to the preset minimum value, and a third gain vector corresponding to the third preset value.

11. The method according to claim 1, characterized in that When the value of the first decoding parameter is greater than the preset maximum value, obtaining a target gain vector based on the first decoding parameter includes: The target gain vector is obtained based on the first decoding parameter, the preset maximum value, a second gain vector corresponding to the preset maximum value, a fourth preset value closest to the preset maximum value, and a fourth gain vector corresponding to the fourth preset value.

12. The method according to claim 1, characterized in that The obtaining a target gain vector based on the first decoding parameter comprises: The target gain vector is obtained based on the first decoding parameter, N preset values ​​and N gain vectors corresponding to the N preset values, wherein N is an integer greater than 2, and the N preset values ​​include the preset minimum value and / or the preset maximum value.

13. The method according to any one of claims 1 to 12, characterized in that The encoding of the image based on the target gain vector comprises: Obtain a first feature map of the image using a neural network; Based on the first feature map and the target gain vector, obtaining a second feature map; quantizing the second feature map to obtain a quantized second feature map; The quantized second feature map is encoded to obtain a code stream.

14. The method according to claim 13, characterized in that The obtaining of a second feature map based on the first feature map and the target gain vector comprises: The target gain vector is multiplied by the first feature map.

15. The method according to claim 13 or 14, characterized in that The first feature map is a tensor with a shape of w×h×d, and the target gain vector is a vector with a dimension of 1×d, where w and h represent the width and height of the first feature map, and d represents the number of channels of the first feature map.

16. The method according to claim 15, characterized in that When the first feature map is a feature map of luminance samples of the image, d is equal to 128; or when the first feature map is a feature map of chrominance samples of the image, d is equal to 64.

17. The method according to any one of claims 1 to 16, characterized in that The method further comprises obtaining a second decoding parameter for the image; When the first decoding parameter is used to encode luma samples of the image, the second decoding parameter is used to encode chroma samples of the image; or, When the first decoding parameters are used to encode chrominance samples of the image, the second decoding parameters are used to encode luma samples of the image.

18. The method according to any one of claims 1 to 17, characterized in that The method further comprises: The first decoding parameter is encoded into a bitstream.

19. The method according to claim 18, characterized in that The first decoding parameter is encoded in a picture parameter set (PPS) of the code stream.

20. The method according to claim 18 or 19, characterized in that The number of bits in the code stream used to indicate the first decoding parameter is less than or equal to 16.

21. An image decoding method, characterized in that: include: obtaining a code stream including decoded image data; Parsing the bitstream to obtain a first decoding parameter, wherein a value of the first decoding parameter is less than a preset minimum value or greater than a preset maximum value; Obtaining a target inverse gain vector based on the first decoding parameter; An image is obtained based on the target inverse gain vector.

22. The method according to claim 21, characterized in that When the value of the first decoding parameter is less than the preset minimum value, obtaining a target inverse gain vector based on the first decoding parameter includes: The target inverse gain vector is obtained based on the first decoding parameter, the preset minimum value, and a first gain vector corresponding to the preset minimum value.

23. The method according to claim 22, characterized in that The obtaining the target inverse gain vector based on the first decoding parameter, the preset minimum value and the first gain vector comprises: Obtaining a first ratio of the first decoding parameter to the preset minimum value; The target inverse gain vector is obtained based on the first ratio and the first gain vector.

24. The method according to claim 23, characterized in that The obtaining the target inverse gain vector based on the first ratio and the first gain vector comprises: multiplying the first ratio by the first gain vector to obtain a target gain vector; The target inverse gain vector is obtained based on the target gain vector.

25. The method according to claim 24, characterized in that The target gain vector satisfies the following conditions: Among them, m v is the target gain vector, β s is the preset minimum value, β v is the first decoding parameter, m s is the first gain vector corresponding to the preset minimum value, and K is a preset value.

26. The method according to claim 21, characterized in that When the value of the first decoding parameter is greater than a preset maximum value, obtaining a target inverse gain vector based on the first decoding parameter includes: The target inverse gain vector is obtained based on the first decoding parameter, the preset maximum value and a second gain vector corresponding to the preset maximum value.

27. The method according to claim 26, characterized in that The obtaining the target inverse gain vector based on the first decoding parameter, the preset maximum value and the second gain vector comprises: Obtaining a second ratio of the first decoding parameter to the preset maximum value; The target inverse gain vector is obtained based on the second ratio and the second gain vector.

28. The method according to claim 27, characterized in that The obtaining the target inverse gain vector based on the second ratio and the second gain vector comprises: multiplying the second ratio by the second gain vector to obtain a target gain vector; The target inverse gain vector is obtained based on the target gain vector.

29. The method according to claim 27, characterized in that The target gain vector satisfies the following conditions: Among them, m v is the target gain vector, β t is the preset maximum value, β v is the first decoding parameter, m t is the second gain vector corresponding to the preset maximum value, and K is a preset value.

30. The method according to claim 24 or 25 or claim 28 or 29, characterized in that The target inverse gain vector satisfies the following conditions: m' v *m v =C, Among them, m' v is the target inverse gain vector, n v is the target gain vector, C is a vector whose elements are all constants, and * represents an element-by-element multiplication operation.

31. The method according to claim 21, characterized in that When the value of the first decoding parameter is less than the preset minimum value, obtaining a target inverse gain vector based on the first decoding parameter includes: The target inverse gain vector is obtained based on the first decoding parameter, the preset minimum value, a first gain vector corresponding to the preset minimum value, a third preset value closest to the preset minimum value, and a third gain vector corresponding to the third preset value.

32. The method according to claim 21, characterized in that When the value of the first decoding parameter is greater than a preset maximum value, obtaining a target inverse gain vector based on the first decoding parameter includes: The target inverse gain vector is obtained based on the first decoding parameter, the preset maximum value, a second gain vector corresponding to the preset maximum value, a fourth preset value closest to the preset maximum value, and a fourth gain vector corresponding to the fourth preset value.

33. The method according to claim 21, characterized in that The obtaining a target inverse gain vector based on the first decoding parameter comprises: The target inverse gain vector is obtained based on the first decoding parameter, N preset values ​​and N gain vectors corresponding to the N preset values, wherein N is an integer greater than 2, and the N preset values ​​include the preset minimum value and / or the preset maximum value.

34. The method according to any one of claims 21 to 33, characterized in that Decoding the code stream based on the target inverse gain vector to obtain an image includes: Parsing the bitstream using entropy decoding to obtain a first latent representation of the image; Obtaining a second latent representation based on the first latent representation and the target inverse gain vector; The second latent representation is decoded using a neural network to obtain the image.

35. The method according to claim 34, characterized in that The obtaining of a second potential representation based on the first potential representation and the target inverse gain vector comprises: The target inverse gain vector is multiplied by the first latent representation.

36. The method according to claim 34 or 35, characterized in that The first potential representation is a tensor with a shape of w×h×d, and the target inverse gain vector is a vector with a dimension of 1×d, where w and h represent the width and height of the first feature map, and d represents the number of channels of the first feature map.

37. The method according to claim 36, characterized in that When the first latent representation is a feature map of luma samples of the image, d is equal to 128; or when the first latent representation is a feature map of chroma samples of the image, d is equal to 64.

38. The method according to any one of claims 21 to 37, characterized in that The method further comprises parsing the code stream to obtain a second decoding parameter; When the first decoding parameter is used to decode luma samples of the image, the second decoding parameter is used to decode chroma samples of the image; or, When the first decoding parameters are used to decode chrominance samples of the image, the second decoding parameters are used to decode luma samples of the image.

39. A computer program product comprising program code, characterized in that The program code is used to perform the method according to any one of claims 1 to 38 when executed on one or more processors.

40. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program executable by one or more processors. When the computer program is executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 38.

41. A decoder, characterized in that: Comprising a processing circuit for performing a method according to any one of claims 1 to 38.

42. A storage medium, characterized in that The storage medium stores a code stream obtained by using the method according to any one of claims 1 to 20.

43. A coded bit stream, characterized in that The method comprises decoded image data and a plurality of syntax elements, wherein the plurality of syntax elements comprises a flag, wherein the flag is used to derive a value of a decoding parameter indicating a compression quality of the decoded image data, wherein the value of the decoding parameter is less than a preset minimum value or greater than a preset maximum value.

44. A coded bit stream, characterized in that The invention comprises decoded image data and a plurality of syntax elements, wherein the plurality of syntax elements comprises a first flag and a second flag, wherein the first flag is used to derive a value of a first decoding parameter indicating a compression quality of a luma sample of the decoded image data, and the second flag is used to derive a value of a second decoding parameter indicating a compression quality of a chroma sample of the decoded image data.

45. A coding device, characterized in that include: a memory including instructions; A processor is coupled to the memory, and is configured to implement the instructions to enable the encoding device to perform the encoding method according to any one of claims 1 to 20.

46. ​​A decoding device, characterized in that include: a memory including instructions; A processor is coupled to the memory, and is configured to implement the instructions to enable the decoding device to perform the decoding method according to any one of claims 21 to 38.

47. A decoding device, characterized in that: include: A receiving unit, used for receiving an image for encoding or receiving a code stream for decoding; A sending unit, coupled to the receiving unit, configured to send the code stream to a decoder or send a decoded image to a display; a memory, coupled to at least one of the receiving unit or the sending unit, the memory being used to store instructions; A processor, coupled to the memory, configured to execute the instructions stored in the memory to perform the method according to any one of claims 1 to 20 and any one of claims 21 to 38.

48. A coding device, characterized in that include: A feature map acquisition module, used to obtain a first feature map from an input image; A gain unit, configured to transform the first feature map based on a target gain vector to obtain a second feature map; A quantization module, configured to quantize the second feature map to obtain a quantized second feature map; The entropy coding module is used to encode the quantized second feature map to obtain a code stream.

49. A decoding device, characterized in that include: An entropy decoding module, configured to parse the bitstream using entropy decoding to obtain a first latent representation of the image; an inverse gain unit, configured to obtain a second potential representation based on the first potential representation and a target inverse gain vector; An image reconstruction module is used to decode the second latent representation using a neural network to obtain the image.

50. A decoding system, characterized in that: include: Encoder; A decoder communicating with the encoder, wherein the encoder or the decoder comprises a decoding device, an encoding device or a decoding apparatus according to any one of claims 45 to 49.

Images