Methods and apparatus for image coding and decoding

Abstract

This disclosure provides techniques for improving signaling efficiency in the context of rate control, which manages a desired ratio of bitrate load to quality in image compression. A method for encoding an image is provided, comprising: acquiring an image; encoding the image into a bitstream based on a first coding parameter, wherein the first coding parameter represents a first displacement between a first target rate control parameter and a first reference rate control parameter; and encoding the first coding parameter into a bitstream. Furthermore, a method for decoding an image is provided, comprising: receiving a bitstream containing coded data of an image; parsing the bitstream to acquire a first coding parameter, wherein the first coding parameter represents a first displacement between a first target rate control parameter and a first reference rate control parameter; and reconstructing the image based on the first coding parameter.

Description

[Technical Field]

[0001] This application claims priority to international application PCT / EP2023 / 060200, filed on 19 April 2023, titled "Method and apparatus for image encoding and decoding," and international application PCT / CN2023 / 112712, filed on 11 August 2023, also titled "Method and apparatus for image encoding and decoding," both of which are incorporated herein by reference.

[0002] Embodiments of this disclosure generally relate to the field of encoding and decoding data based on neural network architectures. More specifically, some embodiments relate to methods and apparatus for encoding images and / or videos into and from bitstreams using multiple processing layers. [Background technology]

[0003] Hybrid image and video codecs have been used for decades to compress image and video data. In such codecs, signals are encoded block by block, typically by predicting blocks and then further coding only the differences between the original blocks and their predictions. In detail, such coding may involve transformation, quantization, and generating a bitstream, usually including some entropy coding. Typically, the three components of a hybrid coding method—transformation, quantization, and entropy coding—are optimized separately. Modern video compression standards such as High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC) also use transformed representations to code the residual signal after prediction.

[0004] Recently, neural network architectures have been applied to image and / or video coding. Generally, these neural network (NN)-based methods can be applied to image and video coding in a variety of different ways. For example, several end-to-end optimized image or video coding frameworks have been discussed. Furthermore, deep learning is used to determine or optimize certain parts of the end-to-end coding framework, such as the selection or compression of prediction parameters. In addition, several neural network-based methods have been discussed for use in hybrid image and video coding frameworks, for example, as trained deep learning models for intra- or inter-prediction in image or video coding.

[0005] A common feature of the end-to-end optimized image or video coding applications described above is that they generate some feature map data that will be transmitted between the encoder and decoder.

[0006] A neural network is a machine learning model that employs one or more layers of nonlinear units that can predict an output based on an incoming input. Some neural networks include one or more hidden layers in addition to an output layer. A corresponding feature map may be provided as the output of each hidden layer. Such a corresponding feature map of each hidden layer may be used as input to a subsequent layer in the network, i.e., a subsequent hidden layer or output layer. Each layer of the network generates an output from an incoming input according to the current values ​​of each set of parameters. In a neural network that is split between devices, for example, between an encoder and a decoder, between a device and the cloud, or between different devices, the feature map at the output of the split location (e.g., the first device) is compressed and sent to the remaining layers of the neural network (e.g., to the second device).

[0007] Further improvements in encoding and decoding using trained network architectures are sometimes desirable. [Overview of the project] [Means for solving the problem]

[0008] More specifically, this specification focuses on improving signaling efficiency in the context of rate control, which manages a desired ratio of bitrate load to quality in image compression.

[0009] The present invention is presented in an appendix of claims. The above and other objectives are achieved by the subject matter of the independent claims. Further implementations can be identified from the dependent claims, description, and drawings.

[0010] According to a first aspect, a method for encoding an image is provided, comprising the steps of: acquiring an image; encoding the image into a bitstream based on a first coding parameter, wherein the first coding parameter represents a first displacement between a first target rate control parameter and a first reference rate control parameter; and encoding the first coding parameter into a bitstream.

[0011] Here and below, an image can be, for example, a still image / picture or a video frame.

[0012] The first coding parameter may be directly encoded and signaled in the bitstream, or another coding parameter obtained based on the first coding parameter may be encoded and signaled in the bitstream.

[0013] The first displacement may be a difference or a ratio. The first target rate control parameter may be a rate control parameter selected by an encoder performing a method for encoding an image, according to the first embodiment and any implementation thereof, and the first reference rate control parameter may be a rate control parameter used in model training. Here and below, the rate control parameter is a parameter used to control the ratio between bitrate and distortion (see also the detailed description below).

[0014] In detail, the first target rate control parameter may be a weighting coefficient between the bitrate and the distortion used in the loss function during training.

[0015] Unlike the prior art, the method for encoding an image, according to the first embodiment and any implementation thereof, signals a first displacement between a first target rate control parameter and a first reference rate control parameter (rather than signaling a rate control parameter selected by the encoder), which results in a significant reduction in the number of bits that need to be signaled for rate control. According to one implementation, the first coding parameter is signaled using N bits, where N is an integer less than or equal to 16. For example, N is equal to 12. Hereinafter, when the first coding parameter is represented by a fixed-point value having N bits, K bits of the N bits represent the fractional part of the first coding parameter, and (NK) bits of the N bits represent the integer part of the first coding parameter, where N is an integer and K is an integer less than or equal to N.

[0016] On the decoder side, the number of operations required to obtain the target gain vector (see detailed explanation below) is also significantly reduced compared to prior art.

[0017] In one implementation, the first coding parameter is represented by a fixed-point value. This allows all calculations based on the signaled first coding parameter involved in rate control to be constrained to fixed-point arithmetic calculations, which can guarantee bit-exact computational behavior on different devices. Unlike prior art, floating-point calculations are not required, for example, to obtain the gain vector. In this context, it should be noted that while fixed-point representations represent and handle numbers in integer format, floating-point representations handle floating-point arithmetic in addition to integer arithmetic; that is, numbers are represented by a combination of a mantissa (i.e., fractional part) and an exponent part, and as a result, the processor requires hardware to handle both of these parts. Consequently, handling floating-point operations involves more logical elements and more cycles (more time) to handle floating-point values. Therefore, the constraint to fixed-point arithmetic saves cost and computation time, in addition to providing arithmetic stability and independence of numerical results from the actual hardware used.

[0018] In one implementation, the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are used to code the rumor data (rumor components) of the image.

[0019] In one implementation, the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are for coding the first-order component of the image, and the first-order component may be the lumen component.

[0020] In principle, rate control is related to both the lumens and chroma components. Therefore, the first embodiment and any of the implementations described above may also include signaling related to the chroma rate control parameter.

[0021] According to one implementation, the method further includes the step of encoding a second coding parameter into a bitstream, the second coding parameter representing a second displacement between a second target rate control parameter and a second reference rate control parameter, the second coding parameter, the second target rate control parameter, and the second reference rate control parameter being for coding the chroma data (chroma components) of the image.

[0022] The second displacement may be a difference or a ratio. The second target rate control parameter may be a rate control parameter selected by an encoder performing a method for encoding an image, according to the first embodiment and any implementation thereof, and the second reference rate control parameter may be a rate control parameter used in model training. In detail, the second target rate control parameter may be a weighting coefficient between bitrate and strain used in the loss function during training.

[0023] In another implementation, the method further includes the step of encoding a second coding parameter into a bitstream, the second coding parameter representing a second displacement between a second target rate control parameter and a second reference rate control parameter, the second coding parameter, the second target rate control parameter, and the second reference rate control parameter being for coding a secondary component of the image, such as the chroma component.

[0024] In this case as well, the second displacement may be a difference or a ratio, the second target rate control parameter may be a rate control parameter selected by an encoder performing a method for encoding an image according to the first embodiment and any implementation thereof, and the second reference rate control parameter may be a rate control parameter used in model training. In detail, the second reference rate control parameter may be a weighting coefficient between bitrate and strain used in the loss function during training.

[0025] Rate control based on a second coding parameter offers advantages similar to those related to rate control based on a first coding parameter, as described above.

[0026] In principle, the same or different rate control parameters may be used for lumar rate control and chroma rate control. Therefore, according to one implementation, the encoding of the second coding parameter is based on a flag indicating that different rate control parameters (e.g., target rate control parameter, reference rate control parameter, and thus the displacement between the target rate control parameter and the reference rate control parameter) are used to code the lumar and chroma data of the image. In most applications, the reference rate control parameters for the first and second-order components (lumar and chroma) are the same, but the target rate control parameters may be different, and therefore the displacements are also different.

[0027] In another implementation, the coding of the second coding parameter is based on a flag indicating that different rate control parameters (e.g., a target rate control parameter, a reference rate control parameter, and thus a displacement between the target rate control parameter and the reference rate control parameter) are used to code the primary (e.g., lumar) and secondary (e.g., chroma) components of the image.

[0028] In one implementation, the second coding parameter, like the first coding parameter, is represented by a fixed-point value.

[0029] The method according to either the first embodiment or any implementation thereof may also include the step of encoding a third coding parameter into a bitstream. The third coding parameter may indicate an index of a trained model associated with at least one of the first and second reference rate control parameters. Depending on the actual application, different trained models may be deemed suitable for accurate image reproduction, and the selection of the most appropriate model can be achieved based on the signaled third coding parameter. According to one implementation, the third coding parameter may be used to derive a target gain vector.

[0030] According to one implementation, at least one of the first coding parameter and the second coding parameter is the following code, i.e., - Binary code, - Unary code, - Abbreviated unary codes, and - Exponential Golomb code Signaling is performed using one or more of the following.

[0031] In one implementation, the first coding parameter is on a logarithmic scale (i.e., it is expressed logarithmically). In another implementation, at least one of the first coding parameter, the second coding parameter, the first target rate control parameter, the second target rate control parameter, the first reference rate control parameter, and the second reference rate control parameter is on a logarithmic scale. Calculations in the logarithmic domain may be performed particularly effectively. One reason for this is that linear domain multiplication corresponds to addition in the logarithmic domain. Therefore, when calculations are performed in the logarithmic domain, more complex operations can be replaced by less complex operations. In detail, addition to the gain vector may be used in the logarithmic domain instead of multiplication by the gain vector.

[0032] According to one implementation, the step of encoding a first coding parameter into a bitstream includes the steps of obtaining a modified value of the first coding parameter and encoding the modified value of the first coding parameter into a bitstream. According to an alternative implementation, the modified value may be the value of any other coding parameter listed above, which can be obtained based on the first coding parameter, encoded directly into a bitstream, and signaled. For example, the modified value of the first coding parameter may be the value of the first coding parameter multiplied by 2 N-1 It is obtained by adding, where N is the number of bits used to signal the first coding parameter.

[0033] According to one implementation, the step of encoding an image into a bitstream based on a first coding parameter includes the step of obtaining a target gain vector based on the first coding parameter, and the step of encoding an image based on the target gain vector.

[0034] According to one implementation, the target gain vector is as follows: m log =m t log +betaDisplacementLog It is defined as, m log represents the target gain vector, m t log `betaDisplacementLog` represents a reference forward gain vector for a model having index t, where t represents the index of the trained model related to at least one of the first and second reference rate control parameters, and `betaDisplacementLog` is related to the first coding parameter. More specifically, `betaDisplacementLog` may represent the first coding parameter, or any other coding parameter listed above that is obtained based on the first coding parameter.

[0035] Where deemed appropriate, the implementation forms of methods for encoding images can be combined.

[0036] A method for decoding an image is provided, which is complementary to and offers similar advantages to a method for encoding an image, according to a first embodiment and any implementation thereof.

[0037] A method for decoding an image is provided, comprising the steps of: receiving a bitstream containing coded data of an image; parsing the bitstream to obtain a first coding parameter, wherein the first coding parameter indicates a first displacement between a first target rate control parameter and a first reference rate control parameter; and reconstructing the image based on the first coding parameter.

[0038] The first displacement may be a difference or a ratio. The first target rate control parameter may be a rate control parameter selected by an encoder performing a method for encoding an image, according to the first embodiment and any implementation thereof, and the first reference rate control parameter may be a rate control parameter used in model training. In detail, the first target rate control parameter may be a weighting coefficient between bitrate and strain used in the loss function during training.

[0039] According to one option, the step of parsing a bitstream to obtain a first coding parameter includes the steps of parsing another coding parameter (syntax element) from the bitstream and obtaining the first coding parameter based on that other coding parameter. This option is shown below as Option A.

[0040] According to another option, the step of parsing the bitstream to obtain the first coding parameter includes the step of directly parsing the first coding parameter from the bitstream. This option is shown below as Option B.

[0041] According to option B, the step of reconstructing an image based on a first coding parameter may include the steps of obtaining another coding parameter based on the parsed first coding parameter, and reconstructing the image based on that other coding parameter.

[0042] The first coding parameter may be represented by a fixed-point value. The first coding parameter may be signaled using N bits, where N is an integer less than or equal to 16. For example, N is equal to 12.

[0043] The first displacement may be a difference or a ratio. The first target rate control parameter may be a rate control parameter selected by an encoder performing a method for encoding an image, according to the first embodiment and any implementation thereof, and the first reference rate control parameter may be a rate control parameter used in model training.

[0044] In one implementation, the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are used to code the rumor data (rumor components) of the image.

[0045] In one implementation, the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are for coding the first-order component of the image, for example, the lumen component.

[0046] According to one implementation, the method for decoding an image further includes the step of parsing the bitstream to obtain a second coding parameter, the second coding parameter representing a second displacement between a second target rate control parameter and a second reference rate control parameter, the second coding parameter, the second target rate control parameter, and the second reference rate control parameter being used to code the chroma data (chroma components) of the image.

[0047] The second displacement may be a difference or a ratio. The second target rate control parameter may be a rate control parameter selected by an encoder performing the method according to the first embodiment and any implementation thereof, and the second reference rate control parameter may be a rate control parameter used in model training. In detail, the second target rate control parameter may be a weighting coefficient between bitrate and strain used in the loss function during training.

[0048] According to one implementation, a method for decoding an image further includes the step of parsing a bitstream to obtain a second coding parameter, the second coding parameter representing a second displacement between a second target rate control parameter and a second reference rate control parameter, the second coding parameter, the second target rate control parameter, and the second reference rate control parameter being for coding a secondary component of the image, such as the chroma component.

[0049] In this case as well, the second displacement may be a difference or a ratio, the second target rate control parameter may be a rate control parameter selected by an encoder performing the method according to the first embodiment and any implementation thereof, and the second reference rate control parameter may be a rate control parameter used in model training.

[0050] In one implementation, obtaining the second coding parameter is based on a flag indicating that different rate control parameters (e.g., a target rate control parameter, a reference rate control parameter, and therefore a displacement between the target rate control parameter and the reference rate control parameter) are used to code the rumor and chroma data of the image.

[0051] In one implementation, obtaining the second coding parameter is based on a flag indicating that different rate control parameters (e.g., a target rate control parameter, a reference rate control parameter, and therefore the displacement between the target rate control parameter and the reference rate control parameter) are used to code the primary and secondary components of the image.

[0052] Similar to the first coding parameter, the second coding parameter may be expressed as a fixed-point value.

[0053] According to one implementation, a method for decoding an image further includes the step of parsing a bitstream to obtain a third coding parameter. The third coding parameter may represent an index of a trained model associated with at least one of a first reference rate control parameter and a second reference rate control parameter. According to one implementation, the third coding parameter may be used to derive a target gain vector.

[0054] According to one implementation, at least one of the first coding parameter and the second coding parameter is the following code, i.e., - Binary code, - Unary code, - Abbreviated unary codes, and - Exponential Golomb Code Signaling is performed using one or more of the following.

[0055] According to one implementation form, the first coding parameter is in a logarithmic scale. According to another implementation form, at least one of the first coding parameter, the second coding parameter, the first target rate control parameter, the second target rate control parameter, the first reference rate control parameter, and the second reference rate control parameter is in a logarithmic scale.

[0056] According to one implementation form, the step of reconstructing an image based on the first coding parameter includes the step of obtaining an intermediate parameter by subtracting 2 from the first coding parameter, where N is the number of bits used to signal the first coding parameter, and the step of reconstructing an image based on the intermediate parameter. This implementation form represents an example of Option B defined above, and the intermediate parameter is equal to other coding parameters. N-1 According to one implementation form, the step of reconstructing an image based on the first coding parameter includes the step of obtaining a target gain vector based on the first coding parameter and the step of reconstructing an image based on the target gain vector. According to one implementation form, the target gain vector is defined as follows, that is,

[0057] m m log =m t log +betaDisplacementLog where m log represents the target gain vector, and m t log`betaDisplacementLog` represents a reference forward gain vector for a model having index t, where t represents the index of the trained model related to at least one of a first reference rate control parameter and a second reference rate control parameter, and `betaDisplacementLog` is related to the first coding parameter. According to option A defined above, `betaDisplacementLog` may represent the first coding parameter. According to option B defined above, `betaDisplacementLog` may represent other coding parameters.

[0058] Where deemed appropriate, the implementation forms of methods for encoding images can be combined.

[0059] A method for encoding an image according to the first embodiment can be implemented in an encoding device, and a method for decoding an image according to the second embodiment can be implemented in a decoding device.

[0060] Furthermore, the following devices, computer program products, storage media, and bitstreams are provided that offer similar advantages to those described above.

[0061] According to a third aspect, an encoding device for encoding an image is provided, comprising a memory containing instructions and a processor coupled to the memory, wherein the processor is configured to execute instructions to cause the encoding device to perform an encoding method according to the first aspect or any implementation thereof.

[0062] According to a fourth aspect, a decoding device for decoding an image is provided, comprising a memory containing instructions and a processor coupled to the memory, wherein the processor is configured to execute instructions to cause the decoding device to perform a decoding method according to the second aspect or any implementation thereof.

[0063] A coding apparatus for coding an image is provided, comprising: a receiver unit configured to receive an image to be encoded or a bitstream to be decoded; a transmitter unit coupled to the receiver unit, configured to transmit a bitstream to a decoder or a decoded image to a display; a memory coupled to at least one of the receiver unit or the transmitter unit, configured to store instructions; and a processor coupled to the memory, wherein the processor is configured to execute instructions stored in the memory to perform a method according to the first or second embodiment or any implementation thereof.

[0064] According to a sixth aspect, an encoding device is provided for encoding an image, comprising an acquisition unit configured to acquire an image and an encoding unit configured to encode the image into a bitstream based on a first coding parameter, wherein the first coding parameter represents a first displacement between a first target rate control parameter and a first reference rate control parameter, and the encoding unit is further configured to encode the first coding parameter into a bitstream. The encoding device according to the sixth aspect may be configured to encode the first coding parameter directly into a bitstream, or to encode another coding parameter acquired based on the first coding parameter. The encoding device according to the sixth aspect may be configured to perform the method according to the first aspect or any implementation thereof.

[0065] According to the seventh aspect, a receiving unit configured to receive a bitstream containing coded data of an image, and a parsing unit configured to parse the bitstream and obtain a first coding parameter, wherein the first coding parameter indicates a first displacement between a first target rate control parameter and a first reference rate control parameter, A reconstruction unit configured to reconstruct an image based on a first coding parameter and A decoding device for decoding images is provided, which includes the following features.

[0066] According to one option, parsing a bitstream to obtain a first coding parameter by the parsing unit includes parsing another coding parameter (syntax element) from the bitstream and obtaining the first coding parameter based on that other coding parameter (see Option A as defined above). According to another option, parsing a bitstream to obtain a first coding parameter by the parsing unit includes directly parsing the first coding parameter from the bitstream (see Option B as defined above). In this case, the reconstruction unit may be configured to reconstruct an image based on the first coding parameter by obtaining another coding parameter based on the parsed first coding parameter and reconstructing the image based on that other coding parameter.

[0067] The decoding device according to the seventh embodiment may be configured to perform the method according to the second embodiment or any implementation thereof.

[0068] According to the eighth aspect, an encoder and The decoder that is communicating with the encoder and A coding system for coding an image is provided, comprising an encoder or decoder including a decoding device according to a fourth or seventh embodiment, an encoding device according to a third or sixth embodiment, or a coding device according to a fifth embodiment.

[0069] According to the ninth aspect, a computer program product is provided which comprises program code for performing the first or second aspect or any implementation thereof when executed on one or more processors.

[0070] According to the tenth aspect, a computer-readable storage medium is provided which stores a computer program that can be executed by one or more processors, and when the computer program is executed by one or more processors, one or more processors execute a method according to the first or second aspect or any implementation thereof.

[0071] According to the eleventh aspect, a coder is provided that includes a processing circuit for performing a method according to the first or second aspect or any implementation thereof.

[0072] According to the twelfth aspect, a storage medium is provided which stores a bitstream obtained by using the method according to the first aspect or any implementation thereof.

[0073] According to the 13th aspect, a bitstream is provided that includes coded data of an image and a first coding parameter, the first coding parameter indicating a first displacement between a first target rate control parameter and a first reference rate control parameter. The bitstream may include the first coding parameter directly, or it may include another coding parameter obtained based on the first coding parameter.

[0074] The first displacement may be a difference or a ratio. The first target rate control parameter may be a rate control parameter selected by an encoder performing a method for encoding an image, according to the first embodiment and any implementation thereof, and the first reference rate control parameter may be a rate control parameter used in model training.

[0075] The first coding parameter may be represented by a fixed-point value.

[0076] In one implementation, the first coding parameter is signaled using N bits, where N is an integer less than or equal to 16, for example, N is equal to 12.

[0077] In one implementation, the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are used to code the rumor data of the image.

[0078] In one implementation, the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are for coding the first-order component of the image, for example, the lumen component.

[0079] In one implementation, the bitstream further includes a second coding parameter, the second coding parameter representing a second displacement between a second target rate control parameter and a second reference rate control parameter, the second coding parameter, the second target rate control parameter, and the second reference rate control parameter are for coding the chroma data of the image.

[0080] The second displacement may be a difference or a ratio. The second target rate control parameter may be a rate control parameter selected by an encoder performing a method for encoding an image, according to the first embodiment and any implementation thereof, and the second reference rate control parameter may be a rate control parameter used in model training.

[0081] In one implementation, the bitstream further includes a second coding parameter, the second coding parameter representing a second displacement between a second target rate control parameter and a second reference rate control parameter, the second coding parameter, the second target rate control parameter, and the second reference rate control parameter are for coding the secondary components of the image, such as the chroma component.

[0082] In this case as well, the second displacement may be a difference or a ratio, the second target rate control parameter may be a rate control parameter selected by an encoder performing a method for encoding an image according to the first embodiment and any implementation thereof, and the second reference rate control parameter may be a rate control parameter used in model training.

[0083] According to one implementation, the bitstream further includes a flag indicating whether different rate control parameters (e.g., a target rate control parameter, a reference rate control parameter, and thus a displacement between the target rate control parameter and the reference rate control parameter) are used to code the rumor and chroma data of the image.

[0084] In one implementation, the bitstream further includes a flag indicating whether different rate control parameters (e.g., a target rate control parameter, a reference rate control parameter, and thus a displacement between the target rate control parameter and the reference rate control parameter) are used to code the primary and secondary components of the image.

[0085] In one implementation, the second coding parameter is represented by a fixed-point value.

[0086] In one implementation, the bitstream further includes a third coding parameter, which indicates an index of the trained model associated with at least one of the first and second reference rate control parameters.

[0087] Where appropriate, the individual implementations of the bitstream according to the 13th aspect can be combined with each other.

[0088] According to the 14th aspect, a system for distributing a bitstream is provided, comprising: at least one storage medium configured to store at least one bitstream according to the 13th aspect or any implementation thereof, or a bitstream generated by a method that provides the first aspect or any implementation thereof; and a video streaming device configured to retrieve a bitstream from one of the at least one storage medium and send the bitstream to a terminal device, wherein the video streaming device comprises a content server or a content distribution server.

[0089] According to one implementation, the system further comprises a) one or more processors configured to perform an encryption process on at least one bitstream to obtain at least one encrypted bitstream, with at least one storage medium configured to store the encrypted bitstream, or b) one or more processors configured to convert a bitstream from at least one encrypted bitstream of a first format to a bitstream of a second format, with at least one storage medium configured to store the bitstream of the second format.

[0090] According to one implementation, the system is A receiver configured to receive a first operation request, A transmitter configured to send a target bitstream to a terminal device and The system further comprises one or more processors configured to determine a target bitstream in at least one storage medium in response to a first operation request.

[0091] In one implementation, one or more processors are further configured to encapsulate a bitstream and obtain a transport stream in a third format, and a transmitter is further configured to send the transport stream in the third format to a terminal device for display or to send the transport stream in the third format to a storage space for storage.

[0092] Where deemed appropriate, the fourteenth embodiment of the system can be combined with each other.

[0093] Embodiments of this disclosure will be described in more detail below with reference to the attached figures and drawings. [Brief explanation of the drawing]

[0094] [Figure 1] This is a schematic diagram showing the channels processed by the layers of a neural network. [Figure 2] This is a schematic diagram illustrating an autoencoder-type neural network. [Figure 3A] This is a schematic diagram showing an exemplary network architecture for the encoder and decoder sides, including a hyperplier model. [Figure 3B] This is a schematic diagram showing a typical network architecture for the encoder side, including a hyperplier model. [Figure 3C] This is a schematic diagram showing a general network architecture for the decoder side, including a hyperplier model. [Figure 4] This is a schematic diagram showing an exemplary network architecture for the encoder and decoder sides, including a hyperplier model. [Figure 5]This is a block diagram showing the structure of a cloud-based solution for machine-based tasks such as machine vision tasks. [Figure 6A] A block diagram showing a neural network-based end-to-end video compression framework. [Figure 6B] This block diagram shows some illustrative details of the application of neural networks for motion field compression. [Figure 6C] This block diagram shows some illustrative details of the application of neural networks for motion compensation. [Figure 7] This is a schematic diagram showing a general method for entropy coders. [Figure 8] This is a schematic diagram illustrating a common method of conventional entropy coding in autoencoder-based coders. [Figure 9] This is a schematic diagram illustrating a typical configuration of an autoencoder-based coder having an entropy coder and a gain unit. [Figure 10] This figure shows the signaled values ​​and decoder operation of a conventional method. [Figure 11] This figure shows the signaled value and decoder operation as proposed in this application. [Figure 12] This is a schematic diagram showing the relationship between the first coding parameter and the gain vector. [Figure 13] This is a flowchart of an exemplary method for encoding an image, as proposed. [Figure 14] This is a flowchart of an exemplary method for decoding an image, as proposed. [Figure 15] This block diagram shows an example of a video encoding device configured to implement an embodiment of the encoding described herein. [Figure 16] A block diagram shows an example of a video decoding device configured to carry out an embodiment of the decoding of the present disclosure. [Figure 17] This figure shows the bitstream structure. [Figure 18] This figure shows the encoding and decoding process using the gain vector scheme. [Figure 19] This is a schematic diagram illustrating how a multicore encoder encodes the channels of input data into substreams and then concatenates these substreams to form a bitstream. [Figure 20] This is a schematic diagram showing an exemplary encoder configured to implement the technique of the present application. [Figure 21] This is a schematic diagram showing an example of a decoder configured to implement the technique of this application. [Figure 22] This block diagram shows an example of a video coding system configured to implement embodiments of the present disclosure. [Figure 23] This block diagram shows another example of a video coding system configured to implement embodiments of the present disclosure. [Figure 24] This block diagram shows another example of a video coding system configured to implement embodiments of the present disclosure. [Figure 25] This is a block diagram showing an example of an encoding or decoding device. [Figure 26] This is a schematic diagram showing an exemplary structure of a content supply system 3100 that realizes a content distribution service. [Figure 27] This is a schematic diagram illustrating an example structure of a terminal service. [Figure 28] This is a block diagram showing another example of an encoding or decoding device. [Figure 29] This is a block diagram showing another example of an encoding or decoding device. [Figure 30] Computer vision task - classification block. [Figure 31] Computer vision task - Super resolution block. [Figure 32] Computer vision task - noise reduction block. [Modes for carrying out the invention]

[0095] Similar reference numbers and indications in different drawings may refer to similar elements.

[0096] In the following description, the accompanying drawings, which form part of this disclosure, are referenced, for example, to illustrate specific aspects of the embodiments of this disclosure, or specific aspects in which the embodiments of this disclosure may be used. It is understood that embodiments of this disclosure may be used in other embodiments, comprising structural or logical modifications not shown in the drawings. Accordingly, the following detailed description should not be understood as restrictive, and the scope of this disclosure is defined by the accompanying claims.

[0097] For example, disclosures relating to a method described may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, if one or more specific steps of a method are described, the corresponding device may include one or more units for performing the steps of the one or more methods described, e.g., functional units (e.g., one unit performing one or more steps, or multiple units each performing one or more of the steps), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, if a particular device is described based on one or more units, e.g., functional units, the corresponding method may include one step for performing the function of one or more units (e.g., one step performing the function of one or more units, or multiple steps each performing one or more of the functions of the units), even if such one or more steps are not explicitly described or illustrated in the drawings. Furthermore, unless otherwise specifically stated, features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0098] The following provides an overview of the technical terms used and some of the frameworks in which embodiments of this disclosure may be employed.

[0099] Artificial neural networks Artificial neural networks (ANNs), or connectionist systems, are computing systems vaguely inspired by the biological neural networks that make up animal brains. Such systems "learn" to perform tasks by examining examples, rather than being programmed with task-specific rules. For example, in image recognition, such a system might learn to identify images containing cats by analyzing exemplary images that have been manually labeled as "cat" or "not a cat," and then using the results to identify cats in other images. Such a system does this without any prior knowledge of cats, such as that cats have fur, tails, whiskers, and cat-like faces. Instead, such systems automatically generate the characteristics to identify from the examples they process.

[0100] ANNs are based on a collection of connected units or nodes called artificial neurons, which roughly model neurons in the biological brain. Each connection, like a synapse in the biological brain, can transmit signals to other neurons. The artificial neurons that receive the signals can then process them and signal the neurons connected to those artificial neurons.

[0101] In ANN implementations, the "signals" at connections are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of its inputs. These connections are called edges. Neurons and edges typically have weights that are adjusted as learning progresses. The weights increase or decrease the strength of the signal at the connection. Neurons may have thresholds such that they only send a signal if the aggregated signal exceeds a threshold. Typically, neurons are aggregated to form layers. Various layers may perform different transformations on their inputs. Signals travel from the first layer (input layer) to the last layer (output layer), possibly traversing multiple layers along the way.

[0102] The original goal of ANN methods was to solve problems in the same way that the human brain would. Over time, attention has shifted to performing specific tasks, leading to a deviation from biology. ANNs are used in a wide range of tasks, including computer vision, speech recognition, machine translation, social network filtering, playboard and video games, and medical diagnosis, as well as in activities traditionally considered reserved for humans, such as painting.

[0103] The name "Convolutional Neural Network" (CNN) indicates that the network employs a mathematical operation called convolution. Convolution is a specialized type of linear operation. A convolutional network is a neural network that uses convolution instead of general matrix multiplication in at least one of its layers.

[0104] Figure 1 schematically illustrates the general concept of processing by neural networks such as CNNs. A convolutional neural network consists of an input layer, an output layer, and several hidden layers. The input layer is the layer to which the input (such as a portion of an input image as shown in Figure 1) is provided for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve using multiplication or other dot products. The result of the layers is one or more feature maps (indicated by empty solid rectangles), sometimes called channels. There may be resampling (such as subsampling) that involves some or all of the layers. As a result, the feature maps may become smaller, as shown in Figure 1. Note that convolutions with a certain stride can also reduce the size (by resampling the input feature maps). The activation function in a CNN is usually a ReLU (Normalized Linear Unit) layer or leaky ReLU, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, which are called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. This layer is colloquially called a convolution, but this is merely a convention. Mathematically, it is technically a sliding dot product or cross-correlation. This has significance to the index in the matrix in that it affects how the weights are determined at a particular index point.

[0105] When programming a CNN for image processing, the input is a tensor with shape (number of images) × (image width) × (image height) × (image depth), as shown in Figure 1. It should be known that the image depth can be composed of the image's channels. After passing through the convolutional layer, the image is abstracted into a feature map with shape (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolutional layer in the neural network should have the following attributes: a convolutional kernel defined by width and height (hyperparameters); the number of input and output channels (hyperparameters); and the depth of the convolutional filter (input channels) should be equal to the number of channels (depth) in the input feature map.

[0106] In the past, conventional multilayer perceptron (MLP) models have been used for image recognition. However, due to their fully connected nature, they struggle with high dimensionality and did not scale well for higher resolution images. A 1000x1000 pixel image using RGB color channels has 3 million weights, which is too many to efficiently handle at a fully connected scale. Furthermore, such network architectures do not take into account the spatial structure of the data, treating distant input pixels as if they were close to each other. This ignores locality of reference in image data, both computationally and semantically. Therefore, the fully connected nature of neurons is wasteful for purposes like image recognition, which are governed by spatially local input patterns.

[0107] Convolutional neural networks (CNNs) are a biologically inspired variant of multilayer perceptrons, specifically designed to emulate the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by leveraging the strong, spatially localized correlations present in natural images. Convolutional layers are the core building blocks of CNNs. The layer parameters consist of a set of learnable filters (kernels, as described above) that have small receptive fields but extend throughout the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, calculating the dot product between the filter's entry and the input, and generating a two-dimensional activation map for that filter. As a result, the network learns filters to activate when it detects certain types of features at certain spatial locations in the input.

[0108] Stacking activation maps for all filters along the depth dimension forms the complete output volume of the convolutional layer. Every entry in the output volume can also be interpreted as the output of a neuron that shares parameters with the neuron in the same activation map, as well as looking at a small region in the input. A feature map or activation map is the output activation for a given filter. Feature map and activation have the same meaning. In some papers, it is called an activation map because it is a mapping corresponding to the activations of different parts of an image, and also a feature map because it is a mapping of where certain types of features can be found in the image. A large activation means that several features have been found.

[0109] Another important concept in CNNs is pooling, which takes the form of nonlinear downsampling. There are several nonlinear functions for performing pooling, the most common of which is max pooling. Max pooling divides the input image into sets of non-overlapping rectangles and outputs the maximum value for each such sub-region.

[0110] Intuitively, the precise location of a feature is less important than its rough location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers work to gradually reduce the spatial size of representations in order to reduce the number of parameters, the memory footprint, and the amount of computation in the network, and therefore also to control overfitting. It is customary to periodically insert pooling layers between consecutive convolutional layers in a CNN architecture. Pooling operations result in another form of translational invariance.

[0111] A pooling layer operates independently on every depth slice of the input, spatially resizing it. The most common form is a pooling layer with a 2x2 filter applied with a stride of 2 to every depth slice in the input, in units of 2 along both width and height, discarding 75% of the activations. In this case, every max operation spans four numbers. The depth dimensions remain unchanged. In addition to max pooling, pooling units can use other functions such as mean pooling or L2 norm pooling. Mean pooling has historically been frequently used, but recently it has fallen out of favor compared to max pooling, which actually performs better more often. Due to the aggressive reduction of representation size, there is a recent trend toward using smaller filters or discarding pooling layers altogether. "Region of Interest" pooling (also called ROI pooling) is a variation of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is a key component of convolutional neural networks for object detection based on fast R-CNN architectures.

[0112] The aforementioned ReLU is an abbreviation for Normalized Linear Unit, which applies a non-saturating activation function. ReLU effectively removes negative values ​​from the activation map by setting them to 0. This enhances the nonlinear properties of the decision function and the network as a whole without affecting the receptive field of the convolutional layer. Other functions, such as the saturating hyperbolic tangent function and the sigmoid function, are also used to enhance nonlinearity. ReLU is often preferred over other functions because it trains the neural network multiple times faster without a significant penalty to generalization accuracy.

[0113] Leaked Normalized Linear Unit (LEAL) activation function, or Leaked ReLU, is a type of activation function based on ReLU, but instead of a flat gradient, it has a small gradient for negative values. The gradient coefficients are determined before training, i.e., they are not learned during training. This type of activation function is common in tasks where it struggles with sparse gradients, such as training generative adversarial networks. Leaked ReLU applies a unit-by-unit function. LeakyReLU(x)=max(0,x)+negative_slope*min(0,x), or

number

[0114] Of those, the parameters are, negative_slope - Controls the angle of the negative slope. Default: 1e-2. inplace - Allows operations to be performed at a fixed location at will. Default: false.

[0115] After several convolutional and max-pooling layers, high-level inference in the neural network is performed via fully connected layers. Neurons in the fully connected layers have connections to all activations in the previous layers, as seen in typical (non-convolutional) artificial neural networks. These activations can thus be computed as affine transformations involving matrix multiplication followed by bias offsets (vector addition of learned or fixed bias terms).

[0116] The "loss layer" (which includes the calculation of the loss function) specifies how training penalizes the deviation between the predicted (output) label and the true label, and is usually the final layer of a neural network. Various loss functions appropriate for different tasks may be used. A softmax loss is used to predict a single class out of K mutually exclusive classes. A sigmoid cross-entropy loss is used to predict K independent probability values ​​in [0,1]. A Euclidean loss is used to regress to real-valued labels.

[0117] In summary, Figure 1 shows the data flow in a typical convolutional neural network. First, the input image is passed through a convolutional layer and abstracted into a feature map with several channels, corresponding to some of the filters in the set of learnable filters in that layer. Next, the feature map is subsampled, for example, using a pooling layer, which reduces the dimensionality of each channel in the feature map. Then the data comes to another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input and output channels are hyperparameters of the layer. To establish the connectivity of the network, these parameters must be synchronized between the two connected layers, so that the number of input channels for the current layer should be equal to the number of output channels for the previous layer. For the first layer processing the input data, e.g., an image, the number of input channels is usually equal to the number of channels in the data representation, e.g., 3 channels for an RGB or YUV representation of an image or video, or 1 channel for a grayscale image or video representation. Channels obtained by one or more convolutional layers (and possibly resampling layers) may be passed to the output layer. In some implementations, such an output layer may be a convolution or a resampling layer. In an exemplary and non-restrictive implementation, the output layer is a fully connected layer.

[0118] Autoencoders and unsupervised learning An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. A schematic diagram of it is shown in Figure 2. The autoencoder includes an encoder side 210 with input x which is input into the input layer of the encoder subnetwork 220, and a decoder side 250 with output x' which is output from the decoder subnetwork 260. The purpose of the autoencoder is to learn representations (encodes) 230 for a set of data x, usually for dimensionality reduction, by training the networks 220, 260 to ignore signal "noise". Along with the reduction (encoder) side subnetwork 220, the reconstruction (decoder) side subnetwork 260 is learned, where the autoencoder attempts to generate a representation x' from the reduced encoding 230 that is as close as possible to its original input x, hence the name. In the simplest example, given one hidden layer, the encoder stage of the autoencoder takes input x and maps it to h. h = σ(Wx + b)

[0119] This image h is usually called code 230, latent variable, or latent representation. Here, σ is an element-level activation function, such as a sigmoid function or normalized linear unit. W is the weight matrix and b is the bias vector. The weights and biases are usually initialized randomly and then updated iteratively during training through backpropagation. After that, the decoder stage of the autoencoder maps h to a reconstructed x' of the same shape as x, x'=σ'(W'h'+b') However, σ', W', and b' for the decoder may be independent of the corresponding σ, W, and b for the encoder.

[0120] Variational autoencoder models make strong hypotheses about the distribution of latent variables. They use variational methods for latent representation learning, which bring an additional loss component and specific estimators to the training algorithm called Stochastic Gradient Variational Bayes (SGVB) estimators. θ It is generated by (x|h), and the encoder is the posterior distribution p θ Approximation q to (h|x) φ Assume that (h|x) is being trained, where φ and θ are the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vector of a VAE usually fits the probability distribution of the training data much better than that of a standard autoencoder. The goal of a VAE has the following form:

number

[0121] Here, D KL This represents the Kullback-Leibler information. The pryor through latent variables is typically a centralized isotropic multivariate Gaussian p. θ (h) is set to N(0,I). Typically, the shapes of the variational and likelihood distributions are chosen such that they are factorized Gaussians. q φ (h|x)=N(ρ(x),ω 2 (x)I) p φ (x|h)=N(μ(h),σ 2 (h)I) However, ρ(x) and ω 2 (x) is the encoder output, and μ(h) and σ 2 (h) is the decoder output.

[0122] Recent advances in the field of artificial neural networks, and particularly in convolutional neural networks, have enabled researchers to apply neural network-based techniques to the tasks of image and video compression. For example, end-to-end optimized image compression using networks based on variational autoencoders has been proposed.

[0123] Therefore, data compression is considered a fundamental and well-considered problem in engineering, commonly formulated with the goal of designing code for a given discrete data ensemble with minimal entropy. Its solution relies heavily on knowledge of the probabilistic structure of the data, and thus the task is closely related to probabilistic source modeling. However, since all practical code must have a finite entropy, continuous data (such as a vector of image pixel intensity) must be quantized into a finite set of discrete values, which introduces errors.

[0124] In this context, known as the irreversible compression problem, two competing costs must be considered: the entropy (rate) of the discretized representation and the error (distortion) resulting from quantization. Various compression applications, such as data storage or data transmission over channels with limited capacity, require different rate-distortion tradeoffs.

[0125] Simultaneous optimization of rate and strain is difficult. The general challenge of optimal quantization in high-dimensional spaces without further constraints is also cumbersome. For this reason, most existing image compression methods operate by linearly transforming the data vector into a suitable continuous-value representation, independently quantizing its elements, and then encoding the resulting discrete representation using a reversible entropy code. This method is called transformative coding due to the central role of the transformation.

[0126] For example, JPEG uses a discrete cosine transform for blocks of pixels, while JPEG2000 uses a multiscale orthogonal wavelet decomposition. In general, the three components of a transform coding method—namely, the transform, the quantizer, and the entropy code—are optimized separately (often through manual parameter tuning). Modern video compression standards such as HEVC, VVC, and EVC also use transformed representations to code residual signals after prediction. For this purpose, several transforms are used, including discrete cosine and sine transforms (DCT, DST), as well as low-frequency non-separable transforms (LFNST), which are manually optimized.

[0127] Variational image compression The Variable Autoencoder (VAE) framework can be considered a nonlinear transformation coding model. The transformation process can be divided into four main parts, as illustrated in Figure 3A, which shows the VAE framework.

[0128] The transformation process can be divided into four main parts. Figure 3A illustrates the VAE framework. In Figure 3A, the encoder 101 maps the input image x to a latent representation (indicated by y) via the function y=f(x). This latent representation may also be referred to below as part of the “latent space” or a point within the “latent space”. The function f() is a transformation function that transforms the input signal x into a more compressible representation y. The quantizer 102 then transforms the latent representation y into

number

number

number

[0129] The latent space can be understood as a compressed representation of data where similar data points are closer to each other within the latent space. The latent space is useful for learning data features and for finding a simpler representation of the data for analysis. Hyperplier 3 quantized latent representation T,

number

number

number

number

number

number

number

[0130] In Figure 3A, the component AE105 is a quantized latent representation

number

number

number

number

[0131] Arithmetic decoding (AD) 106 is a process that reverses the binarization process, converting the binary digits back to sample values. Arithmetic decoding is performed by the arithmetic decoding module 106.

[0132] Please note that this disclosure is not limited to this particular framework. Furthermore, this disclosure is not limited to image or video compression and may also be applicable to object detection, image generation, and recognition systems.

[0133] Figure 3A shows two interconnected subnetworks. In this context, a subnetwork is a logical division between parts of the entire network. For example, in Figure 3A, modules 101, 102, 104, 105, and 106 are called the "encoder / decoder" subnetwork. The "encoder / decoder" subnetwork is responsible for encoding (generating) and decoding (parsing) the first bitstream, "bitstream 1". The second network in Figure 3A, consisting of modules 103, 108, 109, 110, and 107, is called the "hyperencoder / decoder" subnetwork. The second subnetwork is responsible for generating the second bitstream, "bitstream 2". The two subnetworks have different purposes.

[0134] The first subnetwork is responsible for the following: • Transformation of the input image x to its latent representation y (which is easier to compress x) (101), • Quantizing latent expression y

number

number

number

[0135] The purpose of the second subnetwork is to obtain statistical properties of the samples of "bitstream 1" (e.g., mean, variance, and inter-sample correlation of bitstream 1) so that the compression of bitstream 1 by the first subnetwork becomes more efficient. The second subnetwork generates a second bitstream, "bitstream 2," which contains the aforementioned information (e.g., mean, variance, and inter-sample correlation of bitstream 1).

[0136] The second network is a quantized latent representation of side information z.

number

number

number

number

number

number

number

number

number

number

number

number

number

[0137] Figure 3A shows an example of a VAE (Variational Autoencoder), the details of which may differ in different implementations. For example, in certain implementations, additional components may exist to more efficiently obtain the statistical characteristics of the samples in bitstream 1. In such an implementation, there may be a context modeler that targets the extraction of cross-correlation information of bitstream 1. The statistical information provided by the second subnetwork may be used by the AE (Arithmetic Encoder) 105 and AD (Arithmetic Decoder) 106 components.

[0138] Figure 3A shows an encoder and decoder in a single drawing. As will be apparent to those skilled in the art, encoders and decoders may be separate devices and are very often incorporated into such a device.

[0139] Separately, Figure 3B shows the encoder components of the VAE framework, and Figure 3C shows the decoder components of the VAE framework. As input, the encoder receives a picture according to some embodiments. The input picture may include one or more channels, such as a color channel or other types of channels, for example, a depth channel or a motion information channel. The outputs of the encoder (as shown in Figure 3B) are bitstream 1 and bitstream 2. Bitstream 1 is the output of the encoder's first subnetwork, and bitstream 2 is the output of the encoder's second subnetwork.

[0140] Similarly, in Figure 3C, two bitstreams, namely bitstream 1 and bitstream 2, are received as input and the resulting image is reconstructed (decoded).

number

[0141] Specifically, as can be seen in Figure 3B, the encoder includes an encoder 121 that converts the input x into a signal y, and the signal y is then provided to the quantizer 322. The quantizer 122 provides information to the arithmetic coding module 125 and the hyper-encoder 123. The hyper-encoder 123 provides the bitstream 2, which has already been described above, to the hyper-decoder 147 that provides information to the arithmetic coding module 105 (125).

[0142] The output of the arithmetic coding module is the bitstream 1. The bitstream 1 and the bitstream 2 are the outputs of the signal encoding that are then provided (transmitted) to the decoding process. The unit 101 (121) is called an "encoder", but it is also possible to call the complete sub-network described in Figure 3B an "encoder". The encoding process generally means a unit (module) that converts the input into an encoded (e.g., compressed) output. Since the unit 121 performs the conversion of the input x into y, which is a compressed version of x, it can actually be regarded as the core of the overall sub-network, as can be seen from Figure 3B. The compression in the encoder 121 may be achieved, for example, by applying a neural network or generally any processing network having one or more layers. In such a network, the compression may be performed by a cascaded process that includes downsampling to reduce the size and / or number of input channels. Therefore, the encoder may sometimes be called, for example, an encoder based on a neural network (NN).

[0143] The remaining parts in the diagram (quantization unit, hyperencoder, hyperdecoder, arithmetic encoder / decoder) are all responsible for either improving the efficiency of the encoding process or transforming the compressed output y into a sequence of bits (bitstream). Quantization may be performed to further compress the output of the NN encoder 121 by lossy compression. The AE125, combined with the hyperencoder 123 and hyperdecoder 127 used to constitute the AE125, may perform binarization, which can further compress the quantized signal by lossless compression. Thus, the overall subnetwork in Figure 3B can also be called the "encoder".

[0144] Most deep learning (DL)-based image / video compression systems reduce the dimensionality of a signal before converting it to binary digits (bits). In the VAE framework, for example, an encoder, which is a nonlinear transformation, maps an input image x to y, where y has a width and height smaller than x. Since y has a smaller width and height, and therefore a smaller size, the dimensionality (size) of the signal is reduced, and thus it is easier to compress the signal y. It should be noted that, in general, encoders do not necessarily need to reduce the size in both (or generally all) dimensions. Rather, some exemplary implementations may provide an encoder that reduces the size in only one dimension (or generally a subset thereof).

[0145] At the 4th International Conference on Learning Representations 2016, in arXiv e-prints, J. Balle, L. Valero Laparra, and E. P. Simoncelli (2015), "Density Modeling of Images Using a Generalized Normalization Transformation" (hereinafter referred to as "Balle"), the authors proposed a framework for end-to-end optimization of an image compression model based on non-linear transformation. The authors optimize for the mean squared error (MSE), but use a more flexible transformation constructed from a cascade of linear convolution and non-linearity. Specifically, the authors use a generalized divisive normalization (GDN) coupled non-linearity inspired by the model of neurons in the biological visual system, and prove that it is effective when Gaussianizing the image density. This cascaded transformation is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), and uniform scalar quantization effectively implements parametric vector quantization for the original image space. An approximate parametric non-linear inverse transformation is used to reconstruct the compressed image from these quantized values.

[0146] Such an example of the VAE framework is shown in Figure 4 and utilizes six downsampling layers marked with 401 - 406. The network architecture includes a hyperprior model. The left side (g a , g s ) shows the image autoencoder architecture, and the right side (h a , h s ) corresponds to the autoencoder that implements the hyperprior. The factored prior model uses the same architecture for analysis, and the synthesis transforms g a and g s . Q represents quantization, and AE and AD represent the arithmetic encoder and arithmetic decoder, respectively. The encoder maps the input image x to g aThis is passed to produce a spatially varying response y (latent representation) with a standard deviation. Encoding g a This includes multiple convolutional layers with subsampling, and Generalized Partition Normalization (GDN) as the activation function.

[0147] The response is h a It is supplied to summarize the distribution of the standard deviation at z. z is then quantized, compressed, and transmitted as side information. The encoder then processes the quantized vector

number

number

number

number

number

number

number

[0148] Layers that include downsampling are indicated by a downward arrow in the layer description. The layer description "Conv N,k1,2↓" means that the layer is a convolutional layer with N channels and a convolutional kernel size of k1 × k1. For example, k1 may be equal to 5 and k2 may be equal to 3. As mentioned, 2↓ means that downsampling with a coefficient of 2 is performed in this layer. Downsampling with a coefficient of 2 results in one of the dimensions of the input signal being reduced by half in the output. In Figure 4, 2↓ shows that both the width and height of the input image are reduced by a coefficient of 2. Since there are six downsampling layers, if the width and height of the input image 414 (also indicated by x) are given by w and h, the output signal z^413 will have a width and height equal to w / 64 and h / 64, respectively. The modules indicated by AE and AD are the arithmetic encoder and arithmetic decoder, which are described with reference to Figures 3A to 3C. Arithmetic encoders and decoders are specific implementations of entropy coding. AE and AD can be replaced by other means of entropy coding. In information theory, entropy coding is a lossless data compression method used to transform the values ​​of symbols into a binary representation, which is a reversible process. Also, "Q" in the figure corresponds to the quantization operation, which is also mentioned above with respect to Figure 4 and further explained above in the section "Quantization". Furthermore, the quantization operation and the corresponding quantization unit as part of component 413 or 415 are not necessarily present and / or can be replaced by other units.

[0149] Figure 4 also illustrates a decoder with upsampling layers 407-412. A further layer 420, implemented as a convolutional layer but without upsampling the received input, is placed between the upsampling layers 411 and 410 in the order of input processing. A corresponding convolutional layer 430 is also shown for the decoder. Such layers that do not change the size of the input but change certain characteristics may be included in the NN to perform operations on the input. However, the inclusion of such layers is not mandatory.

[0150] When viewed in the processing order of bitstream 2 through the decoder, the upsampling layers are passed in the reverse order, i.e., from upsampling layer 412 to upsampling layer 407. Each upsampling layer is shown here to perform upsampling with an upsampling rate of 2, as indicated by ↑. Of course, it is not necessarily true that all upsampling layers have the same upsampling rate, and other upsampling rates such as 3, 4, 8, etc., may also be used. Layers 407-412 are implemented as convolutional layers (conv). In detail, they may be intended to perform an operation on the input that is the inverse of the encoder input, so the upsampling layers may apply an inverse convolution operation to the received input, and as a result their size increases by a coefficient corresponding to the upsampling rate. However, this disclosure is not limited in general to inverse convolution, and upsampling may be performed in any other way, such as by bilinear interpolation between two adjacent samples or by nearest neighbor sample copying, etc.

[0151] In the first subnetwork, several convolutional layers (401-403) are followed by generalized partition normalization (GDN) on the encoder side and inverse GDN (IGDN) on the decoder side. In the second subnetwork, the activation function applied is ReLU. Note that this disclosure is not limited to such implementations, and other activation functions may be used in general instead of GDN or ReLU.

[0152] Cloud solutions for machine tasks Machine video coding (VCM) is currently another popular computer science trend. The main idea behind this technique is to transmit coded representations of image or video information targeted for further processing by computer vision (CV) algorithms, such as object segmentation, object detection, and object recognition. In contrast to traditional image and video coding, which targeted human perception, the quality characteristic is not the quality being reconstructed, but rather the performance on computer vision tasks, such as object detection accuracy. This is illustrated in Figure 5.

[0153] Video coding for machines, also known as collaborative intelligence, is a relatively new paradigm for the efficient deployment of deep neural networks across mobile cloud infrastructure. By dividing the network between the mobile side 510 and the cloud side 590 (e.g., cloud servers), it is possible to distribute the computational workload, resulting in minimized overall system energy and / or latency. In general, collaborative intelligence is a paradigm in which the processing of a neural network is distributed among two or more different computing nodes, e.g., devices, but generally any functionally defined node. Here, the term “node” does not refer to the neural network nodes described above. Rather, a (computational) node here refers to a separate device / module (physically or at least logically) that implements a part of the neural network. Such devices may be different servers, different end-user devices, a mixture of servers and / or user devices and / or clouds and / or processors, etc. In other words, computing nodes may be considered nodes that belong to the same neural network and communicate with each other to transmit data coded within / for the neural network. For example, to enable the execution of complex calculations, one or more layers may run on a first device (such as a device in mobile-side 510), and one or more layers may run on another device (such as a cloud server in cloud-side 590). However, the distribution may also be finer-grained, with a single layer running on multiple devices. In this disclosure, the term “multiple” refers to two or more. In some existing solutions, a portion of the neural network functionality runs on a device (such as a user device or edge device) or multiple such devices, and the output (feature map) is then passed to the cloud. The cloud is a collection of processing or computing systems located outside of the device that powers a portion of the neural network. The concept of collaborative intelligence has also been extended to model training.In this case, data flows in both directions, from the cloud to the mobile during backpropagation in training and from the mobile to the cloud (shown in FIG. 5) during the forward pass in training and inference.

[0154] Some work has presented semantic image compression by encoding deep features and then reconstructing the input images from them. Compression based on uniform quantization is shown, followed by context-based adaptive arithmetic coding (CABAC) from H.264. In some scenarios, instead of sending the compressed natural image data to the cloud and performing object detection using the reconstructed images, it may be more efficient to send the output of the hidden layer (deep feature map) 550 from the mobile part 510 to the cloud 590. Compressing the data (features) generated by the mobile side 510 can be advantageous in this way, and the mobile side 510 may include a quantization layer 520 for this purpose. Correspondingly, the cloud side 590 may include an inverse quantization layer 560. Efficient compression of feature maps is beneficial for reconstruction for both image and video compression and for both human perception and machine vision. Entropy coding methods, such as arithmetic coding, are a common approach for compressing deep features (i.e., feature maps).

[0155] Today, video content contributes more than 80% of internet traffic, and that percentage is expected to increase further. Therefore, it is crucial to build efficient video compression systems that produce higher-quality frames within a given bandwidth budget. In addition, most video-related computer vision tasks, such as video object detection or video object tracking, are sensitive to the quality of compressed video, and efficient video compression can benefit other computer vision tasks. Meanwhile, techniques in video compression are also useful for motion recognition and model compression. However, for the past several decades, video compression algorithms have relied on handcrafted modules, such as block-based motion estimation and discrete cosine transform (DCT), to reduce redundancy in video sequences, as mentioned above. While each module may be well-designed, the overall compression system is not end-to-end optimized. It is desirable to further improve video compression performance by collaboratively optimizing the overall compression system.

[0156] End-to-end image or video compression DNN-based image compression methods can leverage large-scale end-to-end training and highly nonlinear transformations not used in conventional methods. However, directly applying these techniques to build an end-to-end learning system for video compression is not trivial. Firstly, learning how to generate and compress motion information tailored for video compression remains an unresolved problem. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences.

[0157] A simple solution is to use a learning-based optical flow to represent motion information. However, current learning-based optical flow methods aim to generate the most accurate flow field possible. A precise optical flow is often not optimal for a particular video task. In addition, the data volume of optical flow is significantly larger compared to motion information in conventional compression systems, and directly applying existing compression methods to compress optical flow values ​​significantly increases the number of bits required to store motion information. Secondly, it is unclear how to build a DNN-based video compression system by minimizing rate-distortion-based goals for both residuals and motion information. Rate-distortion optimization (RDO) aims to achieve improved quality of reconstructed frames (i.e., reduced distortion) given the number of bits (or bitrate) for compression. RDO is important for video compression performance. To leverage the power of end-to-end training for learning-based compression systems, an RDO policy is needed to optimize the overall system.

[0158] In "DVC: An End-to-end Deep Video Compression Framework," Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11006-11015, the authors propose an end-to-end deep video compression (DVC) model that collaboratively learns motion estimation, motion compression, and residual coding.

[0159] Such an encoder is shown in Figure 6A. In detail, Figure 6A shows the overall structure of the end-to-end trainable video compression framework. To compress motion information, optical flow v tConvert to a corresponding expression suitable for better compression m t To achieve this, a CNN was specified. In detail, an autoencoder-style network is used to compress the optical flow. The motion vector (MV) compression network is shown in Figure 6B. The network architecture is somewhat similar to ga / gs in Figure 4. Specifically, the optical flow v t This is fed into a series of convolution operations and nonlinear transformations, including GDN and IGDN. The number of output channels c for convolution (deconvolution) is 128 here exemplarily, except for the last deconvolution layer which is equal to 2 in this example. The kernel size is k, for example, k=3. Given an optical flow of size M×N×2, the MV encoder generates a motion representation m of size M / 16×N / 16×128. t This generates the motion representation, which is then quantized (Q), entropy coded, and

number

number

number

[0160] Figure 6C shows the structure of the motion compensation section. Here, the reconstructed previous frame x t-1 Then, using the reconstructed motion information, the warping unit generates a distorted frame (usually with the help of an interpolation filter such as a bilinear interpolation filter). A separate CNN with three inputs then generates the predicted picture. The architecture of the motion-compensated CNN is also shown in Figure 6C.

[0161] The residual information between the original frame and the predicted frame is encoded by a residual encoder network. A highly nonlinear neural network is used to transform the residuals into their corresponding latent representation. Compared to the discrete cosine transform in conventional video compression systems, this method can better leverage the power of nonlinear transformations and achieve higher compression efficiency.

[0162] From the above overview, it can be understood that CNN-based architectures can be applied to both image and video compression, taking into account the different parts of the video framework, including motion estimation, motion compensation, and residual coding. Entropy coding is a common method used for data compression, widely adopted by the industry and equally applicable to feature map compression for either human perception or computer vision tasks.

[0163] Video coding for machines Machine video coding (VCM) is currently another popular computer science trend. The main idea behind this technique is to transmit coded representations of image or video information targeted for further processing by computer vision (CV) algorithms, such as object segmentation, object detection, and object recognition. In detail, Figure 30 shows an example of a classification task, Figure 31 shows an example of a super-resolution task, and Figure 32 shows an example of a denoising task. In contrast to conventional image and video coding that targets human perception, the quality characteristic is not the quality being reconstructed, but rather the performance on computer vision tasks, such as object detection accuracy.

[0164] Recent studies propose a new deployment paradigm called collaborative intelligence, in which deep models are partitioned between mobile and cloud. Extensive experiments under various hardware configurations and wireless connectivity modes have revealed that the optimal operating point in terms of energy consumption and / or computational latency involves partitioning the model, typically at a deep point in the network. Current common solutions, where the model resides entirely in the cloud or entirely on mobile, have been found to be (if any) rarely optimal. The concept of collaborative intelligence has also been extended to model training. In this case, data flows in both directions: from cloud to mobile during backpropagation in training, and from mobile to cloud during the forward pass in training and inference.

[0165] In the context of recent deep models for object detection, irreversible compression of deep feature data based on HEVC intracoding has been considered. The degradation of detection performance with increasing compression levels, and proposed compression-extension training to minimize this loss by generating models more robust to quantization noise in feature values, have attracted attention. However, this remains a suboptimal solution because the codecs employed are extremely complex and optimized for natural scene compression rather than deep feature compression.

[0166] The challenges of deep feature compression for collaborative intelligence are addressed by techniques for object detection tasks, using a common YOLOv2 network for examining the trade-off between compression efficiency and recognition accuracy. Here, the term "deep feature" is synonymous with "feature map." The term "deep" comes from the collaborative intelligence concept when the output feature maps of several hidden (deep) layers are captured and sent to the cloud for inference. This appears to be more efficient than sending compressed natural image data to the cloud and then performing object detection using the reconstructed images.

[0167] Efficient feature map compression is beneficial for image and video compression, as well as reconstruction for both human perception and machine vision. The shortcomings of state-of-the-art autoencoder-based compression methods, as discussed, are also relevant to machine vision tasks.

[0168] Network Training Loss function A loss function may include multiple components. For an image coding task, loss components related to reconstruction quality generally include L1 loss, L2 loss (or MSE loss), MS-SSIM loss, VGG loss, LPIPS loss, GAN loss, etc., and further include loss components related to bitstream size.

[0169] The L1 loss calculates the average error between the points from which the L1 loss value should be obtained. The L1 loss function can better evaluate the reconstruction quality of structured regions in an image.

[0170] Mean squared error (MSE) loss: A function for measuring the distance between two pieces of data. In this embodiment of the present application, the MSE loss is also called the L2 loss function. To obtain the L2 loss value, the mean of the squared errors between points is calculated. The MSE loss may also be used to calculate the PSNR. The L2 loss is also a pixel-level loss. The L2 loss function can also better evaluate the reconstruction quality of structured regions in an image. When the L2 loss function is used to optimize an image coding and decoding network, the optimized image coding and decoding network can achieve a higher PSNR.

[0171] Structural similarity index measure (SSIM): An objective criterion for evaluating image quality. A higher SSIM indicates better image quality. In this embodiment of the present application, the structural similarity between two images on a certain scale is calculated to obtain an SSIM loss value. The SSIM loss is a loss based on artificial features. Compared to the L1 and L2 loss functions, the SSIM loss function can evaluate image reconstruction quality more objectively, i.e., evaluate the structured and unstructured regions of an image in a more balanced manner. When the SSIM loss function is used to optimize an image coding and decoding network, the optimized image coding and decoding network can achieve a higher SSIM.

[0172] Multiscale Structural Similarity Index (multi-scale SSIM, MS-SSIM): An objective criterion for evaluating image quality. A higher SSIM indicates better image quality. Multi-layer low-pass filtering and downsampling are performed separately on two images to obtain image pairs at multiple scales. Contrast maps and structural information are extracted from the image pair at each scale, and the SSIM loss value at the corresponding scale is obtained based on the contrast maps and structural information. Luminance information is extracted from the image pair at the smallest scale, and the luminance loss value at the smallest scale is obtained based on the luminance information. The SSIM loss values ​​and luminance loss values ​​at multiple scales are then aggregated using a method for obtaining MS-SSIM loss values, for example, the aggregation method in equation (1).

number

[0173] In equation (1), the loss values ​​at all scales are aggregated using exponential weighting and multiplication. In this specification, x and y represent two images separately, l represents the loss value based on luminance information, c represents the loss value based on contrast map, and s represents the loss value based on structural information. The subscripts j=1,...,M represent M scales that correspond separately to a total of M downsamplings, with j=1 representing the largest scale and j=M representing the smallest scale. The superscripts α, β, and γ represent the exponential powers of the corresponding terms, respectively.

[0174] The MS-SSIM loss function and the SSIM loss function have similar good image evaluation effects. Compared to L1 loss and L2 loss, the MS-SSIM loss for optimization can improve the subjective experience of the human eye and satisfy objective evaluation metrics. When the MS-SSIM loss function is used to optimize image coding and decoding networks, the optimized image coding and decoding networks can achieve higher MS-SSIM.

[0175] Visual Geometry Group (VGG) Loss: VGG is the name of the group that designs CNN networks, and these are called VGG networks. The image loss value determined based on the VGG network is called the VGG loss value. The process of determining the VGG loss value is essentially as follows: the features of the original image before compression and the features of the decompressed reconstructed image (for example, feature maps obtained through convolution calculations in a certain layer) are extracted separately by using the VGG network, and then the distance between the features of the original image and the features of the reconstructed image at this scale is calculated to obtain the VGG loss value. This process is considered the process of determining the VGG loss value according to the VGG loss function. The VGG loss function focuses on improving the texture reconstruction quality.

[0176] Learned Perceptual Image Patch Similarity (LPIPS) Loss: An Enhanced VGG Loss. Multiscale properties are introduced into the process of determining the LPIPS loss value. The process of determining the LPIPS loss value is essentially as follows: Features of two images at multiple scales are extracted separately using a VGG network; the distance between the two image features at each scale is calculated to obtain multiple VGG loss values; and then a weighted sum is performed on the multiple VGG loss values ​​to obtain the LPIPS loss value. This process is considered a process of determining the LPIPS loss value according to the LPIPS loss function. Similar to the VGG loss function, the LPIPS loss function also focuses on improving the texture reconstruction quality.

[0177] Generative Adversarial Network Loss: Features from two images are extracted separately using a discriminator (also called a discriminator) contained within the GAN, and the distance between the features of the two images is calculated to obtain a Generative Adversarial Network Loss value. This process is considered the process of determining the GAN loss value according to the GAN loss function. The GAN loss function also focuses on improving the quality of texture reconstruction. GAN losses include at least one of the following: standard GAN loss, relative GAN loss, relative mean GAN loss, least squares GAN loss, etc.

[0178] Perceptual loss: Perceptual loss in a broad sense and perceptual loss in a narrow sense. In this embodiment of the present application, perceptual loss in a narrow sense is used as an example for illustrative purposes. VGG loss and LPIPS loss may be considered perceptual loss in a narrow sense. However, in another embodiment, loss calculated based on depth features extracted from an image may be considered perceptual loss in a broad sense. Perceptual loss in a broad sense may include perceptual loss in a narrow sense and may further include losses, such as the GAN loss described above. A perceptual loss function causes the reconstructed image to better satisfy the subjective experience of the human eye, but may result in lower PSNR and MS-SSIM.

[0179] Functional module Variable bitrate module An encoder can output a bitstream at different bitrates. Therefore, in some cases, the output of the encoding network is scaled (for example, each channel is multiplied by a corresponding scaling factor, also called the target gain value), and the input of the decoding network is descaled (for example, each channel is multiplied by the reciprocal of the corresponding scaling factor, also called the target inverse gain value). The scaling factor may be pre-set. Different quality levels or quantization parameters correspond to different target gain values. If the output of the encoding network is scaled to a smaller value, the bitstream size may be smaller; otherwise, the bitstream size may be larger.

[0180] Color format conversion RGB and YUV are common color spaces. Conversion between RGB and YUV may be performed according to formulas specified in standards such as CCIR601 and BT.709.

[0181] Separate structures for luma and chroma Some VAE-based codecs use the YUV color space as the input to the encoder and the output to the decoder. The Y component represents the lumens, and the UV component represents the chromens. The resolution of the UV component may be the same as or lower than the resolution of the Y component. Typical formats include YUV4:4:4, YUV4:2:2, and YUV4:2:0. The Y component is converted to a feature map F_Y through a network, and an entropy coding module generates a bitstream of the Y component based on the feature map F_Y. The UV component is converted to a feature map F_UV through another network, and an entropy coding module generates a bitstream of the UV component based on the feature map F_UV. Under this structure, the feature maps for the Y component and the UV component may be quantized independently so that bits are flexibly allocated to the lumens and chromens. For example, in the case of color-sensitive images, the feature map for the UV component may not need to be quantized as much, and the number of bitstream bits for the UV component may be larger to improve the reconstruction quality of the UV component and achieve a better visual effect.

[0182] In some other methods, the encoder concatenates the Y and UV components and then sends them to a UV component processing module (to convert the image information into a feature map). In addition, the decoder concatenates the reconstructed feature map of the Y component with the reconstructed feature map of the UV component and then sends them to a UV component processing module 2 (to convert the feature map into image information). In this method, correlation between the Y and UV components may be used to reduce the bitstream of the UV component.

[0183] In this specification, “parameters” are values ​​used in the computational processes of each layer forming a neural network, and may include, for example, weights used when input values ​​are applied to several computational representations. Here, parameters may be represented in matrix form. Parameters are values ​​set as a result of training and may be updated through separate training data as needed.

[0184] arithmetic encoding Entropy coding is typically employed as reversible coding. Arithmetic coding is a class of entropy coding that encodes a message as a binary real number within an interval (range) representing the message. In this specification, the term "message" refers to a sequence of symbols. Symbols are selected from a default alphabet of symbols. For example, an alphabet may consist of two values, 0 and 1. A message using such an alphabet is, in this case, a sequence of bits. Symbols (0 and 1) may occur in a message with different frequencies than each other. In other words, symbol probabilities may not be uniform. In fact, the less uniform the distribution, the greater the compression achievable by entropy coding, and more specifically by arithmetic coding. Arithmetic coding utilizes a known probability model that specifies a symbol probability for each symbol in the alphabet. The alphabet does not have to be binary. Rather, an alphabet may consist of, for example, M values ​​from 0 to M-1. In general, any alphabet of any size may be used. Typically, the alphabet is given by the value range of the coded data.

[0185] A modified form of the arithmetic coda improved for practical use is called a range coda, which uses a finite range of integers, for example, from 0 to 255, instead of the interval [0,1). This range is divided according to the probability of each alphabet symbol. If the remaining range becomes too small to represent all alphabet symbols according to their probabilities, the range may be renormalized.

[0186] One of the main types of entropy encoders assigns a unique code to each unique symbol that occurs in the input. These entropy encoders compress data by replacing each fixed-length input symbol with a corresponding variable-length output codeword. For data streams with certain specific entropy properties, simple static codes may be useful. These static codes include general-purpose codes (such as Elias gamma coding or Fibonacci coding) and Golomb codes (such as unary coding or Rice coding). For a general data stream, the codes may be constructed based on the following rule: the length of each codeword is approximately proportional to the negative logarithm of the probability of that codeword occurring. Therefore, the most common symbols use the shortest code. Based on the constructed code table, the coder compresses the data by replacing each fixed-length input symbol with a corresponding variable-length, unprefixed output codeword. One example of such coding is Huffman coding. The main problem with such coding is that at least one bit is required for each input symbol, even if the probability of that event is close to 1. As arithmetic coders have become faster, the asymmetric numeral system (ANS) family of entropy coding techniques has been invented. Such coders offer a combination of processing cost similar to Huffman coding and compression ratio for arithmetic coding.

[0187] An entropy coder encodes symbols of the input alphabet A, of size M, into symbols of the alphabet B, of size R, by using a quantity of output symbols inversely proportional to the probability of the coded symbols. Typically, symbols a from alphabet A... i The probability p i This refers to any sequence of symbols starting with the alphabet A, where the symbol a i This represents the probability of occurrence. In other words, probability pi This means that the received symbol y is a i This represents the probability of an event being equal to . The uneven probabilities of different symbols from the alphabet give the possibility of compression. If all symbols in the alphabet have the same probability pi = 1 / M, where M is the size of the alphabet A, then compression is impossible.

[0188] A typical configuration of an entropy coder is shown in Figure 7. In most cases, the output alphabet is {0,1}, the size of the output alphabet is usually equal to 2, the symbols of the output binary alphabet are called bits, and the sequence of bits corresponding to the sequence of coded symbols from the input alphabet is called a bitstream. As can be seen in Figure 7, the output symbols of the entropy encoder are called a bitstream and are the input to the entropy decoder. The output to the entropy decoder is the same alphabet as the input to the entropy encoder. The output to the entropy decoder is sometimes called the decoded symbols. Furthermore, the alphabet means a set of symbols, and the alphabet size means the number of symbols in the input alphabet. Here, the input symbols are shown as 0,1,2,...,M-1, and therefore there are a total of M different symbols in the input alphabet. It should be noted in this application that the alphabet size M always means the size of the input alphabet.

[0189] In autoencoder-based coding schemes, an entropy coder is used to compress latent space symbols. Distribution estimation can be performed in advance (pre-trained histogram) or using some extra information from the bitstream and / or information from adjacent latents. A common scheme of conventional entropy coding in an autoencoder-based coder is shown in Figure 8. An input signal x is transformed into a feature (or latent) tensor y, where "x" here means the input signal corresponding to the image data, and "y" here means the latent space tensor. The transformation process may here be called feature extraction, and the latent space tensor contains latent space elements, which are quantized and then fed into the entropy encoder as input. In some possible embodiments, the latent space elements are processed by a gain unit as shown in Figure 9, then quantized, and then fed into the entropy encoder as input. The tensor y consists of real numbers (not integers) (e.g., floating-point values), and the range of these numbers is not known in advance. The entropy coder can only function with a finite input alphabet, and therefore the tensor y is an integer tensor containing values ​​from 0 to M-1.

number

number

number

number

number

number

[0190]

number

[0191] The tensor is decoded, and the resulting tensor is reconstructed using the synthesis part of the autoencoder.

number

[0192] As shown in Figure 9, a gain unit may be added to the coding scheme to control the quantization error (distortion) and bitstream size (bitrate). It should be noted that using a higher bitrate may result in better data compression quality, while using a lower bitrate may result in lower data compression quality. The process of adjusting the compression system parameters to achieve a desired ratio between bitrate and quality (or to achieve a desired bitrate) is called rate control. The parameters used during rate control are called rate control parameters β. Rate control parameters β define the quantization steps for the latent space. A gain unit is used for rate control. The gain unit is the gain vector.

number

[0193] Table 1 shows the signaling of the rate control parameter β in the conventional technology, where the syntax element is coded using a uniform probability distribution uf(x). The minimum value of the distribution is 0, and its maximum value is x.

[0194] [Table 1]

[0195] As shown in Figure 10, in the conventional method, the model index (model_idx) and β int g is the signaled value in the bitstream, and on the decoder side, it is the target gain vector g (g relative to the lumens data). luma and g for chroma data chroma To calculate ), the following operations are performed.

number

[0196] As shown in Table 1, the maximum value for beta_luma is 65535, which means that 16 bits are used to signal beta_luma. Similarly, 16 bits are used to signal beta_chroma. If 32 bits are used to signal the rate control parameters for each image, that is a large number of bits and increases the transmission burden of the bitstream.

[0197] Furthermore, the decoder obtains a floating-point betaluma (beta_luma_float) and uses beta_luma_float to g luma When calculating this, the further calculations performed on the decoder side involve six floating-point operations, which, likewise, cannot be reproduced bit-exactly between different devices due to the use of floating-point arithmetic.

[0198] To solve the aforementioned problems, embodiments of this application propose a novel signaling scheme for rate control parameters, specifically, a first coding parameter (beta_displacement_luma or beta_displacement_y) is signaled, and the first coding parameter indicates a first displacement between a first target rate control parameter and a first reference rate control parameter. By signaling a first coding parameter that indicates a first displacement between a first target rate control parameter and a first reference rate control parameter, the number of bits used to signal the rate control parameters is reduced, for example, from 16*2=32 to 10*2=20. Furthermore, the decoder-side algorithm is simplified, and the number of operations for obtaining the target gain vector is reduced from 6 to 2 (or from 3 to 1 for luma / chroma data only).

[0199] Therefore, the number of calculations is reduced, and all calculations are performed using fixed-point arithmetic, which guarantees bit-exact behavior across different devices.

[0200] Table 2 shows the syntax table related to the signaling of the proposed rate control parameters, where the syntax elements are coded using a uniform probability distribution uf(x). The minimum value of the distribution is 0, and its maximum value is x.

[0201] [Table 2]

[0202] As shown in Table 2, a first coding parameter (beta_displacement_luma) is signaled indicating a first displacement (e.g., ratio or difference) between a first target rate control parameter (e.g., rate control parameter beta selected by the encoder for the first-order / luma component) and a first reference rate control parameter (e.g., rate control parameter beta used in model training for the first-order / luma component), and the first coding parameter, first target rate control parameter, and first reference rate control parameter are for coding the luma data of the image, and a flag (different_betas_for_luma_and_chroma_flag, or different_betas_for_y_and_uv_flag, or independent_beta_uv) is signaled indicating whether different rate control parameters are used to code the luma and chroma data of the image, and the flag (different_betas_for When _luma_and_chroma_flag, or different_betas_for_y_and_uv_flag, or independent_beta_uv) is equal to a first value, for example, 1, a second coding parameter (beta_displacement_chroma) is signaled indicating a second displacement (e.g., ratio or difference) between a second target rate control parameter (e.g., a rate control parameter beta selected by the encoder for the second-order / chroma component) and a second reference rate control parameter (e.g., a rate control parameter beta used in model training for the second-order / chroma component), the second coding parameter, the second target rate control parameter, and the second reference rate control parameter are for coding the chroma data of the image, and furthermore, a third coding parameter (model_idx) is signaled indicating the index of the trained model associated with the first and / or second reference rate control parameters.

[0203] In Table 2, parameters and flags are signaled within the image header (i.e., picture header), and in other implementations, parameters and flags may be signaled within other headers or parameter sets of the bitstream, such as the channel header, sequence parameter set, picture parameter set, etc. In one embodiment, the first coding parameter is specified as beta_displacement_luma or beta_displacement_y. In one embodiment, the second coding parameter is specified as beta_displacement_chroma or beta_displacement_uv.

[0204] In Table 2, the maximum value of beta_displacement_luma is 1023, and the maximum value of beta_displacement_chroma is also 1023, which means that only 10 bits are used to signal beta_displacement_luma and 10 bits are used to signal beta_displacement_chroma. In other implementations, N bits are used to signal beta_displacement_luma and M bits are used to signal beta_displacement_chroma, where N is equal to M or N is not equal to M, and N and M are positive integers. For example, N can be equal to 8, 10, 12, 16, etc. M can be equal to 6, 8, 10, 12, 16, etc. Next, a binary code is used with a uniform distribution; for example, 1 is signaled as 00000 00001, and 1023 is signaled as 11111 11111.

[0205] Optionally, the first coding parameter and / or the second coding parameter are represented by fixed-point values. By signaling the first and / or second coding parameters with fixed-point values, fixed-point arithmetic is used in both the encoder and the decoder, and thus a new architecture of the gain unit can be used in the arithmetic stability sensitive codec portion, such as the entropy portion. That is, the stability and versatility of this method are improved.

[0206] As shown in Figure 11, in the proposed signaling scheme for rate control parameters, at least the model index (model_idx) and the first coding parameter (beta_displacement_luma) are the values ​​to be signaled in the bitstream. On the decoder side, the target gain vector g (g relative to the luma data) luma and g for chroma data chroma To calculate ), the following operations are performed. 1. g luma =scaler_table_luma[model_idx]*beta_displacement_luma 2. g chroma =scaler_table_chroma[model_idx]*beta_displacement_chroma

[0207] Optionally, a flag (different_betas_for_luma_and_chroma_flag) is signaled to indicate whether different rate control parameters are used to code the luma and chroma data of an image. In this case, beta_chroma is only signaled when the flag indicates that different rate control parameters are used to code the luma and chroma data of an image.

[0208] Note that a value of 0 should not be allowed, and therefore, 1 must be subtracted before signaling the value to the bitstream. That is, Assuming that beta_displacement is the value that will be signaled to the bitstream, can we add 1 here, like g=scaler_table[idx_model]*(beta_displacement+1)? or Before signaling, the offset is subtracted from beta_displacement, and information can be added to the offset, such as whether it is equal to 1 or another value.

[0209] Typically, we use one beta for the rumor and one beta for the chroma, and they are scalars (not vectors). They can sometimes be similar (see Syntax Table 2 above).

[0210] The values ​​in "scaler_table_luma" are vectors. The number of elements in these vectors is equal to the number of channels in the latent space (y). Currently, the size of such vectors for lumas is 128, and for chromas it is 64.

[0211] Therefore, basically, one vector is selected from the table scaler_table_luma based on model_idx. As long as the number of elements in this vector is equal to the number of channels in the latent space, one element (one number) of the vector scaler_table_luma[model_idx] is used to scale one channel in the latent space.

[0212] In one embodiment, the first coding parameter is in logarithmic form (on a logarithmic scale) and is specified as beta_displacement_log_y or beta_displacement_log_luma or index_displacement_y or index_displacement_luma, where the first coding parameter represents the first displacement (e.g., ratio or difference) between a first target rate control parameter (e.g., a rate control parameter beta selected by the encoder for the first component) and a first reference rate control parameter (e.g., a rate control parameter beta used in model training). On a linear scale, beta_displacement is the ratio between the rate control parameter beta used for training the model and beta selected by the encoder, but (β displacement =β / β train ), on a logarithmic scale, beta_displacement_log is the difference between the logarithm of the rate control parameter beta used for model training and the beta selected by the encoder (β displacement_log =log k (β)-log k (β train )), however, k is a predetermined constant, which may be, for example, Euler's number e, 2, or equal to 10. For example, β displacement_log This uses an additional scaling factor to β displacement It can be calculated from this.

number

[0213] The main advantage of using a logarithmic scale is the reduction in complexity caused by changing multiplication / division operations into addition / subtraction operations.

[0214] Table 3 shows the syntax table relating to the signaling of rate control parameters in logarithmic form, as proposed, where a descriptor u(x) means that the corresponding syntax element (i.e., the syntax element in the same line that has that descriptor) is coded using a uniform probability distribution. The minimum codeword is 0 and the maximum codeword is 2 x It is -1. x is a positive integer, and can be, for example, 12, or any other positive integer less than 12, or any other positive integer greater than 12.

[0215] [Table 3]

[0216] independent_beta_uv is a flag (false / true) indicating whether the rate control parameter (β) for the first and second-order components are the same. In one embodiment, the first-order component refers to the lumen component, and the second-order component refers to the chromen component.

[0217] beta_displacement_log_y is a parameter that represents the ratio between the rate control parameter beta selected by the encoder for the first-order component and the rate control parameter beta used in model training. The parameter beta_displacement_log_y is in logarithmic form. In one embodiment, the first-order component means the luma component. On the encoder side, the original beta_displacement_log_y is [-2 N-1 ,2 N-1It may be within the range of -1], but the descriptor u(x) in Table 3 is allowed to signal only positive values, and therefore the actual beta_displacement_log_y being signaled in the bitstream is the same as the original beta_displacement_log_y (for example, the original beta_displacement_log_y before signaling it is 2 N-1 It may be an adjusted value (by adding ). On the decoder side, after receiving the positive value beta_displacement_log_y, the intermediate parameter betaDisplacementLogY is derived, and betaDisplacementLogY = beta_displacement_log_y - 2 N-1 Here, N is equal to x in the descriptor u(x) for beta_displacement_log_y, i.e., N is the number of bits used to signal beta_displacement_log_y. betaDisplacementLogY is used to derive the gain vector. The number of bits used to signal beta_displacement_log_y may be, for example, 12, or any other positive integer less than 12, or any other positive integer greater than 12.

[0218] beta_displacement_log_uv is a parameter that represents the ratio between the rate control parameter beta selected by the encoder for the quadratic component and the rate control parameter beta used in model training. In one embodiment, the quadratic component refers to the chroma component. The parameter beta_displacement_log_uv is in logarithmic form. On the encoder side, the original beta_displacement_log_uv is [-2 M-1 ,2 M-1It may be within the range of -1], but the descriptor u(x) in Table 3 is allowed to signal only positive values, and therefore the actual beta_displacement_log_uv being signaled in the bitstream is the same as the original beta_displacement_log_uv (for example, the original beta_displacement_log_uv before signaling it is 2 M-1 It may be an adjusted value (by adding to it). On the decoder side, after receiving the positive value beta_displacement_log_uv, the intermediate parameter betaDisplacementLogUV is derived, and betaDisplacementLogUV = beta_displacement_log_uv - 2 M-1 Here, M is equal to x in the descriptor u(x) for beta_displacement_log_uv, i.e., M is the number of bits used to signal beta_displacement_log_uv. betaDisplacementLogUV is used to derive the gain vector. The number of bits used to signal beta_displacement_log_uv may be, for example, 12, or any other positive integer less than 12, or any other positive integer greater than 12.

[0219] `model_id` is an identifier for a pre-stored checkpoint with model weights, where `model_id=0, 1, 2, 3, or 4`.

[0220] In one embodiment, when the first coding parameter and the second coding parameter are on a logarithmic scale, the relationship between Table 2 and Table 3 is as follows: that different_betas_for_luma_and_chroma_flag in Table 2 may be replaced with independent_beta_uv in Table 3, beta_displacement_luma in Table 2 may be replaced with beta_displacement_log_y in Table 3, beta_displacement_chroma in Table 2 may be replaced with beta_displacement_log_uv in Table 3, and model_idx in Table 2 may be replaced with model_id in Table 3.

[0221] In Table 3, parameters and flags are signaled within the model header, while in other implementations, parameters and flags may be signaled within other headers or parameter sets of the bitstream, such as the channel header, sequence parameter set, picture parameter set, picture header, image header, etc.

[0222] Derivation of control parameters and gain vector In one embodiment, a total of five models are trained for different quality ranges. The models are selected based on the modelIdx coded in the picture header. The rate control parameter β controls the compression ratio and defines the operation in the gain unit, inverse gain unit, and sigma scale (may be marked as sGain).

[0223] The forward gain vector m is used on the encoder side within the gain unit. The forward gain vector m is on a logarithmic scale. log This is used within the sigma scale (a module used to scale the standard deviation tensor). The inverse gain vector m -1 These are used on the decoder side within the inverse gain unit. The forward gain vector m and the inverse gain vector m -1Both have a size [C] equal to the number of channels of the residual tensor.

[0224] The learnable model has, for five models t = 0,..., 4, a reference forward gain vector

Number

Number

Number

[0225] The input to the gain vector derivation process is - The 12-bit variable betaDisplacementLog, equal to betaDisplacementLogY for the primary component and betaDisplacementLogUV for the secondary component, - modelIdx indicating the learnable model to be used, - The reference forward gain vector m for t = modelIdx t log That is.

[0226] The output of the gain vector derivation process is - The forward gain vector m on a logarithmic scale log , - The forward gain vector m, - The inverse gain vector m -1 That is.

[0227] The forward gain vector m on a logarithmic scale log Is calculated as follows. - m log = m t log + betaDisplacementLog

[0228] Forward gain vector m and backward gain vector m -1 are, as follows, namely, [Number] is calculated as, provided that k, N σ , σ max , σ min , and sigmaPrecision are default constants, k can be equal to, for example, the Euler's number e or 2 or 10, σ min can be equal to, for example, 0.11 or 0.2, σ max can be equal to, for example, 100 or 30, N σ can be equal to, for example, 35 or 32, and sigmaPrecision is a positive integer and can be equal to, for example, 7.

[0229] In one embodiment, the forward gain vector m can be replaced with the expression g, and thus, the forward gain vector m in logarithmic scale log is the expression g log and can be replaced, and the reference forward gain vector m for t = modelIdx t log is the expression g t log and can be replaced, and the backward gain vector m -1 is the expression g -1 and can be replaced.

[0230] m log is used in the sigma scale (a module used to scale the standard deviation index tensor). An exemplary implementation of the sigma scale (sigma index scale, standard deviation index scale, CDF table index scale) is provided below.

[0231] Sigma scale This process is applied on both the encoder side and the decoder side, and thus, it complies with the specification.

[0232] The input to this process is, - The forward gain vector m is on a logarithmic scale. log , - This is the standard deviation log tensor Iσ of size [C,h4,w4], which is the output of the hyperscale decoder.

[0233] The output of this process is, - It is scaled by the forward gain vector standard deviation log tensor I'σ of size [C,h4,w4] to an adaptive sigma scale.

[0234] The sigma scale is applied by modifying the standard deviation log tensor Iσ[C,h4,w4] as follows. I'σ[c,i,j]=m log [c]+Iσ[c,i,j], i=0,...,h4-1, j=0,...,w4-1, c=0,...,C-1

[0235] All elements of a tensor within the same channel are scaled by the same multiplier.

[0236] Figure 12 is a schematic diagram showing the relationship between a first coding parameter (i.e., network input parameter) β and a gain vector. During the training stage of a neural network (such as encoder 101 in Figure 3A), several values ​​of β are inputs to the network and are trained to obtain the corresponding gain vectors. Figure 12 shows four pre-trained pairs of [β, gain vector]. Note that this disclosure is not limited to such implementations, and generally, any other possible number of [β, gain vector] pairs may be pre-trained, for example, in possible designs, five, eight, or ten pairs of [β, gain vector] pairs may be pre-trained.

[0237] The encoding method 1300 shown in Figure 13 includes the steps of: 1301 acquiring an image to be encoded; 1302 encoding the image into a bitstream based on a first coding parameter, wherein the first coding parameter represents a first displacement between a first target rate control parameter and a first reference rate control parameter; and 1303 encoding the first coding parameter into a bitstream.

[0238] According to the alternative option, the first coding parameter may be directly encoded and signaled in the bitstream, or another coding parameter obtained based on the first coding parameter may be encoded and signaled in the bitstream.

[0239] The first displacement may be a difference or a ratio. The first target rate control parameter may be a rate control parameter selected by an encoder performing a method for encoding an image, according to the first embodiment and any implementation thereof, and the first reference rate control parameter may be a rate control parameter used in model training.

[0240] In possible designs, the first coding parameter is represented by a fixed-point value.

[0241] In a possible design, the first coding parameter is signaled using N bits, where N is an integer less than or equal to 16.

[0242] In a possible design, the first coding parameter is represented by a fixed-point value having N bits, where K bits of the N bits represent the fractional part of the first coding parameter, and (NK) bits of the N bits represent the integer part of the first coding parameter, where N is an integer and K is an integer less than or equal to N.

[0243] In a possible design, N is equal to 10 and K is equal to 5.

[0244] In a possible design, N is equal to 8 and K is equal to 5.

[0245] In possible designs, N and K can be integers of any other value, as long as K is less than or equal to N, and this is not limited here.

[0246] In a possible design, the first coding parameter is given by the following conditions: β d =β / β r -x Defined by, β d is the first coding parameter, β is the target rate control parameter, and β r x is a reference rate control parameter, where x is a default number, for example, x can be 1 or 2. In other possible designs, x is

number

number

[0247] In a possible design, the first coding parameter is given by the following conditions: β d =β-β r -x Defined by, β d is the first coding parameter, β is the target rate control parameter, and β r is a reference rate control parameter, where x is a default number. In this design, x is an integer and may be a positive integer.

[0248] Therefore, in a possible design, encoding the image into a bitstream based on the first coding parameter is, Obtaining the target gain vector based on the first coding parameter, This includes encoding an image based on a target gain vector, The target gain vector is given by the following condition: g=β d +x It is defined by and g is the target gain vector.

[0249] In a possible design, the first coding parameter is given by the following conditions: β d =(β-β r -x) / y Defined by, β d is the first coding parameter, β is the target rate control parameter, and β r is a reference rate control parameter, where x and y are default numbers. In this design, x and y are integers, and may be positive integers.

[0250] In a possible design, the first coding parameter is given by the following conditions: β d =(β / β r -x) / y Defined by, β d is the first coding parameter, β is the target rate control parameter, and β r is a reference rate control parameter, where x and y are default numbers. In this design, x and y are integers, and may be positive integers.

[0251] Therefore, in a possible design, encoding the image into a bitstream based on the first coding parameter is, Obtaining the target gain vector based on the first coding parameter, This includes encoding an image based on a target gain vector, The target gain vector is given by the following condition: g=β d×y+x It is defined by and g is the target gain vector.

[0252] In a possible design, the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are for coding the lumens data (primary or lumens) of the image.

[0253] In a possible design, the method is The process further includes encoding a second coding parameter into a bitstream, where the second coding parameter represents a second displacement between a second target rate control parameter and a second reference rate control parameter. The second coding parameter, the second target rate control parameter, and the second reference rate control parameter are used to code the chroma data (secondary components or chroma components) of the image.

[0254] The second displacement may be a difference or a ratio. The second target rate control parameter may be a rate control parameter selected by an encoder performing a method for encoding an image, according to the first embodiment and any implementation thereof, and the second reference rate control parameter may be a rate control parameter used in model training.

[0255] In a possible design, the coding of the second coding parameter is based on a flag indicating that different rate control parameters are used to code the lumens and chroma data of the image.

[0256] In possible designs, the second coding parameter is represented by a fixed-point value.

[0257] The signaling of the second coding parameter is similar to the signaling of the first coding parameter and may have different bits for signaling the first and second coding parameters.

[0258] In a possible design, the method is This further includes encoding a third coding parameter for deriving the target gain vector into the bitstream.

[0259] In a possible design, the third coding parameter indicates the index of the trained model relative to the first reference rate control parameter and / or the second reference rate control parameter.

[0260] In a possible design, the target gain vector is given by the following condition: g=scaler_table[idx_model]×(β d +x) Defined by, g is the target gain vector, scaler_table is the lookup table, idx_model is the third coding parameter, and β d is the first coding parameter, x is a default number, and (β d (x) is not equal to 0.

[0261] In a possible design, (β d +x) is a positive integer.

[0262] In a possible design, each element in the lookup table is represented by an M-bit fixed-point value, where P bits of the M bits represent the fractional part of one of the elements.

[0263] In a possible design, M is equal to 6 and P is equal to 5.

[0264] In a possible design, the first coding parameter and / or the second coding parameter are the following code, i.e., - Binary code, - Unary code, - Abbreviated unary codes, and - Exponential Golomb Code Signaling is performed using one or more of the following.

[0265] In possible designs, the first coding parameter is on a logarithmic scale.

[0266] In possible designs, the first coding parameter, target rate control parameter, and reference rate control parameter are on a logarithmic scale.

[0267] In possible designs, the second coding parameter is on a logarithmic scale.

[0268] In a possible design, N is equal to 12.

[0269] In a possible design, encoding the first coding parameter into a bitstream is: Obtaining the adjusted value of the first coding parameter, This includes encoding the adjusted value of the first coding parameter into a bitstream.

[0270] In a possible design, the adjusted value of the first coding parameter is 2 times the original value of the first coding parameter. N-1 The result is obtained by adding, where N is the number of bits used to signal the first coding parameter. The adjusted value may be the value of any other coding parameter mentioned above.

[0271] In a possible design, the first coding parameter is given by the following conditions:

number

[0272] In a possible design, k is equal to Euler's number e or 2, σ min σ is equal to 0.11 or 0.2. max is equal to 100 or 30, N σ It is equal to 35 or 32.

[0273] In a possible design, the first coding parameter is given by the following conditions: β d =log k (β / β r ) Defined by, β d is the first coding parameter, β is the target rate control parameter, and β r is the reference rate control parameter, and k is a default constant.

[0274] In possible designs, k is equal to Euler's number e or 2.

[0275] In a possible design, encoding an image into a bitstream based on a first coding parameter is: Obtaining the target gain vector based on the first coding parameter, This includes encoding an image based on a target gain vector, The target gain vector is given by the following condition: g = scaler_table[idx_model] + β d_log Defined by, g is the target gain vector, scaler_table is the lookup table, idx_model indicates the index of the trained model related to the first reference rate control parameter and / or the second reference rate control parameter, β d_log This is the first coding parameter.

[0276] In a possible design, encoding an image into a bitstream based on a first coding parameter is: Obtaining the target gain vector based on the first coding parameter, This includes encoding an image based on a target gain vector, The target gain vector is as follows, that is, m log =m t log +betaDisplacementLog It is defined as, m log m is the target gain vector. t log is a reference forward gain vector for a model with index t, where t represents the index of the trained model relative to a first reference rate control parameter and / or a second reference rate control parameter, and betaDisplacementLog is relative to the first coding parameter. For example, betaDisplacementLog represents the first coding parameter or any other coding parameter mentioned above.

[0277] In a possible design, the first target rate control parameter is the rate control parameter selected by the encoder for the first-order / lumen component, and the first reference rate control parameter is the rate control parameter used in model training (for the first-order / lumen component).

[0278] In a possible design, the second target rate control parameter is the rate control parameter selected by the encoder for the second-order / chroma component, and the second reference rate control parameter is the rate control parameter used in model training (for the second-order / chroma component).

[0279] Decoding method Figure 14 is a flowchart illustrating an exemplary method for decoding an image based on a neural network architecture, comprising: step 1401, receiving a bitstream containing coded data of an image; step 1402, parsing the bitstream to obtain a first coding parameter, wherein the first coding parameter represents a first displacement between a first target rate control parameter and a first reference rate control parameter; and step 1403, reconstructing the image based on the first coding parameter.

[0280] The first displacement may be a difference or a ratio. The first target rate control parameter may be a rate control parameter selected by an encoder performing a method for encoding an image, according to the first embodiment and any implementation thereof, and the first reference rate control parameter may be a rate control parameter used in model training.

[0281] According to one option (Option A), parsing a bitstream to obtain a first coding parameter involves parsing another coding parameter (syntax element) from the bitstream and obtaining the first coding parameter based on that other coding parameter.

[0282] According to another option (Option B), parsing a bitstream to obtain a first coding parameter includes directly parsing the first coding parameter from the bitstream. According to this option, reconstructing an image based on the first coding parameter may include obtaining another coding parameter based on the parsed first coding parameter and then reconstructing the image based on that other coding parameter.

[0283] In possible designs, the first coding parameter is represented by a fixed-point value.

[0284] In a possible design, the first coding parameter is signaled using N bits, where N is an integer less than or equal to 16.

[0285] In a possible design, the first coding parameter is represented by a fixed-point value having N bits, where K bits of the N bits represent the fractional part of the first coding parameter, and (NK) bits of the N bits represent the integer part of the first coding parameter, where N is an integer and K is an integer less than or equal to N.

[0286] In a possible design, N is equal to 10 and K is equal to 5.

[0287] In a possible design, N is equal to 8 and K is equal to 5.

[0288] In a possible design, the first coding parameter is given by the following conditions: β d =β / β r -x Defined by, β d is the first coding parameter, β is the target rate control parameter, and β r is the reference rate control parameter, and x is the default number.

[0289] In a possible design, the first coding parameter is given by the following conditions: β d =β-β r -x Defined by, β d is the first coding parameter, β is the target rate control parameter, and β r is the reference rate control parameter, and x is the default number.

[0290] In a possible design, the first coding parameter is given by the following conditions: β d =(β-β r -x) / y Defined by, β d is the first coding parameter, β is the target rate control parameter, and β r is a reference rate control parameter, where x and y are default values.

[0291] In a possible design, reconstructing the image based on a first coding parameter is possible. Obtaining the target gain vector based on the first coding parameter, This includes reconstructing the image based on the target gain vector, The target gain vector is given by the following condition: g=β d +x It is defined by and g is the target gain vector.

[0292] In a possible design, reconstructing the image based on a first coding parameter is possible. Obtaining the target gain vector based on the first coding parameter, This includes reconstructing the image based on the target gain vector, The target gain vector is given by the following condition: g=β d ×y+x It is defined by and g is the target gain vector.

[0293] In a possible design, the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are used to code the lumens data (first-order components or lumens components) of the image.

[0294] In a possible design, the method is This further involves parsing the bitstream to obtain a second coding parameter, the second coding parameter representing a second displacement between a second target rate control parameter and a second reference rate control parameter. The second coding parameter, the second target rate control parameter, and the second reference rate control parameter are used to code the chroma data (secondary components or chroma components) of the image.

[0295] The second displacement may be a difference or a ratio. The second target rate control parameter may be a rate control parameter selected by an encoder performing the method according to the first embodiment and any implementation thereof, and the second reference rate control parameter may be a rate control parameter used in model training.

[0296] In a possible design, the parsing of the second coding parameter is based on flagging to indicate that different rate control parameters are used to code the rumor and chroma data of the image.

[0297] In possible designs, the second coding parameter is represented by a fixed-point value.

[0298] In a possible design, the method is This further includes parsing the bitstream to obtain a third coding parameter for deriving the target gain vector.

[0299] In a possible design, the third coding parameter indicates the index of the trained model relative to the first reference rate control parameter and / or the second reference rate control parameter.

[0300] In a possible design, the target gain vector is given by the following condition: g=scaler_table[idx_model]×(β d +x) Defined by, g is the target gain vector, scaler_table is the lookup table, idx_model is the third coding parameter, and β d is the first coding parameter, x is a default number, and (β d (x) is not equal to 0.

[0301] In a possible design, (β d +x) is a positive integer.

[0302] In a possible design, each element in the lookup table is represented by an M-bit fixed-point value, where P bits of the M bits represent the fractional part of one of the elements.

[0303] In a possible design, M is equal to 6 and P is equal to 5.

[0304] In a possible design, the first coding parameter and / or the second coding parameter are the following code, i.e., - Binary code, - Unary code, - Abbreviated unary codes, and - Exponential Golomb Code Signaling is performed using one or more of the following.

[0305] In possible designs, the first coding parameter is on a logarithmic scale.

[0306] In possible designs, the first coding parameter, target rate control parameter, and reference rate control parameter are on a logarithmic scale.

[0307] In possible designs, the second coding parameter is on a logarithmic scale.

[0308] In a possible design, N is equal to 12.

[0309] In a possible design, reconstructing the image based on a first coding parameter is possible. From the first coding parameter 2 N-1 The intermediate parameter (for example, other coding parameters according to option B above) is obtained by subtracting, where N is the number of bits used to signal the first coding parameter, This includes reconstructing the image based on intermediate parameters.

[0310] In a possible design, reconstructing the image based on a first coding parameter is possible. Obtaining the target gain vector based on the first coding parameter, This includes reconstructing the image based on the target gain vector, The target gain vector is given by the following condition: g = scaler_table[idx_model] + β d_log Defined by, g is the target gain vector, scaler_table is the lookup table, idx_model indicates the index of the trained model related to the first reference rate control parameter and / or the second reference rate control parameter, β d_log This is the first coding parameter.

[0311] In a possible design, reconstructing the image based on a first coding parameter is possible. Obtaining the target gain vector based on the first coding parameter, This includes reconstructing the image based on the target gain vector, The target gain vector is as follows, that is, m log =m t log +betaDisplacementLog It is defined as, m log m is the target gain vector. t log is a reference forward gain vector for a model with index t, where t represents the index of the trained model relative to a first reference rate control parameter and / or a second reference rate control parameter, and betaDisplacementLog is relative to the first coding parameter. For example, it represents the first coding parameter or any other coding parameter mentioned above.

[0312] In a possible design, the first target rate control parameter is the rate control parameter selected by the encoder for the first-order / lumen component, and the first reference rate control parameter is the rate control parameter used in model training (for the first-order / lumen component).

[0313] In a possible design, the second target rate control parameter is the rate control parameter selected by the encoder for the second-order / chroma component, and the second reference rate control parameter is the rate control parameter used in model training (for the second-order / chroma component).

[0314] Encoding devices and decryption devices Furthermore, as already stated, the Disclosure also provides a device configured to perform steps of the method described above. Figure 15 shows an encoding device 1500 for encoding an image. Device 1510 comprises an acquisition unit configured to acquire an image and an encoding unit 1520 configured to encode the image into a bitstream based on a first coding parameter, the first coding parameter being a first displacement between a first target rate control parameter and a first reference rate control parameter, and the encoding unit 1520 is further configured to encode the first coding parameter into a bitstream. The encoding device 1500 may be configured to perform a method for encoding an image, such as those described in the design options above.

[0315] Corresponding to the encoding device 1500 described above, a decoding device 1600 for decoding a bitstream to reconstruct an image is shown in Figure 16. The device may comprise a receiving unit 1610 configured to receive a bitstream containing coded data of an image, and a parsing unit 1620 configured to parse the bitstream to obtain a first coding parameter, the first coding parameter representing a first displacement between a first target rate control parameter and a first reference rate control parameter, and a reconstruction unit 1630 configured to reconstruct the image based on the first coding parameter. The decoding device 1600 may be configured to perform a method for decoding an image, as described in the design options above.

[0316] It should be noted that these devices may be further configured to perform any additional functions, including the exemplary implementations described above. For example, a device may be provided for decoding a feature map for processing by a neural network based on a bitstream, and the device may include processing circuitry configured to perform any step of the decoding method described above. Similarly, a device may be provided for encoding a feature map for processing by a neural network into a bitstream, and the device may include processing circuitry configured to perform any step of the encoding method described above.

[0317] Further devices utilizing devices 1500 and / or 1600 may be provided. For example, a device for image or video encoding may include encoding device 1500. In addition, it may include decoding device 1600. A device for image or video decoding may include decoding device 1600 and / or encoding device 1500.

[0318] Furthermore, a coding system utilizing devices 1500 and / or 1600 may be provided. For example, the coding system may be deployed in a server. The server receives a bitstream, then uses device 1600 or another decoder to decode the bitstream and obtain an image, and then encodes the image using device 1500 or another encoder. After encoding, the newly obtained bitstream is stored and / or transmitted to another device.

[0319] While the operations are shown in a specific order in the drawings, this should not be interpreted as requiring that such operations be performed in a specific or sequential order shown, or that all illustrated operations be performed, in order to achieve the desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated together in a single software product or packaged in multiple software products.

[0320] Specific embodiments of this subject matter are described. Other embodiments fall within the scope of the following claims. For example, the operations described in the claims can be performed in a different order and still achieve the desired results. As an example, the processes shown in the accompanying drawings do not necessarily require the specific illustrated order or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0321] Figure 17 illustrates the bitstream structure that may be provided.

[0322] It should be noted that the bitstream may be obtained via a wireless or wired network. The bitstream may be transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, microwave, Wi-Fi, Bluetooth, LTE, or 5G.

[0323] A bitstream may be a sequence of bits in the form of a network abstraction layer (NAL) unit stream or byte stream that forms a representation of a sequence of access units (AUs) that make up one or more coded video sequences (CVS).

[0324] In concrete terms, a bitstream format specifies the relationship between a network abstraction layer (NAL) unit stream and a byte stream, both of which are called bitstreams.

[0325] A bitstream can be in one of two formats: a NAL unit stream format or a byte stream format. The NAL unit stream format is conceptually the more "basic" type. It contains a sequence of syntactic structures called NAL units, which are ordered in decryption order. There are constraints on the decryption order (and content) of NAL units within a NAL unit stream.

[0326] A byte stream format may consist of a NAL unit stream format, which is created by ordering NAL units in decoding order and by placing a start code prefix and zero or more zero-value bytes to form a stream of bytes before each NAL unit. The NAL unit stream format may be extracted from the byte stream format by searching for the location of a unique start code prefix pattern within this stream of bytes.

[0327] The bitstream may include coded data of the image and a first coding parameter, the first coding parameter representing a first displacement between a first target rate control parameter and a first reference rate control parameter.

[0328] In possible designs, the first coding parameter is represented by a fixed-point value.

[0329] In a possible design, the first coding parameter is signaled using N bits, where N is an integer less than or equal to 16.

[0330] In a possible design, the first coding parameter is represented by a fixed-point value having N bits.

[0331] In a possible design, the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are for coding the rumor data of the image.

[0332] In a possible design, the bitstream further includes a second coding parameter, which represents a second displacement between a second target rate control parameter and a second reference rate control parameter. The second coding parameter, the second target rate control parameter, and the second reference rate control parameter are for coding the chroma data of the image.

[0333] In a possible design, the bitstream would further include flags indicating whether different rate control parameters are used to code the rumor and chroma data of the image.

[0334] In possible designs, the second coding parameter is represented by a fixed-point value.

[0335] In a possible design, the bitstream further includes a third coding parameter, which indicates the index of the trained model related to the first and / or second reference rate control parameters.

[0336] Figure 18 shows the encoding and decoding process using the gain vector scheme.

[0337] Several exemplary implementation forms in hardware and software Figure 22 is a block diagram showing an example of a video coding system configured to implement an embodiment of the present disclosure.

[0338] A corresponding system in which the encoder-decoder processing chain described above can be deployed is shown in Figure 23. Figure 23 is a schematic block diagram showing exemplary coding systems that may utilize the techniques of this application, for example, video, image, audio, and / or other coding systems (or, simple coding systems). The video encoder 20 (or simple encoder 20) and video decoder 30 (or simple decoder 30) of the video coding system 10 represent examples of devices that may be configured to perform the techniques described in the various examples in this application. For example, video coding and decoding may employ a neural network that can be distributed and to which the bitstream parsing and / or bitstream generation described above can be applied in order to transmit feature maps between distributed computing nodes (two or more).

[0339] As shown in Figure 23, the coding system 10 includes, for example, a source device 12 configured to provide encoded picture data 21 to a destination device 14 for decoding encoded picture data 13.

[0340] The source device 12 includes an encoder 20 and may additionally, i.e., optionally, include a picture source 16, a preprocessor (or preprocessing unit) 18, for example, a picture preprocessor 18, and a communication interface or communication unit 22.

[0341] The picture source 16 may comprise, or may comprise, any type of picture capture device, e.g., a camera for capturing real-world pictures, and / or any type of picture generation device, e.g., a computer graphics processor for generating computer-animated pictures, or any other type of device for acquiring and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). The picture source may comprise any type of memory or storage device for storing any of the pictures described above.

[0342] To distinguish it from the processing performed by the preprocessor 18 and the preprocessing unit 18, the picture or picture data 17 is sometimes referred to as the raw picture or raw picture data 17.

[0343] The preprocessor 18 is configured to receive (raw) picture data 17, perform preprocessing on the picture data 17 to obtain a preprocessed picture 19 or preprocessed picture data 19. The preprocessing performed by the preprocessor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or denoising. It can be understood that the preprocessing unit 18 may consist of optional components. It should also be noted that the preprocessing may employ a neural network using existence index signaling (as in any of Figures 1 to 7).

[0344] The video encoder 20 is configured to receive pre-processed picture data 19 and provide encoded picture data 21.

[0345] The communication interface 22 of the source device 12 may be configured to receive encoded picture data 21 and to transmit the encoded picture data 21 (or any further processed version thereof) to another device, such as the destination device 14 or any other device, via the communication channel 13 for storage or direct reconstruction.

[0346] The destination device 14 includes a decoder 30 (for example, a video decoder 30) and may additionally, i.e., optionally, include a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0347] The communication interface 28 of the destination device 14 is configured to receive encoded picture data 21 (or any further processed version thereof) from, for example, directly from the source device 12 or from any other source, for example, a storage device, for example, an encoded picture data storage device, and to provide the encoded picture data 21 to the decoder 30.

[0348] Communication interfaces 22 and 28 may be configured to transmit or receive encoded picture data 21 or encoded data 13 via a direct communication link between the source device 12 and the destination device 14, for example, via a direct wired or wireless connection, or via any type of network, for example, a wired network or a wireless network or any combination thereof, or any type of private and public network, or any type of combination thereof.

[0349] The communication interface 22 may be configured, for example, to package the encoded picture data 21 into an appropriate format, such as a packet, and / or to process the encoded picture data using any kind of transmit encoding or processing for transmission over a communication link or communication network.

[0350] The communication interface 28, which forms the other end of the communication interface 22, may be configured, for example, to receive transmitted data and process the transmitted data using any kind of corresponding decryption or processing and / or depackaging to obtain encoded picture data 21.

[0351] Both communication interfaces 22 and 28 may be configured as unidirectional or bidirectional communication interfaces, as indicated by the arrows to the communication channel 13 in Figure 23 pointing from the source device 12 to the destination device 14, for example, to set up a connection, a communication link and / or to transmit data, for example, to send and receive messages to acknowledge and exchange any other information relating to the transmission of encoded picture data. The decoder 30 is configured to receive encoded picture data 21 and provide decoded picture data 31 or decoded picture 31.

[0352] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), for example, the decoded picture 31, in order to obtain the post-processed picture data 33, for example, the post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (for example, from YCbCr to RGB), color correction, cropping, or resampling, or any other processing, to prepare the decoded picture data 31 for display by the display device 34.

[0353] The display device 34 of the destination device 14 is configured to receive post-processed picture data 33 for displaying the picture to a user or viewer, for example. The display device 34 may be any type of display for representing the reconstructed picture, for example, an integrated or external display or monitor, or may comprise them. The display may comprise, for example, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a microLED display, a liquid crystal on silicon (LCoS), a digital optical processor (DLP), or any other type of display.

[0354] Figure 23 shows the source device 12 and the destination device 14 as separate devices, but the device embodiment may also include the functionality of both the source device 12 or its corresponding function and the destination device 14 or its corresponding function, or both. In such embodiments, the source device 12 or its corresponding function and the destination device 14 or its corresponding function may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0355] As will be apparent to those skilled in the art based on the description, the functions of different units or the presence and (strict) division of functions within source device 12 and / or destination device 14 as shown in Figure 23 may vary depending on the actual device and application.

[0356] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30, may be implemented by processing circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof dedicated to video coding. The encoder 20 may be implemented by processing circuits 46 to embody various modules, including a neural network or a portion thereof. The decoder 30 may be implemented by processing circuits 46 to embody any coding system or subsystem described herein. The processing circuits may be configured to perform various operations, as described later. If the technique is partially implemented in software, the device may store instructions for the software in a suitable non-temporary computer-readable storage medium, and may use one or more processors to execute the instructions in hardware to perform the technique of this disclosure. Either the video encoder 20 or the video decoder 30 may be integrated as part of a composite encoder / decoder (codec) in a single device, for example, as shown in Figure 24.

[0357] The source device 12 and destination device 14 may comprise any of a wide range of devices, including handheld or stationary devices of any kind, such as notebook or laptop computers, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (such as content service servers or content distribution servers), broadcast receiver devices, broadcast transmitter devices, etc., and may or may not have an operating system. In some cases, the source device 12 and destination device 14 may be equipped for wireless communication. Therefore, the source device 12 and destination device 14 may be wireless communication devices.

[0358] In some cases, the video coding system 10 shown in Figure 23 is merely an example, and the techniques of this application may be applied to video coding configurations (e.g., video coding or video decoding) that do not necessarily involve any data communication between the coding device and the decoding device. In other examples, data may be retrieved from local memory and streamed over a network. The video coding device may code the data and store it in memory, and / or the video decoding device may retrieve the data from memory and decode it. In some examples, coding and decoding are performed by devices that do not communicate with each other but simply code the data into memory and / or retrieve the data from memory and decode it.

[0359] Figure 25 is a schematic diagram of a video coding device 8000 according to one embodiment of the present disclosure. The video coding device 8000 is suitable for implementing the disclosed embodiments as described herein. In one embodiment, the video coding device 8000 may be a decoder, such as the video decoder 30 in Figure 23, or an encoder, such as the video encoder 20 in Figure 23.

[0360] The video coding device 8000 comprises an inlet port 8010 (or input port 8010) and a receiver unit (Rx) 8020 for receiving data, a processor, logic unit, or central processing unit (CPU) 8030 for processing data, a transmitter unit (Tx) 8040 and an exit port 8050 (or output port 8050) for transmitting data, and memory 8060 for storing data. The video coding device 8000 may also comprise optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to the inlet port 8010, a receiver unit 8020, a transmitter unit 8040, and an exit port 8050 for the exit or input of optical or electrical signals.

[0361] The processor 8030 is implemented by hardware and software. The processor 8030 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGAs, ASICs, and DSPs. The processor 8030 communicates with the inlet port 8010, the receiver unit 8020, the transmitter unit 8040, the exit port 8050, and the memory 8060. The processor 8030 includes a neural network-based codec 8070. The neural network-based codec 8070 implements the embodiments disclosed above. For example, the neural network-based codec 8070 performs, processes, prepares, or provides various coding operations. Thus, including the neural network-based codec 8070 results in a significant improvement to the functionality of the video coding device 8000 and affects the conversion of the video coding device 8000 to different states. Alternatively, the neural network-based codec 8070 is implemented as instructions stored in memory 8060 and executed by processor 8030.

[0362] The memory 8060 may comprise one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device for storing a program when such a program is selected for execution, and for storing instructions and data read during program execution. The memory 8060 may be, for example, volatile and / or non-volatile, and may be read-only memory (ROM), random-access memory (RAM), tri-level associative memory (TCAM), and / or static random-access memory (SRAM).

[0363] Figure 28 is a simplified block diagram of an apparatus that may be used as one or both of the source device 12 and destination device 14 from Figure 23, according to an exemplary embodiment.

[0364] The processor 9002 in the device 9000 may be a central processing unit. Alternatively, the processor 9002 may be any other type of device or multiple devices, existing or to be developed in the future, capable of handling or processing information. The disclosed implementation may be realized using a single processor, e.g., processor 9002, as shown in the figure, but advantages in speed and efficiency may be achieved using two or more processors.

[0365] The memory 9004 in the device 9000 may be a read-only memory (ROM) device or a random access memory (RAM) device in one implementation. Any other suitable type of storage device may be used as memory 9004. Memory 9004 may contain code and data 9006 accessed by the processor 9002 using the bus 9012. Memory 9004 may further contain an operating system 9008 and an application program 9010, the application program 9010 containing at least one program that allows the processor 9002 to perform the method described herein. For example, the application program 9010 may contain applications 1 to N, the applications 1 to N further containing a video coding application that performs the method described herein.

[0366] The device 9000 may also include one or more output devices, such as a display 9018. The display 9018 may, in one example, be a touch-sensitive display that combines the display with a touch-sensitive element capable of operating to sense touch input. The display 9018 may be coupled to the processor 9002 via the bus 9012.

[0367] Although shown here as a single bus, the bus 9012 of device 9000 may consist of multiple buses. Furthermore, the secondary storage device may be directly coupled to other components of device 9000 or accessed via a network, and may comprise a single integrated unit such as a memory card or multiple units such as multiple memory cards. Device 9000 can be implemented in a wide variety of configurations.

[0368] Figure 29 is a block diagram of a video coding system 10000 according to one embodiment of the present disclosure.

[0369] Platform 10002 within System 10000 may be a cloud server or a local server. Alternatively, Platform 10002 may be any other type of device or multiple devices capable of computing, storing, transcoding, encrypting, rendering, decrypting, or encoding. The disclosed implementation may be realized with a single platform, such as Platform 10002, as illustrated, but advantages in speed and efficiency may be achieved by using two or more platforms.

[0370] The Content Delivery Network (CDN) 10004 within System 10000 may be a group of geographically distributed servers. Alternatively, CDN 10004 may be any other type of device or multiple devices that can speed up data buffering, scheduling, distribution, or delivery of web content by bringing it closer to the user's location. While the disclosed implementation can be achieved using a single CDN, e.g., CDN 10004, as illustrated, the speed and efficiency advantages may be achieved by using two or more CDNs.

[0371] Terminal 10006 within device 10000 may be a mobile phone, computer, television, laptop, or camera. Alternatively, terminal 10006 may be any other type of device or multiple devices capable of displaying video or images.

[0372] Entropy decoding may be performed in parallel, for example, by a multicore decoder. In addition, only the entropy decoding portion may be performed in parallel. Figure 19 shows an exemplary configuration of a parallel (e.g., multicore) encoder 620. Each of the input data channels 610 may be encoded into individual substreams containing coded bits 630-633 and trailing bits 640-643. The length 650 of the substreams is signaled. In the parallel processing implementation, the bitstream consists of several substreams that are concatenated in the final step. Each substream needs to be complete, because the substreams are encoded independently of each other, and as a result, encoding (and therefore decoding) one substream does not require the prior encoding (or decoding) of one or more other substreams.

[0373] An input data channel may refer to a channel obtained by processing several data by a neural network. For example, the input data may be a feature channel, such as the output channel or latent representation channel of the neural network. In exemplary implementations, the neural network may be a deep neural network and / or a convolutional neural network. The neural network may be trained to process pictures (still or moving). The processing may be for picture coding and reconstruction, or for computer vision such as object recognition, classification, or segmentation. In general, this disclosure is not limited to any particular type of task or neural network. Rather, this disclosure is applicable to coding any type of data coming from multiple channels, which will generally be understood as any source of data. Furthermore, the channels may be provided by preprocessing of the source data.

[0374] Implementation forms within picture coding One possible deployment can be seen in Figures 20 and 21.

[0375] Figure 20 shows a schematic block diagram of an exemplary encoder 20 configured to implement the technique of the present application. In the example of Figure 20, the encoder 20 includes an input unit 201 (or input interface 201), a residual calculation unit 204, a transformation unit 206, a quantization unit 208, an inverse quantization unit 210 and an inverse transformation unit 212, a reconstruction unit 214, a loop filter unit 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output unit 272 (or output interface 272). The entropy coding 270 may implement an arithmetic coding method or apparatus as described above.

[0376] The mode selection unit 260 may include an inter-prediction unit 244, an intra-prediction unit 254, and a segmentation unit 262. The inter-prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The encoder 20 shown in Figure 20 is sometimes called a hybrid encoder or an encoder with a hybrid video / image codec.

[0377] The encoder 20 may be configured to receive, for example, a picture 17 (or picture data 17 or point cloud data, motion flow or other type of media data), for example, a picture or sequence of pictures that form a video or video sequence, via the input 201. The received picture or picture data may also be a pre-processed picture 19 (or pre-processed picture data 19). For simplicity, the following description refers to picture 17. Picture 17 is also sometimes called the current picture or the picture to be coded (in detail, in video coding, to distinguish the current picture from other pictures, for example, previously encoded and / or decoded pictures of the same video sequence, i.e., the video sequence that also includes the current picture).

[0378] A (digital) picture is, or can be considered, a two-dimensional array or matrix of samples having intensity values. A sample in an array is sometimes called a pixel (a short form of picture element) or pel. The number of samples in the horizontal and vertical directions (or axes) of the array or picture determines the size and / or resolution of the picture. For color representation, typically three color components are employed; that is, a picture may represent, or contain, three sample arrays. In the RGB format or RGB color space, a picture contains corresponding red, green, and blue sample arrays. However, in video coding, each pixel is typically represented by a luminance and chrominance format or color space, for example, YCbCr, which includes a luminance component represented by Y (sometimes L is used instead), and two chrominance components represented by Cb and Cr. The luminance (or simply luma) component Y represents brightness or gray level intensity (for example, in a grayscale picture), and the two chrominance (or simply chroma) components Cb and Cr represent chromaticity or color information components. Thus, a picture in YCbCr format contains a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). A picture in RGB format may be converted to or transformed to YCbCr format, and vice versa; this process is also called a color transformation or conversion. If the picture is monochrome, it may contain only a luminance sample array. Thus, a picture may be, for example, an array of luma samples in a monochrome format, or an array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0379] Embodiments of the encoder 20 may include a picture partitioning unit (not shown in Figure 20) configured to divide a picture 17 into multiple (usually non-overlapping) picture blocks 203. These blocks may also be called root blocks, macroblocks (H.264 / AVC), or coding tree blocks (CTBs) or coding tree units (CTUs) (H.265 / HEVC and VVC). The picture partitioning unit may use the same block size and corresponding grid defining that block size for all pictures in the video sequence, or it may be configured to vary the block size between pictures or subsets or groups of pictures and to partition each picture into a corresponding block. The abbreviation AVC stands for Advanced Video Coding.

[0380] In a further embodiment, the encoder 20 may be configured to directly receive blocks 203 of the picture 17, for example, one, some, or all of the blocks that make up the picture 17. The picture blocks 203 may also be called the current picture block or the picture block to be coded.

[0381] Like picture 17, picture block 203 is, in this case as well, a two-dimensional array or matrix of samples having intensity values ​​(sample values), but smaller in dimensions than picture 17, or can be considered as such. In other words, block 203 may contain, for example, one sample array (e.g., a lumen array in the case of monochrome picture 17, or a lumen or chromen array in the case of a color picture) or three sample arrays (e.g., a lumen and two chromen arrays in the case of color picture 17), or any other number and / or type of arrays depending on the color format applied. The number of samples in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Thus, the block may be, for example, an M×N (M columns × N rows) array of samples, or an M×N array of conversion coefficients.

[0382] An embodiment of the encoder 20, as shown in Figure 20, may be configured to encode the picture 17 block by block, for example, encoding and prediction may be performed in units of block 203.

[0383] Embodiments of the encoder 20, as shown in Figure 20, may be further configured to divide and / or encode a picture using slices (also called video slices), where the picture may be divided into one or more slices (usually non-overlapping) or encoded using such slices, each slice may contain one or more blocks (e.g., CTUs).

[0384] Embodiments of the encoder 20, as shown in Figure 20, may be further configured to divide and / or encode a picture using tile groups (also called video tile groups) and / or tiles (also called video tiles), wherein the picture may be divided into one or more tile groups (usually non-overlapping) or encoded using such tile groups, each tile group may, for example, include one or more blocks (e.g., CTUs) or one or more tiles, where each tile may, for example, be rectangular in shape and include one or more blocks (e.g., CTUs), for example, complete or fragmented blocks.

[0385] Figure 21 shows an example of a decoder 30 configured to implement the technique of the present application. The decoder 30 is configured to receive encoded picture data 21 (e.g., encoded bitstream 21), encoded by, for example, the encoder 20, in order to obtain a decoded picture 331. The encoded picture data or bitstream includes information for decoding the encoded picture data, such as data representing picture blocks and associated syntax elements of an encoded slice (and / or tile group or tile or subpicture).

[0386] The entropy decoding unit 304 is configured to parse the bitstream 21 (or, generally, the encoded picture data 21) and, for example, perform entropy decoding to the encoded picture data 21 to obtain, for example, one or all of the following: quantization coefficients 309 and / or decoding coding parameters (not shown in Figure 21), for example, inter-prediction parameters (e.g., reference picture index and motion vector), intra-prediction parameters (e.g., intra-prediction mode or index), transformation parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 may be configured to apply a decoding algorithm or scheme corresponding to an encoding scheme such as the one described with respect to the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 may be further configured to provide the inter-prediction parameters, intra-prediction parameters, and / or other syntax elements to the mode application unit 360, and other parameters to other units of the decoder 30. The decoder 30 may receive syntax elements at the video slice level and / or video block level. In addition to slices and their respective syntax elements, or as an alternative thereto, tile groups and / or tiles and their respective syntax elements may be received and / or used. Entropy decoding may implement any of the arithmetic decoding methods or apparatus described above.

[0387] The reconstruction unit 314 (for example, an adder or summer 314) may be configured to obtain the reconstructed block 315 in the sample region by adding the sample values ​​of the reconstructed residual block 313 to the predicted block 365, for example, by adding the sample values ​​of the predicted block 365 to the reconstructed residual block 313.

[0388] Embodiments of the decoder 30, as shown in Figure 21, may be configured to divide and / or decode a picture using slices (also called video slices), where the picture may be divided into one or more slices (usually non-overlapping) or decoded using such slices, each slice may contain one or more blocks (e.g., CTUs).

[0389] Embodiments of the decoder 30, as shown in Figure 21, may be configured to segment and / or decode a picture using tile groups (also called video tile groups) and / or tiles (also called video tiles), wherein the picture may be segmented into one or more tile groups (usually non-overlapping) or decoded using such tile groups, each tile group may comprise, for example, one or more blocks (e.g., CTUs) or one or more tiles, where each tile may be, for example, rectangular in shape and comprise one or more blocks (e.g., CTUs), for example, complete or fragmented blocks.

[0390] Other variations of the decoder 30 may be used to decode the encoded picture data 21. For example, the decoder 30 can generate an output video stream without using the loop filtering unit 320. For example, an unconverted decoder 30 can directly dequantize the residual signal for some blocks or frames without using the inverse conversion unit 312. In another implementation, the decoder 30 can combine the inverse quantization unit 310 and the inverse conversion unit 312 into a single unit.

[0391] It should be understood that within the encoder 20 and decoder 30, the processing result of the current step may be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, further operations such as clipping or shifting may be performed on the processing result of the interpolation filtering, motion vector derivation, or loop filtering.

[0392] Figure 26 is a block diagram showing a content supply system 3100 for realizing a content distribution service. This content supply system 3100 includes a capture device 3102, a terminal device 3106, and optionally a display 3126. The capture device 3102 communicates with the terminal device 3106 via a communication link 3104. The communication link may include the communication channel 13 described above. The communication link 3104 may include, but is not limited to, Wi-Fi, Ethernet, cable, wireless (3G / 4G / 5G), USB, or any combination thereof.

[0393] The capture device 3102 generates data and may encode the data using the encoding method shown in the above embodiment. Alternatively, the capture device 3102 may distribute the data to a streaming server (not shown in the figure), which encodes the data and transmits the encoded data to the terminal device 3106. The capture device 3102 includes, but is not limited to, a camera, a smartphone or tablet, a computer or laptop, a video conferencing system, a PDA, an in-vehicle device, or any combination thereof. For example, the capture device 3102 may include the source device 12 as described above. When the data includes video, the video encoder 20 included in the capture device 3102 may actually perform the video encoding process. When the data includes audio (i.e., voice), the audio encoder included in the capture device 3102 may actually perform the audio encoding process. In some practical scenarios, the capture device 3102 distributes the encoded video and audio data by multiplexing them together. In other practical scenarios, for example, in a video conferencing system, the encoded audio data and encoded video data are not multiplexed. The capture device 3102 separately distributes encoded audio data and encoded video data to the terminal device 3106.

[0394] In the content supply system 3100, the terminal device 310 receives and plays back encoded data. The terminal device 3106 may be a device with data receiving and recovery capabilities, such as a smartphone or tablet 3108, a computer or laptop 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a TV 3114, a set-top box (STB) 3116, a video conferencing system 3118, a video surveillance system 3120, a personal digital assistant (PDA) 3122, an in-vehicle device 3124, or any combination of these capable of decoding the encoded data described above. For example, the terminal device 3106 may include the destination device 14 as described above. When the encoded data includes video, the video decoder 30 included in the terminal device is preferred for performing video decoding. When the encoded data includes audio, the audio decoder included in the terminal device is preferred for performing audio decoding.

[0395] In the case of terminal devices having a display, such as a smartphone or tablet 3108, a computer or laptop 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a TV 3114, a personal digital assistant (PDA) 3122, or an in-vehicle device 3124, the terminal device can supply the decoded data to its display. In the case of terminal devices not equipped with a display, such as an STB 3116, a video conferencing system 3118, or a video surveillance system 3120, an external display 3126 is made contact thereto to receive and present the decoded data.

[0396] When each device in this system performs encoding or decoding, a picture encoding device or picture decoding device as shown in the embodiments described above may be used.

[0397] Figure 27 shows the structure of an example terminal device 3106. After the terminal device 3106 receives a stream from the capture device 3102, the protocol progression unit 3202 analyzes the transmission protocol of the stream. The protocol may include, but is not limited to, Real-Time Streaming Protocol (RTSP), Hypertext Transfer Protocol (HTTP), HTTP Live Streaming Protocol (HLS), MPEG-DASH, Real-Time Transport Protocol (RTP), Real-Time Messaging Protocol (RTMP), or any combination thereof.

[0398] After the protocol processing unit 3202 processes the stream, a stream file is generated. The file is output to the demultiplexing unit 3204. The demultiplexing unit 3204 can separate the multiplexed data into encoded audio data and encoded video data. As described above, in some real-world scenarios, for example in a video conferencing system, the encoded audio data and encoded video data are not multiplexed. In this situation, the encoded data is sent to the video decoder 3206 and audio decoder 3208 without passing through the demultiplexing unit 3204.

[0399] Through demultiplexing processing, a video base stream (ES), an audio ES, and optionally a subtitle are generated. The video decoder 3206, including the video decoder 30 as described in the above embodiment, decodes the video ES by the decoding method shown in the above embodiment to generate video frames and supplies this data to the synchronization unit 3212. The audio decoder 3208 decodes the audio ES to generate audio frames and supplies this data to the synchronization unit 3212. Alternatively, video frames may be stored in a buffer (not shown in Figure 27) before supplying them to the synchronization unit 3212. Similarly, audio frames may be stored in a buffer (not shown in Figure 27) before supplying them to the synchronization unit 3212.

[0400] The synchronization unit 3212 synchronizes video frames and audio frames and supplies video / audio to the video / audio display 3214. For example, the synchronization unit 3212 synchronizes the presentation of video with audio information. The information may be coded in the syntax using timestamps for the presentation of coded audio and visual data, as well as timestamps for the delivery of the data stream itself.

[0401] If the stream contains subtitles, the subtitle decoder 3210 decodes the subtitles, synchronizes them with the video and audio frames, and supplies the video / audio / subtitle to the video / audio / subtitle display 3216.

[0402] A coder is provided that includes a processing circuit for performing a method according to one of the proposed methods.

[0403] A storage medium is provided which stores a bitstream obtained by using one of the proposed methods.

[0404] A bitstream is provided containing coded image data and a first coding parameter, the first coding parameter representing a first displacement between a first target rate control parameter and a first reference rate control parameter.

[0405] In possible designs, the first coding parameter is represented by a fixed-point value.

[0406] In a possible design, the first coding parameter is signaled using N bits, where N is an integer less than or equal to 16.

[0407] In a possible design, the first coding parameter is represented by a fixed-point value having N bits.

[0408] In a possible design, the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are for coding the rumor data of the image.

[0409] In a possible design, the bitstream further includes a second coding parameter, which represents a second displacement between a second target rate control parameter and a second reference rate control parameter. The second coding parameter, the second target rate control parameter, and the second reference rate control parameter are for coding the chroma data of the image.

[0410] In a possible design, the bitstream would further include flags indicating whether different rate control parameters are used to code the rumor and chroma data of the image.

[0411] In possible designs, the second coding parameter is represented by a fixed-point value.

[0412] In a possible design, the bitstream further includes a third coding parameter, which indicates the index of the trained model related to the first and / or second reference rate control parameters.

[0413] Note that all remaining aspects of the encoding or decoding method are also related to the bitstream structure.

[0414] A system for distributing bitstreams is provided, and the system is As described above, at least one storage medium configured to store at least one bitstream, The system comprises a video streaming device configured to acquire a bitstream from at least one storage medium and send the bitstream to a terminal device, the video streaming device comprising a content server or content distribution server.

[0415] In a possible design, the system will One or more processors configured to perform an encryption process on at least one bitstream and obtain at least one encrypted bitstream, At least one storage medium configured to store an encrypted bitstream, or One or more processors configured to convert a bitstream from at least one bitstream of a first format to a bitstream of a second format. The system further comprises at least one storage medium configured to store a bitstream of a second format.

[0416] In a possible design, the system will A receiver configured to receive a first operation request, One or more processors configured to determine a target bitstream in at least one storage medium in response to a first operation request, It further comprises a transmitter configured to send a target bitstream to a terminal device.

[0417] In a possible design, one or more processors, Further configured to encapsulate the bitstream and obtain a transport stream in a third format, The transmitter is Send a transport stream in a third format to the terminal device for display, or The transport stream of the third format is further configured to send it to memory space for storage.

[0418] The present invention is not limited to the system described above, and either the picture encoding device or the picture decoding device in the above embodiments may be incorporated into other systems, such as automotive systems.

[0419] For example, and without limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. Also, any connection is appropriately called a computer-readable medium. For example, if instructions are transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but instead refer to non-temporary tangible storage media. As used herein, the terms "disk" and "disc" include compact discs (CDs), laser discs, optical discs, digital multipurpose discs (DVDs), floppy disks, and Blu-ray® discs, where a disc typically reproduces data magnetically and a disc reproduces data optically using a laser. Combinations of the above should also be included within the scope of computer-readable media.

[0420] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or separate logic circuits. Thus, the term “processor” as used herein may refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. In addition, in some embodiments, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a composite codec. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

[0421] The techniques of this disclosure may be implemented in a wide variety of devices or apparatus, including wireless handsets, cloud servers, application servers, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to highlight the functional aspects of a device configured to perform the techniques disclosed, but implementation by different hardware units is not necessarily required. Rather, as described above, various units may be combined within a codec hardware unit, or provided by a set of interoperable hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware. [Explanation of symbols]

[0422] 10 Video Coding Systems 12 Source Devices 13. Encoded picture data, communication channel 14 Destination device 16 Picture Sources 17. Picture data, unprocessed picture data, monochrome picture, color picture 18. Preprocessor, pre-processing unit 19 Preprocessed picture data 20 Video Encoders 21 Encoded picture data, Encoded bitstream 22 Communication interface, communication unit 28 Communication interface, communication unit 30 video decoders 31 Decoded picture data 32 Post-processors, Post-processing Units 33 Post-processed picture data 34 Display Devices 46 Processing Circuit 101 Encoder 102 Quantizer 103 Hyper Encoder / Hyper Decoder, Hyper Plier 104 Decoder 105 Arithmetic coding module, arithmetic encoder 106 Arithmetic Decoding Module, Arithmetic Decoder 121 encoders 122 Quantizer 123 Hyper Encoder 125 Arithmetic Coding Module 127 Hyper Decoder 147 Hyper Decoder 201 Input section, input interface 203 Picture Block 204 Residual Calculation Unit 206 Conversion Processing Unit 208 Quantization Units 210 Encoder side, inverse quantization unit 212 Inverse Transform Processing Unit 214 Reconfiguration Unit 220 encoder subnetworks, loop filter units 230 Representation, encoding, coding, decoded picture buffer (DPB) 244 Interpretation Units 250 Decoder side 254 Intra Prediction Units 260 Decoder Subnetwork, Mode Selection Unit 262 division units 270 Entropy Coding Units 272 Output section, output interface 304 Entropy Decoding Unit 309 Quantization coefficient 310 Inverse Quantization Unit 312 Inverse Transform Processing Unit 313 Reconstructed residual blocks 314 Reconfiguration Unit, Adder 315 Reconstruction Block 320 Loop Filtering Unit 322 Quantizer 331 Decrypted Picture 360 Mode Applicable Unit 365 Prediction Block 401-403 Convolutional Layers 407-412 Upsampling Layers 413 Output signal 414 Input Images 430 Convolutional Layers 510 Mobile side 520 Quantization layer 550 Hidden Layers, Deep Feature Maps 560 Inverse quantization layer 590 Cloud side 610 Input Data Channels 620 parallel encoders 630-633 coded bits 640~643 Trailing Bit 650 Substream length 1500 encoding devices 1510 units acquired 1520 encoding unit 1600 decryption devices 1610 Receiving Unit 1620 Syntax Analysis Unit 1630 Reconfiguration Unit 3100 Content Supply System 3102 Capture Device 3104 Communication Link 3106 Terminal device 3108 Smartphones, Tablets 3110 Computers, Laptops 3112 Network Video Recorder (NVR) / Digital Video Recorder (DVR) 3114 TV 3116 Set-top box (STB) 3118 Video conferencing system 3120 Video Surveillance System 3122 Personal Digital Assistant (PDA) 3124 In-vehicle devices 3126 External Display 3202 Protocol Progress Unit 3204 Demultiplexing Unit 3206 Video Decoder 3208 Audio Decoder 3210 Subtitle Decoder 3212 Synchronization Unit 3214 Video / Audio Display 3216 Video / Audio / Subtitle Display 8000 video coding devices 8010 Inlet port, Input port 8020 Receiver Unit (Rx) 8030 Processor, Logical Unit, Central Processing Unit (CPU) 8040 Transmitter Unit (Tx) 8050 exit port, output port 8060 memory 8070 Neural Network-Based Codec 9000 equipment 9002 Processor 9004 memory 9006 Data 9008 Operating System 9010 Application Program 9012 Bus 9018 Display 10,000 video coding systems 10002 Platform 10004 Content Delivery Network (CDN) 10006 terminal

Claims

1. A method for encoding an image (1300), The steps include acquiring the aforementioned image (1301), A step (1302) of encoding the image into a bitstream based on a first coding parameter, wherein the first coding parameter represents a first displacement between a first target rate control parameter and a first reference rate control parameter, The steps (1303) include encoding the first coding parameter into the bitstream and A method that includes this.

2. The method according to claim 1 (1300), wherein the first coding parameter is represented by a fixed-point value.

3. The method according to claim 1 or 2 (1300), wherein the first coding parameter is signaled using N bits, where N is an integer less than or equal to 16.

4. The method according to claim 3 (1300), wherein N is equal to 12.

5. The method according to any one of claims 1 to 4 (1300), wherein the first displacement is one of a ratio and a difference.

6. The method according to any one of claims 1 to 5 (1300), wherein the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are for coding the rumor data of the image.

7. The method according to any one of claims 1 to 5 (1300), wherein the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are for coding the first-order component of the image.

8. The method according to claim 7 (1300), wherein the primary component is a luma component.

9. The above method (1300) The process further includes the step of encoding a second coding parameter into the bitstream, wherein the second coding parameter represents a second displacement between a second target rate control parameter and a second reference rate control parameter. The method according to claim 6 (1300), wherein the second coding parameter, the second target rate control parameter, and the second reference rate control parameter are for coding the chroma data of the image.

10. The above method (1300) The process further includes the step of encoding a second coding parameter into the bitstream, wherein the second coding parameter represents a second displacement between a second target rate control parameter and a second reference rate control parameter. The method according to claim 7 or 8 (1300), wherein the second coding parameter, the second target rate control parameter, and the second reference rate control parameter are for coding the quadratic components of the image.

11. The method according to claim 10 (1300), wherein the secondary component is a chromatic component.

12. The method according to claim 9 (1300), wherein the encoding of the second coding parameter is based on a flag indicating that different rate control parameters are used to code the rumor data and chroma data of the image.

13. The method according to claim 10 or 11 (1300), wherein the encoding of the second coding parameter is based on a flag indicating that different rate control parameters are used to code the primary and secondary components of the image.

14. The method according to any one of claims 9 to 13 (1300), wherein the second displacement is one of a ratio and a difference.

15. The method according to any one of claims 9 to 14 (1300), wherein the second coding parameter is represented by a fixed-point value.

16. The above method (1300) The method according to any one of claims 1 to 15 (1300), further comprising the step of encoding a third coding parameter into the bitstream.

17. The method according to claim 16 (1300), wherein the third coding parameter indicates an index of a trained model associated with at least one of the first reference rate control parameter and the second reference rate control parameter.

18. At least one of the first coding parameter and the second coding parameter is the following code, i.e., - Binary code, - Unary code, - Abbreviated unary codes, and - Exponential Golomb Code The method according to any one of claims 1 to 17 (1300), which is signaled using one or more of the following.

19. The method according to any one of claims 1 to 18 (1300), wherein the first coding parameter is on a logarithmic scale.

20. The method according to any one of claims 9 to 18 (1300), wherein at least one of the first coding parameter, the second coding parameter, the first target rate control parameter, the second target rate control parameter, the first reference rate control parameter, and the second reference rate control parameter is on a logarithmic scale.

21. The step (1302) of encoding the first coding parameter into the bitstream is, The steps include obtaining the adjusted value of the first coding parameter, The steps include: encoding the adjusted value of the first coding parameter into the bitstream; The method according to any one of claims 1 to 20, including (1300).

22. The adjusted value of the first coding parameter is 2 N-1 The method according to claim 21 (1300), obtained by adding, where N is the number of bits used to signal the first coding parameter.

23. The step (1302) of encoding the image into a bitstream based on the first coding parameter is, A step of obtaining a target gain vector based on the first coding parameter, The steps of encoding the image based on the target gain vector and The method according to any one of claims 19 to 22, including (1300).

24. The target gain vector is as follows, that is, m log =m t log +betaDisplacementLog It is defined as, m log The above-mentioned target gain vector is shown, m t log The method according to claim 23 (1300), wherein is a reference forward gain vector for a model having index t, where t is an index of a trained model related to at least one of the first reference rate control parameter and the second reference rate control parameter, and betaDisplacementLog is related to the first coding parameter.

25. The method (1300) according to any one of claims 1 to 24, wherein the first target rate control parameter is a rate control parameter selected by an encoder performing the method (1300), and the first reference rate control parameter is a rate control parameter used in model training.

26. The method (1300) according to any one of claims 9 to 25, wherein the second target rate control parameter is a rate control parameter selected by an encoder performing the method (1300), and the second reference rate control parameter is a rate control parameter used in model training.

27. A method (1400) for decoding an image, The steps include receiving a bitstream containing coded data of the aforementioned image (1401), A step (1402) of parsing the bitstream to obtain a first coding parameter, wherein the first coding parameter represents a first displacement between a first target rate control parameter and a first reference rate control parameter. The steps of (1403) and reconstructing the image based on the first coding parameters A method that includes this.

28. The method according to claim 27 (1400), wherein the first coding parameter is represented by a fixed-point value.

29. The method according to claim 27 or 28 (1400), wherein the first coding parameter is signaled using N bits, where N is an integer less than or equal to 16.

30. The method according to claim 29 (1400), wherein N is equal to 12.

31. The method according to any one of claims 27 to 30 (1400), wherein the first displacement is one of a ratio and a difference.

32. The method according to any one of claims 27 to 31 (1400), wherein the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are for coding the rumor data of the image.

33. The method according to any one of claims 22 to 31 (1400), wherein the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are for coding the first-order component of the image.

34. The method according to claim 33, wherein the primary component is a luma component (1400).

35. The above method (1400) The process further includes the step of parsing the bitstream to obtain a second coding parameter, wherein the second coding parameter represents a second displacement between a second target rate control parameter and a second reference rate control parameter. The method according to any one of claims 27 to 34 (1400), wherein the second coding parameter, the second target rate control parameter, and the second reference rate control parameter are for coding the chroma data of the image.

36. The above method (1400) The process further includes the step of parsing the bitstream to obtain a second coding parameter, wherein the second coding parameter represents a second displacement between a second target rate control parameter and a second reference rate control parameter. The method according to any one of claims 22 to 34 (1400), wherein the second coding parameter, the second target rate control parameter, and the second reference rate control parameter are for coding the quadratic components of the image.

37. The method according to claim 36 (1400), wherein the secondary component is a chromatic component.

38. The method according to claim 35 (1400), wherein the acquisition of the second coding parameter is based on a flag indicating that different rate control parameters are used to code the rumor data and chroma data of the image.

39. The method according to claim 36 or 37 (1400), wherein the acquisition of the second coding parameter is based on a flag indicating that different rate control parameters are used to code the primary and secondary components of the image.

40. The method according to any one of claims 35 to 39 (1400), wherein the second displacement is one of a ratio and a difference.

41. The method according to any one of claims 35 to 40 (1400), wherein the second coding parameter is represented by a fixed-point value.

42. The above method (1400) The method according to any one of claims 27 to 41 (1400), further comprising the step of parsing the bitstream to obtain a third coding parameter.

43. The method according to claim 42 (1400), wherein the third coding parameter indicates an index of a trained model associated with at least one of the first reference rate control parameter and the second reference rate control parameter.

44. At least one of the first coding parameter and the second coding parameter is the following code, i.e., - Binary code, - Unary code, - Abbreviated unary codes, and - Exponential Golomb Code The method according to any one of claims 27 to 43 (1400), which is signaled using one or more of the following.

45. The method according to any one of claims 27 to 44 (1400), wherein the first coding parameter is on a logarithmic scale.

46. The method according to any one of claims 35 to 44 (1400), wherein at least one of the first coding parameter, the second coding parameter, the first target rate control parameter, the second target rate control parameter, the first reference rate control parameter, and the second reference rate control parameter is on a logarithmic scale.

47. The step (1403) of reconstructing the image based on the first coding parameter is, From the first coding parameter mentioned above, 2 N-1 A step of obtaining an intermediate parameter by subtracting, wherein N is the number of bits used to signal the first coding parameter, The step of reconstructing the image based on the intermediate parameters. The method according to any one of claims 27 to 46, including (1400).

48. The step (1403) of reconstructing the image based on the first coding parameter is, A step of obtaining a target gain vector based on the first coding parameter, The steps of reconstructing the image based on the target gain vector and The method according to claim 45, or claim 46 in combination with claim 45, or claim 47 in combination with claim 45 (1400).

49. The target gain vector is as follows, that is, m log =m t log +betaDisplacementLog It is defined as, m log The above-mentioned target gain vector is shown, m t log The method according to claim 48 (1400), wherein is a reference forward gain vector for a model having index t, where t is an index of a trained model relating to at least one of the first reference rate control parameter and the second reference rate control parameter, and betaDisplacementLog is relating to the first coding parameter.

50. The method according to any one of claims 27 to 49 (1400), wherein the first target rate control parameter is a rate control parameter selected by an encoder, and the first reference rate control parameter is a rate control parameter used in model training.

51. The method according to any one of claims 35 to 49 (1400), wherein the second target rate control parameter is a rate control parameter selected by the encoder, and the second reference rate control parameter is a rate control parameter used in model training.

52. An encoding device for encoding images, Memory containing instructions, The processor coupled to the aforementioned memory and An encoding device comprising the processor, wherein the processor is configured to execute the instructions to cause the encoding device to perform the encoding method (1300) according to any one of claims 1 to 26.

53. A decoding device for decoding images, Memory containing instructions, The processor coupled to the aforementioned memory and A decoding device comprising the processor, wherein the processor is configured to execute the instructions to cause the decoding device to perform the decoding method (1400) according to any one of claims 27 to 51.

54. A coding device for coding images, A receiver unit configured to receive an image to be encoded or a bitstream to be decoded, A transmitter unit coupled to the receiver unit, configured to transmit the bitstream to a decoder or to transmit the decoded image to a display, A memory connected to at least one of the receiver unit or the transmitter unit, the memory being configured to store instructions, The processor coupled to the aforementioned memory and A coding device comprising, wherein the processor is configured to execute the instructions stored in the memory to perform the method according to any one of claims 1 to 51 (1300, 1400).

55. An encoding device (1500) for encoding an image, An acquisition unit (1510) configured to acquire the aforementioned image, An encoding unit (1520) configured to encode the image into a bitstream based on a first coding parameter and The system comprises a first coding parameter that indicates a first displacement between a first target rate control parameter and a first reference rate control parameter. An encoding device (1500) wherein the encoding unit (1520) is further configured to encode the first coding parameters into the bitstream.

56. An encoding device (1500) according to claim 55, further configured to perform the method (1300) described in any one of claims 1 to 26.

57. A decoding device (1600) for decoding an image, A receiving unit (1610) configured to receive a bitstream containing coded data of the aforementioned image, A parsing unit (1620) configured to parse the bitstream and obtain a first coding parameter, wherein the first coding parameter indicates a first displacement between a first target rate control parameter and a first reference rate control parameter, A reconstruction unit (1630) configured to reconstruct the image based on the first coding parameter and A decoding device (1600) equipped with the following.

58. A decoding device (1600) according to claim 57, further configured to perform the method (1400) described in any one of claims 27 to 51.

59. A coding system for coding images, Encoder and The decoder that is communicating with the encoder and A coding system comprising the encoder or decoder including a decoding device according to claim 53 or any one of claims 57 and 58, an encoding device according to claim 52 or any one of claims 55 and 56, or a coding device according to claim 54.

60. A computer program product comprising program code for performing the method (1300, 1400) according to any one of claims 1 to 51 when executed on one or more processors.

61. A computer-readable storage medium that stores a computer program that can be executed by one or more processors, wherein when the computer program is executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 51 (1300, 1400).

62. A coder comprising a processing circuit for performing the method (1300, 1400) according to any one of claims 1 to 51.

63. A storage medium for storing a bitstream obtained by using the method (1300) described in any one of claims 1 to 26.

64. A bitstream comprising coded data of an image and a first coding parameter, wherein the first coding parameter indicates a first displacement between a first target rate control parameter and a first reference rate control parameter.

65. The bitstream according to claim 64, wherein the first coding parameter is represented by a fixed-point value.

66. The bitstream according to claim 64 or 65, wherein the first coding parameter is signaled using N bits, where N is an integer less than or equal to 16.

67. The bitstream according to claim 66, wherein N is equal to 12.

68. The bitstream according to any one of claims 64 to 67, wherein the first displacement is one of a ratio and a difference.

69. The bitstream according to any one of claims 64 to 68, wherein the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are for coding the rumor data of the image.

70. The bitstream according to any one of claims 64 to 68, wherein the first coding parameter, the first target rate control parameter, and the first reference rate control parameter are for coding the first-order component of the image.

71. The bitstream according to claim 70, wherein the primary component is a luma component.

72. The bitstream further includes a second coding parameter, the second coding parameter representing a second displacement between a second target rate control parameter and a second reference rate control parameter. The bitstream according to any one of claims 64 to 71, wherein the second coding parameter, the second target rate control parameter, and the second reference rate control parameter are for coding the chroma data of the image.

73. The bitstream further includes a second coding parameter, the second coding parameter representing a second displacement between a second target rate control parameter and a second reference rate control parameter. The bitstream according to any one of claims 64 to 71, wherein the second coding parameter, the second target rate control parameter, and the second reference rate control parameter are for coding the quadratic components of the image.

74. The bitstream according to claim 73, wherein the secondary component is a chroma component.

75. The bitstream according to claim 72, further comprising a flag indicating whether the bitstream uses different rate control parameters to code the rumen data and chroma data of the image.

76. The bitstream according to claim 73 or 74, further comprising a flag indicating whether the bitstream uses different rate control parameters to code the primary and secondary components of the image.

77. The bitstream according to any one of claims 72 to 76, wherein the second displacement is one of a ratio and a difference.

78. The bitstream according to any one of claims 72 to 77, wherein the second coding parameter is represented by a fixed-point value.

79. The bitstream according to any one of claims 64 to 78, further comprising a third coding parameter, the third coding parameter indicating an index of a trained model related to at least one of the first reference rate control parameter and the second reference rate control parameter.

80. A system for distributing bitstreams, A storage medium configured to store at least one bitstream according to any one of claims 64 to 79, or the bitstream generated by the method (1300) according to any one of claims 1 to 26, A video streaming device configured to acquire a bitstream from one of the at least one storage media and send the bitstream to a terminal device. A system comprising a video streaming device that includes a content server or a content distribution server.

81. The aforementioned system a) further comprising one or more processors configured to perform an encryption process on the at least one bitstream to obtain at least one encrypted bitstream, wherein the at least one storage medium is configured to store the encrypted bitstream, or b) The system according to claim 80, further comprising one or more processors configured to convert a bitstream from at least one encrypted bitstream of a first format to a bitstream of a second format, wherein at least one storage medium is configured to store the bitstream of the second format.

82. The aforementioned system A receiver configured to receive a first operation request, A transmitter configured to send a target bitstream to a terminal device and Furthermore, The system according to claim 80 or 81, wherein one or more processors are configured to determine a target bitstream in the at least one storage medium in response to the first operation request.

83. The one or more processors described above Further configured to encapsulate the bitstream and obtain a transport stream in a third format, The aforementioned transmitter, Send the transport stream in the third format to the terminal device for display, or The system according to claim 82, further configured to send the transport stream of the third format to a storage space for storage.

Images