Encoding method and apparatus, decoding method and apparatus, device, storage medium, and computer program

Abstract

The embodiments of the present application disclose an encoding method and apparatus, a decoding method and apparatus, a device, a storage medium, and a computer program, which relate to the field of encoding and decoding technology. In this method, a first latent variable is scaled based on a first variable scale factor to obtain a second latent variable, and the number of coding bits of the entropy coding result of the second latent variable meets a preset encoding rate condition. This ensures that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of media data can meet the preset encoding rate condition, that is, the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of media data can basically keep consistency instead of changing dynamically, thereby meeting the encoder's requirement for a stable encoding rate.

Description

[Technical field]

[0001] This application claims priority to Chinese Patent Application No. 202110559102.7, entitled “ENCODING METHOD AND APPARATUS, DECODING METHOD AND APPARATUS, DEVICE, STORAGE MEDIUM, AND COMPUTER PROGRAM,” filed on May 21, 2021, the entire contents of which are incorporated herein by reference.

[0002] FIELD OF THE DISCLOSURE Embodiments of the present application relate to the field of encoding and decoding technologies, and in particular to an encoding method and apparatus, a decoding method and apparatus, a device, a storage medium, and a computer program. [Background technology]

[0003] Encoding and decoding technology is essential in media applications, such as media communication and media broadcasting, etc. Therefore, how to implement encoding and decoding has become one of the industry concerns.

[0004] The related art proposes a method for encoding and decoding an audio signal. In this method, when the audio signal is encoded, a modified discrete cosine transform (MDCT) process is performed on the time-domain audio signal to obtain a frequency-domain audio signal. The frequency-domain audio signal is processed by using an encoding neural network model to obtain latent variables. The latent variables indicate characteristics of the frequency-domain audio signal. A quantization process is performed on the latent variables to obtain quantized latent variables, an entropy coding is performed on the quantized latent variables, and the entropy coding result is written into a bitstream. When the audio signal is decoded, the quantized latent variables are determined based on the bitstream, and an inverse quantization process is performed on the quantized latent variables to obtain latent variables. The latent variables are processed by using a decoding neural network model to obtain a frequency-domain audio signal, and an inverse modified discrete cosine transform (IMDCT) process is performed on the frequency-domain audio signal to obtain a reconstructed time-domain audio signal.

[0005] However, entropy coding is to encode elements with different probabilities by using different numbers of bits. Therefore, for two adjacent frames of an audio signal, the probability of occurrence of elements in the latent variables corresponding to the two frames of the audio signal may be different. As a result, the number of coding bits of the latent variables of the two frames of the audio signal is different, and the requirement of a stable encoding rate cannot be met. Summary of the Invention [Means for solving the problem]

[0006] The embodiments of the present application provide an encoding method and apparatus, a decoding method and apparatus, a device, a storage medium, and a computer program to meet the encoder's requirements for a stable encoding rate. The technical solutions are as follows:

[0007] According to a first aspect, an encoding method is provided. The method may be applied to a codec that does not include a context model, or may be applied to a codec that includes a context model. In addition, not only may the latent variables generated by using the media data to be encoded be scaled based on a scale factor, but also the latent variables determined by using the context model may be scaled based on a scale factor. Therefore, in the following, the method is explained and described in detail in several cases.

[0008] In a first case, the method includes the steps of: processing media data to be encoded by using a first encoding neural network model to obtain a first latent variable, where the first latent variable indicates a characteristic of the media data to be encoded; determining a first variable scale factor based on the first latent variable, where the first variable scale factor is used to enable a number of coding bits of an entropy coding result of a second latent variable to satisfy a preset encoding rate condition, where the second latent variable is obtained by scaling the first latent variable based on the first variable scale factor; obtaining an entropy coding result of the second latent variable; and writing the entropy coding result of the second latent variable and the encoding result of the first variable scale factor into a bitstream.

[0009] The number of coding bits of the entropy coding result of the second latent variable satisfies a preset encoding rate condition. This ensures that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can satisfy the preset encoding rate condition, i.e., the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can be basically kept consistent instead of dynamically changing, thereby meeting the encoder's requirement for a stable encoding rate. Furthermore, when side information (e.g., a window type, a Temporal Noise Shaping (TNS) parameter, a Frequency-domain noise shaping (FDNS) parameter, and / or a bandwidth extension (BWE) parameter) needs to be transmitted, it can be ensured that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data is basically consistent with the number of coding bits of the side information, thereby meeting the encoder's requirement for a stable encoding rate.

[0010] The media data to be encoded may be an audio signal, a video signal, an image, etc. In addition, the media data to be encoded may be in any form, which is not limited in this embodiment of the present application.

[0011] An implementation process of processing media data to be encoded by using a first encoding neural network model is as follows: inputting the media data to be encoded into the first encoding neural network model to obtain a first latent variable output by the first encoding neural network model; or pre-processing the media data to be encoded and inputting the pre-processed media data into the first encoding neural network model to obtain a first latent variable output by the first encoding neural network model.

[0012] In other words, the media data to be encoded may be used as an input of the first encoding neural network model to determine the first latent variables, or the media data to be encoded may be pre-processed and then used as an input of the first encoding neural network model to determine the first latent variables.

[0013] Optionally, when the media data to be encoded is encoded at a constant bit rate, satisfying the preset encoding rate condition includes the number of coding bits being smaller than or equal to a target number of coding bits, satisfying the preset encoding rate condition includes the number of coding bits being smaller than or equal to the target number of coding bits and a difference between the number of coding bits and the target number of coding bits being smaller than a bit number threshold, or satisfying the preset encoding rate condition includes the number of coding bits being smaller than or equal to a maximum number of coding bits of the target number of coding bits.

[0014] Optionally, when the media data to be encoded is encoded at a variable bit rate, satisfying the preset encoding rate condition includes: an absolute value of a difference between the number of coding bits and the target number of coding bits is smaller than a bit number threshold. In other words, satisfying the preset encoding rate condition includes: the number of coding bits is smaller than or equal to the target number of coding bits, and the difference between the target number of coding bits and the number of coding bits is smaller than a bit number threshold; or satisfying the preset encoding rate condition includes: the number of coding bits is larger than or equal to the target number of coding bits, and the difference between the number of coding bits and the target number of coding bits is smaller than a bit number threshold.

[0015] The target number of coding bits may be preset. Of course, the target number of coding bits may also be determined based on the encoding rate, with different encoding rates corresponding to different target numbers of coding bits.

[0016] In this embodiment of the present application, the media data to be encoded may be encoded at a constant bit rate, or the media data to be encoded may be encoded at a variable bit rate.

[0017] When the media data to be encoded is encoded at a constant bit rate, the number of bits of the media data to be encoded of a current frame may be determined based on the constant bit rate, and then the number of used bits of the current frame is subtracted to obtain a target number of coding bits of the current frame. The number of used bits may be the number of bits for encoding side information, etc. In addition, the side information of each frame of media data is generally different. Therefore, the target number of coding bits of each frame of media data is usually different.

[0018] When the media data to be encoded is encoded at a variable bit rate, the bit rate is usually specified, and the actual bit rate fluctuates around the specified bit rate. In this case, the number of bits of the media data to be encoded of the current frame may be determined based on the specified bit rate, and then the number of used bits of the current frame is subtracted to obtain a target number of coding bits for the current frame. The number of used bits may be the number of bits for encoding side information, etc. In some cases, the side information of different frames of media data may be different. Thus, the target number of coding bits of different frames of media data are usually different.

[0019] An initial number of coding bits may be determined based on the first latent variable, and a first variable scale factor may be determined based on the initial number of coding bits and a target number of coding bits.

[0020] The initial number of coding bits is the number of coding bits of the entropy coding result of the first latent variable, or the initial number of coding bits is the number of coding bits of the entropy coding result of the first latent variable scaled based on a first initial scale factor. The first initial scale factor may be a first preset scale factor.

[0021] An implementation process for scaling the first latent variable based on the first initial scale factor is to multiply each element in the first latent variable by a corresponding element in the first initial scale factor to obtain a scaled first latent variable as follows:

[0022] It should be noted that the above implementation process is just an example, and in practical application, other methods may be used for scaling. For example, each element in the first latent variable may be divided by the corresponding element in the first initial scale factor to obtain a scaled first latent variable. The scaling method is not limited in this embodiment of the present application.

[0023] There may be several implementations for determining the first variable scale factor based on the initial number of coding bits and the target number of coding bits. Three implementations are described below.

[0024] In a first implementation, when the initial number of coding bits is equal to the target number of coding bits, the first initial scale factor is determined as the first variable scale factor.When the initial number of coding bits is not equal to the target number of coding bits, the first variable scale factor is determined in a first cyclic manner based on the initial number of coding bits and the target number of coding bits.

[0025] The i-th cyclic process of the first cyclic scheme includes the following steps: determining a scale factor of the i-th cyclic process, where i is a positive integer; scaling the first latent variable based on the scale factor of the i-th cyclic process to obtain an i-th scaled first latent variable; determining a number of coding bits of an entropy coding result of the i-th scaled first latent variable to obtain an i-th number of coding bits; performing the (i+1)-th cyclic process of the first cyclic scheme when the i-th number of coding bits satisfies the continuous scaling condition, or terminating the execution of the first cyclic scheme and determining a first variable scale factor based on the scale factor of the i-th cyclic process when the i-th number of coding bits does not satisfy the continuous scaling condition.

[0026] An implementation process of determining a scale factor of an i-th cyclic process is to determine a scale factor of the i-th cyclic process based on the scale factor of the (i-1)-th cyclic process of a first cyclic scheme, the (i-1)-th number of coding bits, and the target number of coding bits as follows: When i=1, the scale factor of the (i-1)-th cyclic process is a first initial scale factor, and the (i-1)-th number of coding bits is the initial number of coding bits.

[0027] In this case, the successive scaling condition includes both the (i-1)th number of coding bits and the i-th number of coding bits being less than the target number of coding bits, or the successive scaling condition includes both the (i-1)th number of coding bits and the i-th number of coding bits being greater than the target number of coding bits.

[0028] In other words, the continuous scaling condition includes that the i-th number of coding bits does not exceed the target number of coding bits. In this specification, "does not exceed" means that the number of coding bits for the first (i-1) times is always smaller than the target number of coding bits, and the i-th number of coding bits continues to be smaller than the target number of coding bits. Alternatively, the number of coding bits for the first (i-1) times is always larger than the target number of coding bits, and the i-th number of coding bits continues to be larger than the target number of coding bits. Conversely, "exceeds" means that the number of coding bits for the first (i-1) times is always smaller than the target number of coding bits, and the i-th number of coding bits is larger than the target number of coding bits. Alternatively, the number of coding bits for the first (i-1) times is always larger than the target number of coding bits, and the i-th number of coding bits is smaller than the target number of coding bits.

[0029] An implementation process for determining a first variable scale factor based on a scale factor of the i-th cyclic process includes determining a scale factor of the i-th cyclic process as the first variable scale factor when the i-th number of coding bits is equal to a target number of coding bits, or determining the first variable scale factor based on the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process of the first cyclic scheme when the i-th number of coding bits is not equal to the target number of coding bits.

[0030] In other words, the scale factor of the i-th cyclic process is the scale factor last obtained in the first cyclic manner, and the i-th number of coding bits is the number of coding bits last obtained. When the number of coding bits last obtained is equal to the target number of coding bits, the scale factor last obtained is determined as the first variable scale factor. When the number of coding bits last obtained is not equal to the target number of coding bits, the first variable scale factor is determined based on the scale factors obtained in the last two times.

[0031] An implementation process for determining a first variable scale factor based on a scale factor of an i-th cyclic process and a scale factor of an (i-1)-th cyclic process of a first cyclic scheme includes determining an average value of the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process, and determining the first variable scale factor based on the average value.

[0032] The average value may be directly determined as the first variable scale factor, or the average value may be multiplied by a preset constant to obtain the first variable scale factor. Optionally, the constant may be less than 1.

[0033] Of course, the implementation process of determining a first variable scale factor based on the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process of the first cyclic method may further include determining a first variable scale factor in a second cyclic method based on the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process as follows:

[0034] In one example, the jth cyclic process of the second cyclic scheme includes the following steps: determining a third scale factor of the jth cyclic process based on a first scale factor of the jth cyclic process and a second scale factor of the jth cyclic process, where when j is equal to 1, the first scale factor of the jth cyclic process is one of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, and the second scale factor of the jth cyclic process is the other of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, the first scale factor of the jth cyclic process corresponds to a jth first number of coding bits, the second scale factor of the jth cyclic process corresponds to a jth second number of coding bits, the jth first number of coding bits is a number of coding bits of an entropy coding result of a first latent variable scaled based on the first scale factor of the jth cyclic process, and the jth third scale factor of the coding bits is a number of coding bits of an entropy coding result of a first latent variable scaled based on the first scale factor of the jth cyclic process. determining that the second number is a number of coding bits of the entropy coding result of the first latent variable scaled based on the second scale factor of the jth cyclic process, the jth first number of coding bits being smaller than the jth second number of coding bits; obtaining a jth third number of coding bits, the jth third number of coding bits being a number of coding bits of the entropy coding result of the first latent variable scaled based on the third scale factor of the jth cyclic process; if the jth third number of coding bits does not satisfy a consecutive cycle condition, terminating execution of the second cyclic scheme and determining the third scale factor of the jth cyclic process as the first variable scale factor; if the jth third number of coding bits satisfies the consecutive cycle condition, is greater than the target number of coding bits and is smaller than the jth second number of coding bits, using the third scale factor of the jth cyclic process as the second scale factor of the (j+1)th cyclic process;and performing the (j+1)th cyclic process of the second cyclic scheme by using the first scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process, or performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process and using the second scale factor of the jth cyclic process as the second scale factor of the (j+1)th cyclic process if the jth third number of coding bits satisfies a consecutive cycle condition, is smaller than the target number of coding bits and is larger than the jth first number of coding bits.

[0035] In another example, the jth cyclic process of the second cyclic scheme includes the steps of: determining a third scale factor of the jth cyclic process based on the first scale factor of the jth cyclic process and the second scale factor of the jth cyclic process, where when j is equal to 1, the first scale factor of the jth cyclic process is one of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, and the second scale factor of the jth cyclic process is the other of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, the first scale factor of the jth cyclic process corresponds to the jth first number of coding bits, the second scale factor of the jth cyclic process corresponds to the jth second number of coding bits, the jth first number of coding bits is smaller than the jth second number of coding bits, and j is a positive integer; and obtaining the jth third number of coding bits. obtaining a j-th third number of coding bits, the number of coding bits being a number of coding bits of an entropy coding result of the first latent variable scaled based on the third scale factor of the j-th cyclic process; terminating execution of the second cyclic scheme and determining the third scale factor of the j-th cyclic process as a first variable scale factor if the j-th third number of coding bits does not satisfy a consecutive cycle condition; terminating execution of the second cyclic scheme and determining the first variable scale factor based on the first scale factor of the j-th cyclic process if j has not reached a maximum number of cycles and the j-th third number of coding bits satisfies the consecutive cycle condition; using the third scale factor of the j-th cyclic process as a second scale factor of the (j+1)-th cyclic process if j has not reached a maximum number of cycles and the j-th third number of coding bits satisfies the consecutive cycle condition and is greater than a target number of coding bits and less than the j-th second number of coding bits;and performing the (j+1)th cyclic process of the second cyclic scheme by using the first scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process, or performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process and using the second scale factor of the jth cyclic process as the second scale factor of the (j+1)th cyclic process if j has not reached the maximum number of cycles and the jth third number of coding bits satisfies a consecutive cycle condition, is smaller than the target number of coding bits and is larger than the jth first number of coding bits.

[0036] An implementation process for determining a third scale factor of a jth cyclic process based on a first scale factor of the jth cyclic process and a second scale factor of the jth cyclic process includes: determining an average value of the first scale factor of the jth cyclic process and the second scale factor of the jth cyclic process; and determining a third scale factor of the jth cyclic process based on the average value. In one example, the average value may be directly determined as the third scale factor of the jth cyclic process, or the average value may be multiplied by a preset constant to obtain the third scale factor of the jth cyclic process. Optionally, the constant may be less than 1.

[0037] In addition, an implementation process for obtaining the jth third number of coding bits includes scaling the first latent variable based on the third scale factor of the jth cyclic process to obtain a scaled first latent variable, performing a quantization process on the scaled first latent variable to obtain a quantized first latent variable, performing entropy coding on the quantized first latent variable, and counting the number of coding bits of the entropy coding result to obtain the jth third number of coding bits.

[0038] When the media data to be encoded is encoded at a constant bit rate, an implementation process for determining a first variable scale factor based on a first scale factor of a jth cyclic process includes determining the first scale factor of the jth cyclic process as the first variable scale factor. When the media data to be encoded is encoded at a variable bit rate, an implementation process for determining a first variable scale factor based on a first scale factor of a jth cyclic process includes determining a first difference between a target number of coding bits and a jth first number of coding bits, determining a second difference between the jth second number of coding bits and the target number of coding bits, and determining the first scale factor of the jth cyclic process as the first variable scale factor if the first difference is smaller than the second difference, determining the second scale factor of the jth cyclic process as the first variable scale factor if the second difference is smaller than the first difference, or determining the first scale factor of the jth cyclic process as the first variable scale factor or determining the second scale factor of the jth cyclic process as the first variable scale factor if the first difference is equal to the second difference.

[0039] When the media data to be encoded is encoded at a constant bit rate, the successive cycle condition includes that the jth third number of coding bits is greater than the target number of coding bits, or the successive cycle condition includes that the jth third number of coding bits is smaller than the target number of coding bits, and the difference between the target number of coding bits and the jth third number of coding bits is greater than the bit number threshold. When the media data to be encoded is encoded at a variable bit rate, the successive cycle condition includes that the absolute value of the difference between the target number of coding bits and the jth third number of coding bits is greater than the bit number threshold. In other words, the successive cycle condition includes that the jth third number of coding bits is greater than the target number of coding bits, and the difference between the jth third number of coding bits and the target number of coding bits is greater than the bit number threshold, or the successive cycle condition includes that the jth third number of coding bits is smaller than the target number of coding bits, and the difference between the target number of coding bits and the jth third number of coding bits is greater than the bit number threshold.

[0040] In the second implementation, when the initial number of coding bits is equal to the target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the initial number of coding bits is not equal to the target number of coding bits, the first variable scale factor is determined according to the above-mentioned first implementation. However, the difference from the above-mentioned first implementation is that when the initial number of coding bits is smaller than the target number of coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the first step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling conditions include that the i-th number of coding bits is smaller than the target number of coding bits. When the initial number of coding bits is larger than the target number of coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the second step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling conditions include that the i-th number of coding bits is larger than the target number of coding bits.

[0041] Scaling the scale factor of the (i-1)th cyclic process of the first cyclic method based on a first step may refer to increasing the scale factor of the (i-1)th cyclic process based on the first step, and further, scaling the scale factor of the (i-1)th cyclic process of the first cyclic method based on a second step may refer to decreasing the scale factor of the (i-1)th cyclic process based on the second step.

[0042] The increase and decrease processes may be linear or nonlinear. For example, the sum of the scale factor of the (i-1)th cyclic process and the first step may be determined as the scale factor of the i-th cyclic process, or the difference between the scale factor of the (i-1)th cyclic process and the second step may be determined as the scale factor of the i-th cyclic process.

[0043] It should be noted that the first step and the second step may be preset, and the first step and the second step may be adjusted based on different requirements. In addition, the first step may be equal or unequal to the second step.

[0044] In a third implementation, when the initial number of coding bits is smaller than or equal to the target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the initial number of coding bits is larger than the target number of coding bits, the first variable scale factor may be determined according to the first implementation described above. Alternatively, the first variable scale factor may be determined in the second implementation described above when the initial number of coding bits is larger than the target number of coding bits.

[0045] In a second case, based on the first case, the method further includes: obtaining an entropy coding result of a third latent variable, where the third latent variable is determined based on the second latent variable by using the context model, and the third latent variable indicates a probability distribution of the second latent variable; and writing the entropy coding result of the third latent variable into a bitstream. A total number of coding bits of the entropy coding result of the second latent variable and the coding bits of the entropy coding result of the third latent variable satisfies a preset encoding rate condition.

[0046] Determining a first variable scale factor based on the first latent variable includes determining an initial number of coding bits based on the first latent variable, and determining the first variable scale factor based on the initial number of coding bits and the target number of coding bits.

[0047] There can be two ways to determine the initial number of coding bits based on the first latent variable, which are described separately below.

[0048] In a first implementation form, a corresponding initial number of context coding bits and initial entropy coding model parameters are determined based on a first latent variable by using a context model, a number of coding bits of an entropy coding result of the first latent variable is determined based on the initial entropy coding model parameters to obtain a basic initial number of coding bits, and the initial number of coding bits is determined based on the initial number of context coding bits and the basic initial number of coding bits.

[0049] The context model includes a context encoding neural network model and a context decoding neural network model. An implementation process of determining a corresponding initial number of context coding bits and initial entropy coding model parameters based on a first latent variable by using the context model includes: processing the first latent variable by using the context encoding neural network model to obtain a fifth latent variable, where the fifth latent variable indicates a probability distribution of the first latent variable; determining an entropy coding result of the fifth latent variable, and using a number of coding bits of the entropy coding result of the fifth latent variable as the initial number of context coding bits; and processing the fifth latent variable obtained through the reconstruction by using the context decoding neural network model to reconstruct the fifth latent variable based on the entropy coding result of the fifth latent variable and obtain the initial entropy coding model parameters.

[0050] In a second implementation form, a first preset scale factor is used as a first initial scale factor, a first latent variable is scaled based on the first initial scale factor, a corresponding initial number of context coding bits and initial entropy coding model parameters are determined based on the scaled first latent variable by using the context model, and further, a number of coding bits of an entropy coding result of the scaled first latent variable is determined based on the initial entropy coding model parameters to obtain a basic initial number of coding bits, and the initial number of coding bits is determined based on the initial number of context coding bits and the basic initial number of coding bits.

[0051] The context model includes a context encoding neural network model and a context decoding neural network model. An implementation process of determining a corresponding initial number of context coding bits and initial entropy coding model parameters based on the scaled first latent variable by using the context model includes: processing the scaled first latent variable by using the context encoding neural network model to obtain a sixth latent variable, where the sixth latent variable indicates a probability distribution of the scaled first latent variable; determining an entropy coding result of the sixth latent variable, and using a number of coding bits of the entropy coding result of the sixth latent variable as the initial number of context coding bits; and processing the sixth latent variable obtained through the reconstruction by using the context decoding neural network model to reconstruct the sixth latent variable based on the entropy coding result of the sixth latent variable and obtain the initial entropy coding model parameters.

[0052] There may be several implementations for determining the first variable scale factor based on the initial number of coding bits and the target number of coding bits. Three implementations are described below.

[0053] In a first implementation, when the initial number of coding bits is equal to the target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the initial number of coding bits is not equal to the target number of coding bits, the first variable scale factor is determined in a first cyclic manner based on the initial number of coding bits and the target number of coding bits.

[0054] The i-th cyclic process of the first cyclic scheme includes the steps of: determining a scale factor of the i-th cyclic process, where i is a positive integer; scaling the first latent variable based on the scale factor of the i-th cyclic process to obtain an i-th scaled first latent variable; determining a corresponding i-th number of context coding bits and an i-th entropy coding model parameter based on the i-th scaled first latent variable by using a context model; and determining an i-th entropy coding model parameter based on the i-th scaled first latent variable by using a context model to obtain an i-th base number of coding bits. determining a number of coding bits of an entropy coding result of the i-th scaled first latent variable based on the parameters; determining the i-th number of coding bits based on the i-th number of context coding bits and the i-th base number of coding bits; and performing an (i+1)-th cyclic process of the first cyclic scheme when the i-th number of coding bits satisfies the continuous scaling condition, or terminating the execution of the first cyclic scheme and determining a first variable scale factor based on the scale factor of the i-th cyclic process when the i-th number of coding bits does not satisfy the continuous scaling condition.

[0055] In the second implementation, when the initial number of coding bits is equal to the target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the initial number of coding bits is not equal to the target number of coding bits, the first variable scale factor is determined according to the above-mentioned first implementation. However, the difference from the above-mentioned first implementation is that when the initial number of coding bits is smaller than the target number of coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the first step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling conditions include that the i-th number of coding bits is smaller than the target number of coding bits. When the initial number of coding bits is larger than the target number of coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the second step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling conditions include that the i-th number of coding bits is larger than the target number of coding bits.

[0056] Scaling the scale factor of the (i-1)th cyclic process of the first cyclic method based on a first step may refer to increasing the scale factor of the (i-1)th cyclic process based on the first step, and further, scaling the scale factor of the (i-1)th cyclic process of the first cyclic method based on a second step may refer to decreasing the scale factor of the (i-1)th cyclic process based on the second step.

[0057] The increase and decrease processes may be linear or nonlinear. For example, the sum of the scale factor of the (i-1)th cyclic process and the first step may be determined as the scale factor of the i-th cyclic process, or the difference between the scale factor of the (i-1)th cyclic process and the second step may be determined as the scale factor of the i-th cyclic process.

[0058] It should be noted that the first step and the second step may be preset, and the first step and the second step may be adjusted based on different requirements. In addition, the first step may be equal or unequal to the second step.

[0059] In a third implementation, when the initial number of coding bits is smaller than or equal to the target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the initial number of coding bits is larger than the target number of coding bits, the first variable scale factor may be determined according to the first implementation described above. Alternatively, the first variable scale factor may be determined in the second implementation described above when the initial number of coding bits is larger than the target number of coding bits.

[0060] In a third case, based on the first case, the method further includes the steps of: determining a second variable scale factor based on the first latent variable; obtaining an entropy coding result of a fourth latent variable, where the fourth latent variable is obtained by scaling the third latent variable based on the second variable scale factor, and the third latent variable is determined based on the second latent variable by using a context model, and the third latent variable indicates a probability distribution of the second latent variable; and writing the entropy coding result of the fourth latent variable and the encoding result of the second variable scale factor into a bitstream. A total number of coding bits of the entropy coding result of the second latent variable and the coding bits of the entropy coding result of the fourth latent variable satisfies a preset encoding rate condition.

[0061] There may be two ways to determine the first variable scale factor and the second variable scale factor based on the first latent variable, which are respectively as follows:

[0062] In a first implementation form, a corresponding initial number of context coding bits and initial entropy coding model parameters are determined based on a first latent variable by using a context model, a number of coding bits of an entropy coding result of the first latent variable is determined based on the initial entropy coding model parameters to obtain a basic initial number of coding bits, and a first variable scale factor and a second variable scale factor are determined based on the initial number of context coding bits, the basic initial number of coding bits, and a target number of coding bits.

[0063] In a second implementation, the first preset scale factor is used as the first initial scale factor, and the second preset scale factor is used as the second initial scale factor. The first latent variable is scaled based on the first initial scale factor, and a corresponding initial number of context coding bits and initial entropy coding model parameters are determined based on the scaled first latent variable and the second initial scale factor by using a context model, and a number of coding bits of the entropy coding result of the first latent variable is determined based on the initial entropy coding model parameters to obtain a basic initial number of coding bits. The first variable scale factor and the second variable scale factor are determined based on the initial number of context coding bits, the basic initial number of coding bits, and the target number of coding bits.

[0064] The context models include a context encoding neural network model and a context decoding neural network model. An implementation process for determining a corresponding initial number of context coding bits and initial entropy coding model parameters based on a scaled first latent variable and a second initial scale factor by using a context model includes: processing the scaled first latent variable by using a context encoding neural network model to obtain a sixth latent variable, where the sixth latent variable indicates a probability distribution of the scaled first latent variable; scaling the sixth latent variable based on the second initial scale factor to obtain a seventh latent variable; determining an entropy coding result of the seventh latent variable and using a number of coding bits of the entropy coding result of the seventh latent variable as the initial number of context coding bits; reconstructing the seventh latent variable based on the entropy coding result of the seventh latent variable and processing the seventh latent variable obtained through the reconstruction by using a context decoding neural network model to obtain initial entropy coding model parameters.

[0065] An implementation process for scaling the sixth latent variable based on the second initial scale factor is to multiply each element in the sixth latent variable by the corresponding element in the second initial scale factor to obtain a seventh latent variable as follows:

[0066] It should be noted that the above implementation process is just an example, and in practical application, other methods may be used for scaling. For example, each element in the sixth latent variable may be divided by the corresponding element in the second initial scale factor to obtain the seventh latent variable. The scaling method is not limited in this embodiment of the present application.

[0067] There may be two implementation processes for determining the first variable scale factor and the second variable scale factor based on the initial number of context coding bits, the basic initial number of coding bits, and the target number of coding bits. The implementation processes are described below individually.

[0068] In a first implementation, the second variable scale factor is set to a second initial scale factor, and the base target number of coding bits is determined based on the target number of coding bits and at least one of the base initial number of coding bits and the initial number of context coding bits. The first variable scale factor and the base actual number of coding bits are determined based on the second initial scale factor, the base target number of coding bits and the base initial number of coding bits, and the base actual number of coding bits is the number of coding bits of the entropy coding result of the first latent variable scaled based on the first variable scale factor. The target number of context coding bits is determined based on the target number of coding bits and the base actual number of coding bits. The second variable scale factor is determined based on the target number of context coding bits and the initial number of context coding bits.

[0069] The implementation process of determining the basic target number of coding bits based on the target number of coding bits and at least one of the basic initial number of coding bits and the initial number of context coding bits includes: subtracting the initial number of context coding bits from the target number of coding bits to obtain the basic target number of coding bits; determining a ratio of the basic initial number of coding bits to the initial number of context coding bits and determining the basic target number of coding bits based on the ratio and the target number of coding bits; or determining the basic target number of coding bits based on the ratio of the target number of coding bits to the basic initial number of coding bits. Of course, it may be further determined by using another implementation process.

[0070] In addition, there are three implementations for determining the first variable scale factor according to the second initial scale factor, the base target number of coding bits, and the base initial number of coding bits. The following describes the implementations individually.

[0071] Scheme 11: When the basic initial number of coding bits is equal to the basic target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the basic initial number of coding bits is not equal to the basic target number of coding bits, the first variable scale factor is determined in a first cyclic manner based on the basic initial number of coding bits and the basic target number of coding bits.

[0072] The i-th cyclic process of the first cyclic scheme includes the steps of: determining a scale factor of the i-th cyclic process, where i is a positive integer; scaling the first latent variable based on the scale factor of the i-th cyclic process to obtain an i-th scaled first latent variable; determining an i-th number of context coding bits and an i-th entropy coding model parameter based on the i-th scaled first latent variable and a second initial scale factor by using a context model; and determining an i-th base number of coding bits based on the i-th scaled first latent variable and a second initial scale factor by using a context model. determining a number of coding bits of an entropy coding result of the ith scaled first latent variable based on a scaling model parameter; determining the ith number of coding bits based on the ith number of context coding bits and the ith base number of coding bits; and performing an (i+1)th cyclic process of the first cyclic scheme when the ith number of coding bits satisfies the continuous scaling condition, or terminating the execution of the first cyclic scheme and determining a first variable scale factor based on the scale factor of the ith cyclic process when the ith number of coding bits does not satisfy the continuous scaling condition.

[0073] Scheme 12: When the basic initial number of coding bits is equal to the basic target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the basic initial number of coding bits is not equal to the basic target number of coding bits, the first variable scale factor is determined in the above scheme 11. However, the difference from the above scheme 11 is that when the basic initial number of coding bits is smaller than the basic target number of coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the first step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling conditions include that the i-th number of coding bits is smaller than the basic target number of coding bits. When the basic initial number of coding bits is larger than the basic target number of coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the second step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling conditions include that the i-th number of coding bits is larger than the basic target number of coding bits.

[0074] Manner 13: When the basic initial number of coding bits is less than or equal to the basic target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the basic initial number of coding bits is greater than the basic target number of coding bits, the first variable scale factor may be determined in the above manner 11. Alternatively, when the basic initial number of coding bits is greater than the basic target number of coding bits, the first variable scale factor may be determined in the above manner 12.

[0075] The implementation process of determining the second variable scale factor based on the target number of context coding bits and the initial number of context coding bits is similar to the implementation process of determining the first variable scale factor based on the base initial number of coding bits and the base target number of coding bits. The following describes the three schemes individually.

[0076] Scheme 21: When the initial number of context coding bits is equal to the target number of context coding bits, the second initial scale factor is determined as the second variable scale factor. When the initial number of context coding bits is not equal to the target number of context coding bits, the first latent variable is scaled based on the first variable scale factor to obtain the second latent variable. The second variable scale factor is determined in a first cyclic manner based on the initial number of context coding bits, the target number of context coding bits, and the second latent variable.

[0077] The ith cyclic process of the first cyclic scheme includes the following steps: determining a scale factor of the ith cyclic process, where i is a positive integer; determining an ith number of context coding bits and an ith entropy coding model parameter based on the second latent variable and the scale factor of the ith cyclic process by using a context model; determining a number of coding bits of an entropy coding result of the second latent variable based on the ith entropy coding model parameter to obtain an ith base number of coding bits; determining the ith number of coding bits based on the ith number of context coding bits and the ith base number of coding bits; performing an (i+1)th cyclic process of the first cyclic scheme when the ith number of coding bits satisfies a continuous scaling condition, or terminating the execution of the first cyclic scheme and determining a second variable scale factor based on the scale factor of the ith cyclic process when the ith number of coding bits does not satisfy the continuous scaling condition.

[0078] An implementation process for determining the i-th number of context coding bits and the i-th entropy coding model parameters based on the second latent variable and the scale factor of the i-th cyclic process by using a context model includes processing the second latent variable by using a context encoding neural network model to obtain a third latent variable, where the third latent variable indicates a probability distribution of the second latent variable; scaling the third latent variable based on the scale factor of the i-th cyclic process to obtain an i-th scaled third latent variable; and determining an entropy coding result of the i-th scaled third latent variable. determining a number of coding bits of an entropy coding result of the i-th scaled third latent variable as the i-th number of context coding bits; reconstructing the i-th scaled third latent variable based on the entropy coding result of the i-th scaled third latent variable; scaling the reconstructed i-th scaled third latent variable based on a scale factor of the i-th cyclic processing to obtain a reconstructed third latent variable; and processing the reconstructed third latent variable by using a context decoding neural network model to obtain the i-th entropy coding model parameters.

[0079] Scheme 22: When the initial number of context coding bits is equal to the target number of context coding bits, the second initial scale factor is determined as the second variable scale factor. When the initial number of context coding bits is not equal to the target number of context coding bits, the second variable scale factor is determined in the above-mentioned scheme 21. However, the difference from the above-mentioned scheme 21 is that when the initial number of context coding bits is smaller than the target number of context coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the first step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling condition includes that the i-th number of coding bits is smaller than the target number of context coding bits. When the initial number of context coding bits is larger than the target number of context coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the second step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling condition includes that the i-th number of coding bits is greater than the target number of context coding bits.

[0080] Scheme 23: When the initial number of context coding bits is less than or equal to the target number of context coding bits, the second initial scale factor is determined as the second variable scale factor. When the initial number of context coding bits is greater than the target number of context coding bits, the second variable scale factor may be determined as in scheme 21 above. Alternatively, the second variable scale factor may be determined as in scheme 22 above when the initial number of context coding bits is greater than the target number of context coding bits.

[0081] In a second implementation, the target number of coding bits is divided into a base target number of coding bits and a target number of context coding bits, and a first variable scale factor is determined based on the base target number of coding bits and the base initial number of coding bits, and a second variable scale factor is determined based on the target number of context coding bits and the initial number of context coding bits.

[0082] According to a second aspect, a decoding method is provided. The method is also described in several cases.

[0083] In a first case, the method includes the steps of determining a reconstructed second latent variable and a reconstructed first variable scale factor based on the bitstream; scaling the reconstructed second latent variable based on the reconstructed first variable scale factor to obtain a reconstructed first latent variable, where the reconstructed first latent variable is indicative of a characteristic of the media data to be decoded; and processing the reconstructed first latent variable by using a first decoding neural network model to obtain the reconstructed media data.

[0084] In a second case, the method includes the steps of determining a reconstructed third latent variable and a reconstructed first variable scale factor based on the bitstream; determining a reconstructed second latent variable based on the bitstream and the reconstructed third latent variable; scaling the reconstructed second latent variable based on the reconstructed first variable scale factor to obtain a reconstructed first latent variable, where the reconstructed first latent variable is indicative of a characteristic of the media data to be decoded; and processing the reconstructed first latent variable by using a first decoding neural network model to obtain the reconstructed media data.

[0085] Determining a reconstructed second latent variable based on the bitstream and the reconstructed third latent variable includes processing the reconstructed third latent variable by using a context decoding neural network model to obtain reconstructed first entropy coding model parameters, and determining a reconstructed second latent variable based on the bitstream and the reconstructed first entropy coding model parameters.

[0086] In a third case, the method includes the steps of determining a reconstructed fourth latent variable, a reconstructed second variable scale factor, and a reconstructed first variable scale factor based on the bitstream; determining a reconstructed second latent variable based on the bitstream, the reconstructed fourth latent variable, and the reconstructed second variable scale factor; determining a reconstructed second latent variable based on the bitstream and the reconstructed third latent variable; scaling the reconstructed second latent variable based on the reconstructed first variable scale factor to obtain a reconstructed first latent variable, where the reconstructed first latent variable is indicative of a characteristic of the media data to be decoded; and processing the reconstructed first latent variable by using a first decoding neural network model to obtain the reconstructed media data.

[0087] Determining a reconstructed second latent variable based on the bitstream, the reconstructed fourth latent variable, and the reconstructed second variable scale factor includes scaling the reconstructed fourth latent variable based on the reconstructed second variable scale factor to obtain a reconstructed third latent variable, processing the reconstructed third latent variable by using a context decoding neural network model to obtain reconstructed second entropy coding model parameters, and determining the reconstructed second latent variable based on the bitstream and the reconstructed second entropy coding model parameters.

[0088] The media data is an audio signal, a video signal or an image.

[0089] According to a third aspect, an encoding device is provided, the encoding device having the capability of implementing the behavior of the encoding method according to the first aspect, the encoding device including at least one module, the at least one module being configured to implement the encoding method according to the first aspect.

[0090] According to a fourth aspect, there is provided a decoding device, the decoding device having a function for implementing the behavior of the decoding method according to the second aspect, the decoding device including at least one module, the at least one module being configured to implement the decoding method according to the second aspect.

[0091] According to a fifth aspect, there is provided an encoder-side device, the encoder-side device including a processor and a memory. The memory is configured to store a program for implementing the encoding method according to the first aspect. The processor is configured to execute the program stored in the memory to implement the encoding method according to the first aspect.

[0092] Optionally, the encoder side device may further include a communication bus, the communication bus configured to establish a connection between the processor and the memory.

[0093] According to a sixth aspect, a decoder-side device is provided. The decoder-side device includes a processor and a memory. The memory is configured to store a program for implementing the decoding method according to the second aspect. The processor is configured to execute the program stored in the memory to implement the decoding method according to the second aspect.

[0094] Optionally, the decoder-side device may further include a communication bus, the communication bus configured to establish a connection between the processor and the memory.

[0095] According to a seventh aspect, there is provided a computer-readable storage medium. The storage medium stores instructions which, when executed on a computer, enable the computer to perform steps of the encoding method according to the first aspect or steps of the decoding method according to the second aspect.

[0096] According to an eighth aspect, there is provided a computer program product comprising instructions, which when executed on a computer, enable the computer to perform steps of the encoding method according to the first aspect or steps of the decoding method according to the second aspect. Alternatively, a computer program is provided, which when executed causes each step of the encoding method according to the first aspect or each step of the decoding method according to the second aspect to be performed.

[0097] According to a ninth aspect, there is provided a computer-readable storage medium comprising a bitstream obtained by using the encoding method according to the first aspect.

[0098] The technical effects obtained in the third, fourth, fifth, sixth, seventh, eighth and ninth aspects are similar to those obtained through the corresponding technical means in the first or second aspects, and the details are not repeated here.

[0099] The technical solutions provided in the embodiments of the present application may achieve at least the following beneficial effects:

[0100] The first latent variable is scaled based on the first variable scale factor to obtain the second latent variable, and the number of coding bits of the entropy coding result of the second latent variable satisfies a preset encoding rate condition. This ensures that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can satisfy the preset encoding rate condition, i.e., the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can be basically kept consistent instead of dynamically changing, thereby meeting the encoder's requirement for a stable encoding rate. Furthermore, when side information (e.g., a window type, a Temporal Noise Shaping (TNS) parameter, a Frequency-domain noise shaping (FDNS) parameter, and / or a bandwidth extension (BWE) parameter) needs to be transmitted, it can be ensured that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data is basically consistent with the number of coding bits of the side information, thereby meeting the encoder's requirement for a stable encoding rate. [Brief description of the drawings]

[0101] [Figure 1] FIG. 1 is a schematic diagram of an implementation environment according to an embodiment of the present application. [Diagram 2] FIG. 1 is a schematic diagram of an implementation environment of a terminal scene according to an embodiment of the present application. [Diagram 3] FIG. 1 is a schematic diagram of an implementation environment of a transcoding scene of a wireless or core network device according to one embodiment of the present application. [Figure 4] FIG. 1 is a schematic diagram of a radio and television scene implementation environment according to one embodiment of the present application. [Diagram 5] FIG. 1 is a schematic diagram of an implementation environment for a virtual reality streaming scene according to one embodiment of the present application. [Figure 6] 2 is a flowchart of a first encoding method according to an embodiment of the present application. [Figure 7] FIG. 2 is a schematic diagram of a first latent variable form according to an embodiment of the present application. [Figure 8] FIG. 2 is a schematic diagram of a second latent variable form according to an embodiment of the present application. [Figure 9] 2 is a flowchart of a first decoding method according to an embodiment of the present application; [Figure 10] 4 is a flowchart of a second encoding method according to an embodiment of the present application. [Figure 11] 4 is a flowchart of a second decoding method according to an embodiment of the present application. [Figure 12] FIG. 11 is an exemplary block diagram of the encoding method shown in FIG. 10 according to one embodiment of the present application. [Figure 13] FIG. 12 is an exemplary block diagram of the decoding method shown in FIG. 11 according to one embodiment of the present application. [Figure 14] 11 is a flowchart of a third encoding method according to an embodiment of the present application. [Figure 15] 4 is a flowchart of a third decoding method according to an embodiment of the present application. [Figure 16A] FIG. 15 is an exemplary block diagram of the encoding method shown in FIG. 14 according to one embodiment of the present application. [Figure 16B]FIG. 15 is an exemplary block diagram of the encoding method shown in FIG. 14 according to one embodiment of the present application. [Figure 17] FIG. 16 is an exemplary block diagram of the decoding method shown in FIG. 15 according to one embodiment of the present application. [Figure 18] FIG. 2 is a schematic diagram of the structure of an encoding device according to an embodiment of the present application; [Figure 19] 1 is a schematic diagram of the structure of a decoding device according to an embodiment of the present application; [Figure 20] 1 is a schematic block diagram of an encoding and decoding device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0102] To clarify the objectives, technical solutions and advantages of the embodiments of the present application, the following further describes the implementation of the present application in detail with reference to the accompanying drawings.

[0103] Before the encoding and decoding methods provided in the embodiments of the present application are explained and described in detail, the terminology and implementation environment in the embodiments of the present application are first described.

[0104] For ease of understanding, terms in the embodiments of this application are first explained.

[0105] Encoding is the process of compressing the media data to be encoded into a bitstream. The media data to be encoded mainly includes audio signals, video signals, and images. Encoding an audio signal is the process of compressing an audio frame sequence contained in the audio signal to be encoded into a bitstream, encoding a video signal is the process of compressing an image sequence contained in the video signal to be encoded into a bitstream, and encoding an image is the process of compressing an image to be encoded into a bitstream.

[0106] It should be noted that after being compressed into a bitstream, the media data may be referred to as encoded media data or compressed media data. For example, for an audio signal, after the audio signal is compressed into a bitstream, the audio signal may be referred to as an encoding audio signal or a compressed audio signal, after a video signal is compressed into a bitstream, the video signal may also be referred to as an encoding video signal or a compressed video signal, and after an image is compressed into a bitstream, the image may also be referred to as an encoding image or a compressed image.

[0107] Decoding is the process of restoring an encoding bitstream to reconstructed media data according to certain syntax rules and processing methods. Decoding an audio bitstream is the process of restoring an audio bitstream to a reconstructed audio signal, decoding a video bitstream is the process of restoring a video bitstream to a reconstructed video signal, and decoding an image bitstream is the process of restoring an image bitstream to a reconstructed image.

[0108] Entropy coding is a coding method in which information is not lost during coding according to the entropy principle. In other words, it is a lossless data compression method. Entropy coding is performed based on the occurrence probability of an element, that is, for the same element, when the occurrence probability of the element is different, the number of coding bits of the entropy coding result of the element is different. Entropy coding usually includes arithmetic coding, range coding (RC), Huffman coding, etc.

[0109] Constant bit rate (CBR) means that the encoding bit rate is a constant value, i.e. the target encoding bit rate.

[0110] Variable bit rate (VBR) means that the encoding rate can exceed the target encoding bit rate or can be less than the target encoding bit rate, but the difference between the encoding rate and the target encoding bit rate is small.

[0111] The implementation environment in the embodiment of the present application will be described below.

[0112] FIG. 1 is a schematic diagram of an implementation environment according to an embodiment of the present application. The implementation environment includes a source device 10, a destination device 20, a link 30, and a storage device 40. The source device 10 may generate encoded media data. Therefore, the source device 10 may also be referred to as a media data encoding device. The destination device 20 may decode the encoded media data generated by the source device 10. Therefore, the destination device 20 may also be referred to as a media data decoding device. The link 30 may receive the encoded media data generated by the source device 10 and transmit the encoded media data to the destination device 20. The storage device 40 may receive the encoded media data generated by the source device 10 and store the encoded media data. In this case, the destination device 20 may directly obtain the encoded media data from the storage device 40. Alternatively, the storage device 40 may correspond to a file server or another intermediate storage device that may store the encoded media data generated by the source device 10. In this case, destination device 20 may stream or download the encoded media data stored in storage device 40 .

[0113] Both source device 10 and destination device 20 may include one or more processors and memory coupled to the one or more processors. The memory may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, any other medium that may be used to store necessary program code in the form of instructions or data structures accessible by a computer, etc. Source device 10 and destination device 20 may include desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, television sets, cameras, display devices, digital media players, video game consoles, in-vehicle computers, etc.

[0114] The link 30 may include one or more media or devices that may transmit the encoded media data from the source device 10 to the destination device 20. In a possible implementation, the link 30 may include one or more communication media that may enable the source device 10 to transmit the encoded media data directly to the destination device 20 in real time. In this embodiment of the present application, the source device 10 may modulate the encoded media data based on a communication standard. The communication standard may be a wireless communication protocol or the like, and may transmit the modulated media data to the destination device 20. The one or more communication media may include wireless and / or wired communication media. For example, the one or more communication media may include a radio frequency (RF) frequency spectrum, or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, which may be a local area network, a wide area network, or a global network (e.g., the Internet), or the like. The one or more communication media may include a router, a switch, a base station, or another device that facilitates communication from the source device 10 to the destination device 20, or the like. This is not particularly limited in this embodiment of the present application.

[0115] In a possible implementation, the storage device 40 may store the received encoded media data sent by the source device 10, and the destination device 20 may obtain the encoded media data directly from the storage device 40. In such a situation, the storage device 40 may include any one of a number of distributed or locally accessed data storage media. For example, any one of a number of distributed or locally accessed data storage media may be a hard disk drive, a Blu-ray disc, a digital versatile disc (DVD), a compact disc read-only memory (CD-ROM), a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium used to store the encoded media data.

[0116] In a possible implementation, the storage device 40 may correspond to a file server or another intermediate storage device that may store the encoded media data generated by the source device 10, and the destination device 20 may perform streaming transmission or download the media data stored in the storage device 40. The file server may be any type of server that may store the encoded media data and transmit the encoded media data to the destination device 20. In a possible implementation, the file server may include a network server, a file transfer protocol (FTP) server, a network attached storage (NAS) device, a local disk drive, etc. The destination device 20 may obtain the encoded media data through any standard data connection (including an Internet connection). Any standard data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL) or a cable modem), or a combination of a wireless channel and a wired connection suitable for obtaining the encoded media data stored on the file server. The transmission of the encoded media data from storage device 40 may be a streaming transmission, a download transmission, or a combination thereof.

[0117] The implementation environment shown in Fig. 1 is only a possible implementation. In addition, the technology in this embodiment of the present application is not only applicable to the source device 10 that can encode media data and the destination device 20 that can decode the encoded media data shown in Fig. 1, but also applicable to another device that can encode media data and decode the encoded media data. This is not particularly limited in this embodiment of the present application.

[0118] In the implementation environment shown in FIG. 1, source device 10 includes data source 120, encoder 100, and output interface 140. In some embodiments, output interface 140 may include a modulator / demodulator (modem) and / or a transmitter. A transmitter may also be referred to as a transmitter. Data source 120 may include an image capture device (e.g., a camera), an archive containing previously captured media data, a feed-in interface for receiving media data from a media data content provider, and / or a computer graphics system for generating media data, or a combination of these sources of media data.

[0119] The data source 120 may send media data to the encoder 100. The encoder 100 may encode the received media data sent by the data source 120 to obtain encoded media data. The encoder may send the encoded media data to an output interface. In some embodiments, the source device 10 sends the encoded media data directly to the destination device 20 through the output interface 140. In another embodiment, the encoded media data may be further stored in the storage device 40, such that the destination device 20 subsequently retrieves the encoded media data and uses it for decoding and / or displaying the encoded media data.

[0120] In the implementation environment shown in FIG. 1, the destination device 20 includes an input interface 240, a decoder 200, and a display device 220. In some embodiments, the input interface 240 includes a receiver and / or a modem. The input interface 240 may receive encoded media data through the link 30 and / or from the storage device 40, and then transmit the encoded media data to the decoder 200. The decoder 200 may decode the received encoded media data to obtain decoded media data. The decoder may transmit the decoded media data to the display device 220. The display device 220 may be integrated with the destination device 20 or may be located external to the destination device 20. In general, the display device 220 displays the decoded media data. The display device 220 may be any one of several types of display devices. For example, the display device 220 may be a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0121] 1, in some aspects, the encoder 100 and the decoder 200 may each be an integrated encoder and decoder and may include appropriate multiplexer-demultiplexer (MUX-DEMUX) units, or other hardware and software, for encoding both audio and video in a shared or separate data stream. In some embodiments, if applicable, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol or another protocol, such as, for example, the user datagram protocol (UDP).

[0122] The encoder 100 and the decoder 200 may each be any one of the following circuits: one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the technology in this embodiment of the present application is implemented partially in software, the device may store instructions used in the software in a suitable non-volatile computer-readable storage medium and execute the instructions in hardware using one or more processors to implement the technology in this embodiment of the present application. Any one of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered as one or more processors. The encoder 100 and the decoder 200 may each be included in one or more encoders or decoders. Any one of the encoders or decoders may be integrated as part of an encoder / decoder combination (codec) in a corresponding device.

[0123] In this embodiment of the application, the encoder 100 may be generally referred to as "signaling" or "transmitting" some information to another device, e.g., the decoder 200. The terms "signaling" or "transmitting" may generally refer to the transmission of syntax elements and / or other data used to decode the compressed media data. Such transmission may be performed in real time or near real time. Alternatively, the communication may be performed after a period of time, e.g., when the syntax elements are stored in a computer-readable storage medium in an encoded bitstream during encoding, and the decoding device may retrieve the syntax elements at any time after they are stored in the medium.

[0124] The encoding and decoding method provided in the embodiments of the present application can be applied to multiple scenes. In the following, several scenes are described individually by using an example in which the media data to be encoded is an audio signal.

[0125] 2 is a schematic diagram of an implementation environment in which the encoding and decoding method according to an embodiment of the present application is applied to a terminal scene. The implementation environment includes a first terminal 101 and a second terminal 201, and the first terminal 101 and the second terminal 201 are in communication connection. The communication connection may be a wireless network connection or a wired network connection. This is not limited in this embodiment of the present application.

[0126] The first terminal 101 may be a sending device or a receiving device. Similarly, the second terminal 201 may be a receiving device or a sending device. When the first terminal 101 is a sending device, the second terminal 201 is a receiving device. When the first terminal 101 is a receiving device, the second terminal 201 is a sending device.

[0127] In the following, an explanation is provided by using an example in which the first terminal 101 is a sending device and the second terminal 201 is a receiving device.

[0128] The first terminal 101 may be the source device 10 in the implementation environment shown in Fig. 1. The second terminal 201 may be the destination device 20 in the implementation environment shown in Fig. 1. Both the first terminal 101 and the second terminal 201 include an audio capture module, an audio playback module, an encoder, a decoder, a channel encoding module, and a channel decoding module.

[0129] The audio capture module of the first terminal 101 captures an audio signal and sends the audio signal to an encoder. The encoder encodes the audio signal by using the encoding method provided in this embodiment of the present application. The encoding may be referred to as source encoding. Then, in order to transmit the audio signal on a channel, the channel encoding module further needs to perform channel encoding, and then transmit the bitstream obtained through encoding on a digital channel by using a wireless or wired network communication device.

[0130] The second terminal 201 receives a bitstream transmitted on a digital channel by using a wireless or wired network communication device. The channel decoding module performs channel decoding on the bitstream. Then, the decoder obtains an audio signal through decoding by using the decoding method provided in this embodiment of the present application, and then plays the audio signal by using an audio playback module.

[0131] The first terminal 101 and the second terminal 201 may be any electronic product that can perform human-computer interaction with a user in one or more ways, such as a keyboard, a touchpad, a touchscreen, a remote control device, a voice interaction device, or a handwriting device, such as a personal computer (PC), a mobile phone, a smartphone, a personal digital assistant (PDA), a wearable device, a pocket personal computer (PPC), a tablet computer, a smart head unit, a smart TV, or a smart speaker.

[0132] Those skilled in the art should understand that the above terminals are merely examples. If another existing or future terminal is applicable to the embodiments of the present application, it should also be included within the protection scope of the embodiments of the present application and is hereby incorporated by reference.

[0133] 3 is a schematic diagram of an implementation environment in which the encoding and decoding method according to an embodiment of the present application is applied to a transcoding scene of a wireless or core network device. The implementation environment includes a channel decoding module, an audio decoder, an audio encoder, and a channel encoding module.

[0134] The audio decoder may be a decoder using the decoding method provided in the embodiment of the present application, or may be a decoder using another decoding method. The audio encoder may be an encoder using the encoding method provided in the embodiment of the present application, or may be an encoder using another encoding method. When the audio decoder is a decoder using the decoding method provided in the embodiment of the present application, the audio encoder is an encoder using another encoding method. When the audio decoder is a decoder using another decoding method, the audio encoder is an encoder using the encoding method provided in the embodiment of the present application.

[0135] In the first case, the audio decoder is a decoder that uses the decoding method provided in the embodiments of the present application, and the audio encoder is an encoder that uses another encoding method.

[0136] In this case, the channel decoding module is configured to perform channel decoding on the received bitstream. The audio decoder is configured to perform source decoding by using the decoding method provided in the embodiment of the present application. And the audio encoder performs encoding by using another encoding method. Conversion from any format to another format, i.e., transcoding, is performed. Then, the data is transmitted after channel encoding.

[0137] In the second case, the audio decoder is a decoder using another decoding method, and the audio encoder is an encoder using the encoding method provided in the embodiments of the present application.

[0138] In this case, the channel decoding module is configured to perform channel decoding on the received bitstream. The audio decoder is configured to perform source decoding by using another decoding method. Then, the audio encoder performs encoding by using the encoding method provided in the embodiment of the present application. Conversion from any format to another format, i.e., transcoding, is performed. Then, the data is transmitted after channel encoding.

[0139] The wireless device may be a wireless access point, a wireless router, a wireless connector, etc. The core network device may be a mobility management entity, a gateway, etc.

[0140] Those skilled in the art should understand that the above-mentioned wireless devices or core network devices are merely examples. If another existing or future wireless or core network device is applicable to the embodiments of the present application, it should also be included within the protection scope of the embodiments of the present application and is hereby incorporated by reference.

[0141] 4 is a schematic diagram of an implementation environment in which the encoding and decoding method according to an embodiment of the present application is applied to radio and television scenes. The radio and television scenes include live broadcast scenes and post-production scenes. For the live broadcast scenes, the implementation environment includes a live program three-dimensional sound production module, a three-dimensional sound encoding module, a set-top box, and a speaker group. The set-top box includes a three-dimensional sound decoding module. For the post-production scenes, the implementation environment includes a post-program three-dimensional sound production module, a three-dimensional sound encoding module, a network receiver, a mobile terminal, a headset, etc.

[0142] In a live broadcast scene, a live program three-dimensional sound production module generates a three-dimensional sound signal. The three-dimensional sound signal is encoded by using the encoding method in the embodiment of the present application to obtain a bitstream. The bitstream is transmitted to a user side through a radio and television network. A three-dimensional sound decoder in a set-top box decodes the bitstream by using the decoding method provided in the embodiment of the present application to reconstruct a three-dimensional sound signal. A speaker group plays the reconstructed three-dimensional sound signal. Alternatively, the bitstream is transmitted to a user side through the Internet. A three-dimensional sound decoder in a network receiver decodes the bitstream by using the decoding method provided in the embodiment of the present application to reconstruct a three-dimensional sound signal. A speaker group plays the reconstructed three-dimensional sound signal. Alternatively, the bitstream is transmitted to a user side through the Internet. A three-dimensional sound decoder in a mobile terminal decodes the bitstream by using the decoding method provided in the embodiment of the present application to reconstruct a three-dimensional sound signal. A headset plays the reconstructed three-dimensional sound signal.

[0143] In a post-production scene, a post-programmed three-dimensional sound production module generates a three-dimensional sound signal. The three-dimensional sound signal is encoded by using the encoding method in the embodiment of the present application to obtain a bitstream. The bitstream is transmitted to a user end through a radio and television network. A three-dimensional sound decoder in a set-top box decodes the bitstream by using the decoding method provided in the embodiment of the present application to reconstruct a three-dimensional sound signal. A speaker group plays the reconstructed three-dimensional sound signal. Alternatively, the bitstream is transmitted to a user end through the Internet. A three-dimensional sound decoder in a network receiver decodes the bitstream by using the decoding method provided in the embodiment of the present application to reconstruct a three-dimensional sound signal. A speaker group plays the reconstructed three-dimensional sound signal. Alternatively, the bitstream is transmitted to a user end through the Internet. A three-dimensional sound decoder in a mobile terminal decodes the bitstream by using the decoding method provided in the embodiment of the present application to reconstruct a three-dimensional sound signal. A headset plays the reconstructed three-dimensional sound signal.

[0144] 5 is a schematic diagram of an implementation environment in which the encoding and decoding method is applied to a virtual reality streaming scene according to an embodiment of the present application. The implementation environment includes an encoder side and a decoder side. The encoder side includes an ingestion module, a pre-processing module, an encoding module, an encapsulation module, and a transmission module. The decoder side includes a de-encapsulation module, a decoding module, a rendering module, and a headset.

[0145] The capture module captures the audio signal. Then, the pre-processing module performs the pre-processing operation. The pre-processing operation includes removing the low-frequency part from the signal, usually the boundary point is 20Hz or 50Hz, extracting the directional information from the signal, etc. Then, the encoding module performs the encoding process by using the encoding method provided in the embodiment of this application. After encoding, the encapsulation module performs the encapsulation. Then, the transmission module transmits the encapsulated signal to the decoder side.

[0146] The decapsulation module on the decoder side first performs decapsulation. The decoding module performs decoding by using the decoding method provided in the embodiment of the present application. Then, the rendering module performs binaural rendering processing on the decoded signal. The signal obtained through the rendering processing is mapped to the listener's headset. The headset can be an independent headset or a headset of a virtual reality-based glasses device.

[0147] The following will explain and describe in detail the encoding and decoding methods provided in the embodiment of the present application. With reference to the implementation environment shown in Fig. 1, it should be noted that any one of the following encoding methods can be implemented by the encoder 100 in the source device 10. Any one of the following decoding methods can also be implemented by the decoder 200 in the destination device 20.

[0148] It should be noted that the embodiment of the present application may be applied to a codec that does not include a context model, or may be applied to a codec that includes a context model. In addition, in the embodiment of the present application, not only the latent variables generated by using the media data to be encoded are scaled based on the scale factor, but also the latent variables determined by using the context model may be scaled based on the scale factor. Therefore, the following will explain and describe in detail the encoding and decoding methods provided in the embodiment of the present application in multiple embodiments. In addition, the scale factor and the variable scale factor in the embodiment of the present application may be a non-quantized value or a quantized value. This is not limited in the embodiment of the present application.

[0149] 6 is a flowchart of a first encoding method according to an embodiment of the present application. This method does not include a context model, and only the latent variables generated by using the media data to be encoded are scaled based on the scale factor. The encoding method is applied to an encoder side device and includes the following steps:

[0150] Step 601: Process media data to be encoded by using a first encoding neural network model to obtain a first latent variable, where the first latent variable indicates a feature of the media data to be encoded.

[0151] The media data to be encoded may be an audio signal, a video signal, an image, etc. In addition, the media data to be encoded may be in any form, which is not limited in this embodiment of the present application.

[0152] For example, the media data to be encoded may be time domain media data, or frequency domain media data obtained through performing a time-frequency transformation on the time domain media data, e.g., frequency domain media data obtained through performing an MDCT on the time domain media data, or frequency domain media data obtained through performing a fast Fourier transformation (FFT) on the time domain media data. Alternatively, the media data to be encoded may be complex frequency domain media data obtained through filtering the time domain media data with a quadrature mirror filter (QMF), the media data to be encoded may be a feature signal extracted from the time domain media data, e.g., Mel-frequency cepstral coefficients, or the media data to be encoded may be a residual signal, e.g., another encoding residual signal, or a residual signal obtained through linear predictive coding (LPC) filtering.

[0153] An implementation process of processing media data to be encoded by using a first encoding neural network model is as follows: inputting the media data to be encoded into the first encoding neural network model to obtain a first latent variable output by the first encoding neural network model; or pre-processing the media data to be encoded and inputting the pre-processed media data into the first encoding neural network model to obtain a first latent variable output by the first encoding neural network model.

[0154] In other words, the media data to be encoded may be used as an input of the first encoding neural network model to determine the first latent variables, or the media data to be encoded may be pre-processed and then used as an input of the first encoding neural network model to determine the first latent variables.

[0155] The pre-processing operations may be temporal noise shaping (TNS) processing, frequency domain noise shaping (FDNS) processing, channel downmix processing, and so on.

[0156] The first encoding neural network model is trained in advance. The network structure and training method of the first encoding neural network model are not limited in this embodiment of the present application. For example, the network structure of the first encoding neural network model may be a fully connected network or a convolutional neural network (CNN) network. In addition, the number of layers included in the network structure of the first encoding neural network model and the number of nodes in each layer are also not limited in this embodiment of the present application.

[0157] The forms of latent variables output by encoding neural network models of different network structures may be different. For example, as shown in FIG. 7, when the network structure of the first encoding neural network model is a fully connected network, the first latent variable is a vector, and the number of dimensions of the vector, M, is the latent size of the latent variable. As shown in FIG. 8, when the network structure of the first encoding neural network model is a CNN network, the first latent variable is an N*M dimensional matrix, where N is the number of channels of the CNN network, and M is the latent size of each channel of the CNN network. It should be noted that FIG. 7 and FIG. 8 only provide a schematic diagram of the latent variables of the fully connected network and a schematic diagram of the latent variables of the CNN network. The channel sequence number may start from 1 or 0, and the element sequence numbers of the latent variables of all channels are the same.

[0158] Step 602: Determine a first variable scale factor based on a first latent variable, where the first variable scale factor is used to enable the number of coding bits of the entropy coding result of the second latent variable to meet a preset encoding rate condition, and the second latent variable is obtained by scaling the first latent variable based on the first variable scale factor.

[0159] In some embodiments, the initial number of coding bits may be determined based on the first latent variable, and the first variable scale factor may be determined based on the initial number of coding bits and the target number of coding bits.

[0160] The target number of coding bits may be preset. Of course, the target number of coding bits may also be determined based on the encoding rate, with different encoding rates corresponding to different target numbers of coding bits.

[0161] In this embodiment of the present application, the media data to be encoded may be encoded at a constant bit rate, or the media data to be encoded may be encoded at a variable bit rate.

[0162] When the media data to be encoded is encoded at a constant bit rate, the number of bits of the media data to be encoded of the current frame may be determined based on the constant bit rate, and then the number of used bits of the current frame is subtracted to obtain a target number of coding bits of the current frame. The number of used bits may be the number of bits for encoding side information, etc. In addition, in general, the side information of each frame of media data is different. Therefore, the target number of coding bits of each frame of media data is usually different.

[0163] When the media data to be encoded is encoded at a variable bit rate, the bit rate is usually specified, and the actual bit rate fluctuates around the specified bit rate. In this case, the number of bits of the media data to be encoded of the current frame may be determined based on the specified bit rate, and then the number of used bits of the current frame is subtracted to obtain a target number of coding bits for the current frame. The number of used bits may be the number of bits for encoding side information, etc. In some cases, the side information of different frames of media data may be different. Thus, the target number of coding bits of different frames of media data are usually different.

[0164] The following will explain and describe in detail the process of determining the initial number of coding bits and the first variable scale factor separately.

[0165] Determine the initial number of coding bits There can be two ways to determine the initial number of coding bits based on the first latent variable, which are described separately below.

[0166] In a first implementation form, the number of coding bits of the entropy coding result of the first latent variable is determined to obtain an initial number of coding bits. In other words, the initial number of coding bits is the number of coding bits of the entropy coding result of the first latent variable.

[0167] In one example, a quantization process is performed on the first latent variable to obtain a quantized first latent variable, an entropy coding is performed on the quantized first latent variable to obtain an initial encoding result of the first latent variable, and a number of coding bits of the initial encoding result of the first latent variable is counted to obtain an initial number of coding bits.

[0168] There may be multiple ways to perform quantization processing on the first latent variable. For example, scalar quantization is performed on each element in the first latent variable. The quantization step of the scalar quantization may be determined based on different encoding rates. In other words, the correspondence between the encoding rate and the quantization step may be stored in advance, and the corresponding quantization step may be obtained from the correspondence based on the encoding rate used in this embodiment of the present application. In addition, the scalar quantization may further have an offset, that is, after the offset processing is performed on the first latent variable based on the offset, the scalar quantization is performed based on the quantization step.

[0169] When the entropy coding is performed on the quantized first latent variable, the entropy coding may be performed based on a scalable entropy coding model, or the entropy coding may be performed by using an entropy coding model with a preset probability distribution. This is not limited in this embodiment of the present application. The entropy coding may use one of arithmetic coding, range coding (RC), and Huffman coding. This is not limited in this embodiment of the present application.

[0170] It should be noted that the following quantization processing method and entropy coding method are the same as those in this specification. For the following quantization processing method and entropy coding method, please refer to the method in this specification. The details will not be described below in this embodiment of the present application.

[0171] In a second implementation, the first latent variable is scaled based on a first initial scale factor, and a number of coding bits of the entropy coding result of the scaled first latent variable is determined to obtain an initial number of coding bits. In other words, the initial number of coding bits is the number of coding bits of the entropy coding result of the first latent variable scaled based on the first initial scale factor. The first initial scale factor may be a first preset scale factor.

[0172] In one example, the first latent variable is scaled based on a first initial scale factor to obtain a scaled first latent variable, and a quantization process is performed on the scaled first latent variable to obtain a quantized first latent variable. Entropy coding is performed on the quantized first latent variable to obtain an initial encoding result of the first latent variable. The number of coding bits of the initial encoding result of the first latent variable is counted to obtain an initial number of coding bits.

[0173] An implementation process for scaling the first latent variable based on the first initial scale factor is to multiply each element in the first latent variable by a corresponding element in the first initial scale factor to obtain a scaled first latent variable as follows:

[0174] It should be noted that the above implementation process is just an example, and in practical application, other methods may be used for scaling. For example, each element in the first latent variable may be divided by the corresponding element in the first initial scale factor to obtain a scaled first latent variable. The scaling method is not limited in this embodiment of the present application.

[0175] It should be noted that in this embodiment of the present application, an initial value of a scale factor is set for the first latent variable, and the initial value of the scale factor is usually equal to 1. The first preset scale factor may be greater than or equal to the initial value of the scale factor, or may be smaller than the initial value of the scale factor. For example, the first preset scale factor is a constant, such as 1 or 2. In the first implementation, when the initial number of coding bits is determined, the first initial scale factor is the initial value of the scale factor. In the second implementation, when the initial number of coding bits is determined, the first initial scale factor is the first preset scale factor.

[0176] In addition, the first initial scale factor may be a scalar or a vector. For example, it is assumed that the network structure of the first encoding neural network model is a fully connected network, the first latent variable output by the first encoding neural network model is a vector, and the number of dimensions M of the vector is the size of the first latent variable. If the first initial scale factor is a scalar, the scale factor values ​​corresponding to all elements in the first latent variable, which has a number of dimensions M and is a vector, are the same, i.e., the first initial scale factor includes one element. If the first initial scale factor is a vector, the scale factor values ​​corresponding to all elements in the first latent variable vector, which has a number of dimensions M and is a vector, are not the same, and multiple elements can share one scale factor value, i.e., the first initial scale factor includes multiple elements, and each element corresponds to one or more elements in the first latent variable.

[0177] Similarly, it is assumed that the network structure of the first encoding neural network model is a CNN network, and the first latent variable output by the first encoding neural network model is an N*M dimensional matrix, where N is the number of channels of the CNN network, and M is the size of the latent variable of each channel of the CNN network. If the first initial scale factor is a scalar, the scale factor values ​​corresponding to all elements in the N*M dimensional first latent variable matrix are the same, i.e., the first initial scale factor contains one element. If the first initial scale factor is a vector, the scale factor values ​​corresponding to all elements in the N*M dimensional first latent variable matrix are not the same, and the elements of the latent variables falling in the same channel can correspond to the same scale factor value, i.e., the first initial scale factor contains N elements, and each element corresponds to M elements having the same channel sequence number in the first latent variable.

[0178] Determine the first variable scale factor The first variable scale factor is a final value of the scale factor of the first latent variable. The second latent variable is obtained by scaling the first latent variable based on the first variable scale factor, and the number of coding bits of the entropy coding result of the second latent variable satisfies a preset encoding rate condition. When the media data to be encoded is encoded at a constant bit rate, satisfying the preset encoding rate condition includes: the number of coding bits is smaller than or equal to a target number of coding bits. Alternatively, satisfying the preset encoding rate condition includes: the number of coding bits is smaller than or equal to a target number of coding bits, and the difference between the number of coding bits and the target number of coding bits is smaller than a bit number threshold. When the media data to be encoded is encoded at a variable bit rate, satisfying the preset encoding rate condition includes: the absolute value of the difference between the number of coding bits and the target number of coding bits is smaller than a bit number threshold. In other words, satisfying the preset encoding rate condition includes: the number of coding bits is smaller than or equal to the target number of coding bits, and the difference between the target number of coding bits and the number of coding bits is smaller than a bit number threshold. Alternatively, satisfying the preset encoding rate condition includes: the number of coding bits is larger than or equal to the target number of coding bits, and the difference between the number of coding bits and the target number of coding bits is smaller than a bit number threshold.

[0179] It should be noted that the bit number threshold may be preset, and the bit number threshold may be adjusted based on different requirements.

[0180] There may be several implementations for determining the first variable scale factor based on the initial number of coding bits and the target number of coding bits. Three implementations are described below.

[0181] In a first implementation, when the initial number of coding bits is equal to the target number of coding bits, the first initial scale factor is determined as the first variable scale factor.When the initial number of coding bits is not equal to the target number of coding bits, the first variable scale factor is determined in a first cyclic manner based on the initial number of coding bits and the target number of coding bits.

[0182] The i-th cyclic process of the first cyclic scheme includes the following steps: determining a scale factor of the i-th cyclic process, where i is a positive integer; scaling the first latent variable based on the scale factor of the i-th cyclic process to obtain an i-th scaled first latent variable; determining a number of coding bits of an entropy coding result of the i-th scaled first latent variable to obtain an i-th number of coding bits; performing the (i+1)-th cyclic process of the first cyclic scheme when the i-th number of coding bits satisfies the continuous scaling condition, or terminating the execution of the first cyclic scheme and determining a first variable scale factor based on the scale factor of the i-th cyclic process when the i-th number of coding bits does not satisfy the continuous scaling condition.

[0183] An implementation process of determining a scale factor of an i-th cyclic process is to determine a scale factor of the i-th cyclic process based on the scale factor of the (i-1)-th cyclic process of a first cyclic scheme, the (i-1)-th number of coding bits, and the target number of coding bits as follows: When i=1, the scale factor of the (i-1)-th cyclic process is a first initial scale factor, and the (i-1)-th number of coding bits is the initial number of coding bits.

[0184] In this case, the successive scaling condition includes both the (i-1)th number of coding bits and the i-th number of coding bits being less than the target number of coding bits, or the successive scaling condition includes both the (i-1)th number of coding bits and the i-th number of coding bits being greater than the target number of coding bits.

[0185] In other words, the continuous scaling condition includes that the i-th number of coding bits does not exceed the target number of coding bits. In this specification, "does not exceed" means that the number of coding bits for the first (i-1) times is always smaller than the target number of coding bits, and the i-th number of coding bits continues to be smaller than the target number of coding bits. Alternatively, the number of coding bits for the first (i-1) times is always larger than the target number of coding bits, and the i-th number of coding bits continues to be larger than the target number of coding bits. Conversely, "exceeds" means that the number of coding bits for the first (i-1) times is always smaller than the target number of coding bits, and the i-th number of coding bits is larger than the target number of coding bits. Alternatively, the number of coding bits for the first (i-1) times is always larger than the target number of coding bits, and the i-th number of coding bits is smaller than the target number of coding bits.

[0186] In one example, when the scale factor is a quantized value, the scale factor of the i-th cyclic process may be determined based on the scale factor of the (i-1)-th cyclic process of the first cyclic scheme, the (i-1)-th number of coding bits, and the target number of coding bits according to the following equation (1): scale(i)=Q{scale(i-1)*[target / curr(i-1)]} (1)

[0187] In the above formula (1), scale(i) is the scale factor of the i-th cyclic process, scale(i-1) is the scale factor of the (i-1)-th cyclic process, target is the target number of coding bits, and curr(i-1) is the (i-1)-th number of coding bits. i is a positive integer greater than 0. Q{x} is to obtain the quantized value of x.

[0188] In addition, when the scale factor is a non-quantized value, when the scale factor of the i-th cyclic processing is determined according to the above equation (1), the right-hand side of the above equation (1) does not need to be processed by using Q{x}.

[0189] An implementation process for determining a first variable scale factor based on a scale factor of the i-th cyclic process includes determining the scale factor of the i-th cyclic process as the first variable scale factor when the i-th number of coding bits is equal to a target number of coding bits, or determining the first variable scale factor based on the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process when the i-th number of coding bits is not equal to the target number of coding bits.

[0190] In other words, the scale factor of the i-th cyclic process is the scale factor last obtained in the first cyclic manner, and the i-th number of coding bits is the number of coding bits last obtained. When the number of coding bits last obtained is equal to the target number of coding bits, the scale factor last obtained is determined as the first variable scale factor. When the number of coding bits last obtained is not equal to the target number of coding bits, the first variable scale factor is determined based on the scale factors obtained in the last two times.

[0191] In some embodiments, an implementation process for determining a first variable scale factor based on a scale factor of the i-th cyclic process and a scale factor of the (i-1)-th cyclic process includes determining an average value of the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process, and determining the first variable scale factor based on the average value.

[0192] In one example, the average value may be directly determined as the first variable scale factor, or the average value may be multiplied by a preset constant to obtain the first variable scale factor. Optionally, the constant may be less than 1.

[0193] In some other embodiments, the implementation process for determining a first variable scale factor based on a scale factor of the i-th cyclic process and a scale factor of the (i-1)-th cyclic process includes determining the first variable scale factor in a second cyclic manner based on the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process.

[0194] In one example, the jth cyclic process of the second cyclic scheme includes the following steps: determining a third scale factor of the jth cyclic process based on a first scale factor of the jth cyclic process and a second scale factor of the jth cyclic process, where when j is equal to 1, the first scale factor of the jth cyclic process is one of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, and the second scale factor of the jth cyclic process is the other of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, the first scale factor of the jth cyclic process corresponds to a jth first number of coding bits, the second scale factor of the jth cyclic process corresponds to a jth second number of coding bits, the jth first number of coding bits is a number of coding bits of an entropy coding result of a first latent variable scaled based on the first scale factor of the jth cyclic process, and the jth third scale factor of the coding bits is a number of coding bits of an entropy coding result of a first latent variable scaled based on the first scale factor of the jth cyclic process. determining that the second number is a number of coding bits of the entropy coding result of the first latent variable scaled based on the second scale factor of the jth cyclic process, the jth first number of coding bits being smaller than the jth second number of coding bits; obtaining a jth third number of coding bits, the jth third number of coding bits being a number of coding bits of the entropy coding result of the first latent variable scaled based on the third scale factor of the jth cyclic process; if the jth third number of coding bits does not satisfy a consecutive cycle condition, terminating execution of the second cyclic scheme and determining the third scale factor of the jth cyclic process as the first variable scale factor; if the jth third number of coding bits satisfies the consecutive cycle condition, is greater than the target number of coding bits and is smaller than the jth second number of coding bits, using the third scale factor of the jth cyclic process as the second scale factor of the (j+1)th cyclic process;and performing the (j+1)th cyclic process of the second cyclic scheme by using the first scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process, or performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process and using the second scale factor of the jth cyclic process as the second scale factor of the (j+1)th cyclic process if the jth third number of coding bits satisfies a consecutive cycle condition, is smaller than the target number of coding bits and is larger than the jth first number of coding bits.

[0195] In another example, the jth cyclic process of the second cyclic scheme includes the steps of: determining a third scale factor of the jth cyclic process based on the first scale factor of the jth cyclic process and the second scale factor of the jth cyclic process, where when j is equal to 1, the first scale factor of the jth cyclic process is one of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, and the second scale factor of the jth cyclic process is the other of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, the first scale factor of the jth cyclic process corresponds to the jth first number of coding bits, the second scale factor of the jth cyclic process corresponds to the jth second number of coding bits, the jth first number of coding bits is smaller than the jth second number of coding bits, and j is a positive integer; and obtaining the jth third number of coding bits. obtaining a j-th third number of coding bits, the number of coding bits being a number of coding bits of an entropy coding result of the first latent variable scaled based on the third scale factor of the j-th cyclic process; terminating execution of the second cyclic scheme and determining the third scale factor of the j-th cyclic process as a first variable scale factor if the j-th third number of coding bits does not satisfy a consecutive cycle condition; terminating execution of the second cyclic scheme and determining the first variable scale factor based on the first scale factor of the j-th cyclic process if j has not reached a maximum number of cycles and the j-th third number of coding bits satisfies the consecutive cycle condition; using the third scale factor of the j-th cyclic process as a second scale factor of the (j+1)-th cyclic process if j has not reached a maximum number of cycles and the j-th third number of coding bits satisfies the consecutive cycle condition and is greater than a target number of coding bits and less than the j-th second number of coding bits;and performing the (j+1)th cyclic process of the second cyclic scheme by using the first scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process, or performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process and using the second scale factor of the jth cyclic process as the second scale factor of the (j+1)th cyclic process if j has not reached the maximum number of cycles and the jth third number of coding bits satisfies a consecutive cycle condition, is smaller than the target number of coding bits and is larger than the jth first number of coding bits.

[0196] An implementation process for determining a third scale factor of a jth cyclic process based on a first scale factor of the jth cyclic process and a second scale factor of the jth cyclic process includes: determining an average value of the first scale factor of the jth cyclic process and the second scale factor of the jth cyclic process; and determining a third scale factor of the jth cyclic process based on the average value. In one example, the average value may be directly determined as the third scale factor of the jth cyclic process, or the average value may be multiplied by a preset constant to obtain the third scale factor of the jth cyclic process. Optionally, the constant may be less than 1.

[0197] In addition, an implementation process for obtaining the jth third number of coding bits includes scaling the first latent variable based on the third scale factor of the jth cyclic process to obtain a scaled first latent variable, performing a quantization process on the scaled first latent variable to obtain a quantized first latent variable, performing entropy coding on the quantized first latent variable, and counting the number of coding bits of the entropy coding result to obtain the jth third number of coding bits.

[0198] When the media data to be encoded is encoded at a constant bit rate, an implementation process for determining a first variable scale factor based on a first scale factor of a jth cyclic process includes determining the first scale factor of the jth cyclic process as the first variable scale factor. When the media data to be encoded is encoded at a variable bit rate, an implementation process for determining a first variable scale factor based on a first scale factor of a jth cyclic process includes determining a first difference between a target number of coding bits and a jth first number of coding bits, determining a second difference between the jth second number of coding bits and the target number of coding bits, and determining the first scale factor of the jth cyclic process as the first variable scale factor if the first difference is smaller than the second difference, determining the second scale factor of the jth cyclic process as the first variable scale factor if the second difference is smaller than the first difference, or determining the first scale factor of the jth cyclic process as the first variable scale factor or determining the second scale factor of the jth cyclic process as the first variable scale factor if the first difference is equal to the second difference.

[0199] When the media data to be encoded is encoded at a constant bit rate, the successive cycle condition includes that the jth third number of coding bits is greater than the target number of coding bits, or the successive cycle condition includes that the jth third number of coding bits is smaller than the target number of coding bits, and the difference between the target number of coding bits and the jth third number of coding bits is greater than the bit number threshold. When the media data to be encoded is encoded at a variable bit rate, the successive cycle condition includes that the absolute value of the difference between the target number of coding bits and the jth third number of coding bits is greater than the bit number threshold. In other words, the successive cycle condition includes that the jth third number of coding bits is greater than the target number of coding bits, and the difference between the jth third number of coding bits and the target number of coding bits is greater than the bit number threshold, or the successive cycle condition includes that the jth third number of coding bits is smaller than the target number of coding bits, and the difference between the target number of coding bits and the jth third number of coding bits is greater than the bit number threshold.

[0200] In other words, when the media data to be encoded is encoded at a constant bit rate, the successive cycle condition is bits_curr>target||(bits_curr<target&&(target-bits_curr)> TH), and when the media data to be encoded is encoded at a variable bit rate, the successive cycle condition is bits_curr>target&&(bits_curr-target)>TH||(bits_curr<target&&(target-bits_curr)> TH), where bits_curr is the jth third number of coding bits, target is the target number of coding bits, and TH is the bit number threshold.

[0201] The first scale factor of the jth cyclic process may be denoted as scale_lower, the jth first number of coding bits may be denoted as bits_lower, the second scale factor of the jth cyclic process may be denoted as scale_upper, the jth second number of coding bits may be denoted as bits_upper, and the third scale factor of the jth cyclic process may be denoted as scale_curr; and further, pseudocode for using the third scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process or the second scale factor of the (j+1)th cyclic process is as follows: if bits_curr>target&bits_curr <bits_upper scale_upper=scale_curr bits_upper=bits_curr else if bits_curr<target&bits_curr> bits_lower scale_lower=scale_curr bits_lower = bits_curr.

[0202] In the second implementation, when the initial number of coding bits is equal to the target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the initial number of coding bits is not equal to the target number of coding bits, the first variable scale factor is determined according to the above-mentioned first implementation. However, the difference from the above-mentioned first implementation is that when the initial number of coding bits is smaller than the target number of coding bits, the scale factor of the (i-1)th cyclic process is scaled based on the first step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling conditions include that the i-th number of coding bits is smaller than the target number of coding bits. When the initial number of coding bits is larger than the target number of coding bits, the scale factor of the (i-1)th cyclic process is scaled based on the second step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling conditions include that the i-th number of coding bits is larger than the target number of coding bits.

[0203] Scaling the scale factor of the (i-1)th cyclic process based on a first step may refer to increasing the scale factor of the (i-1)th cyclic process based on the first step, and scaling the scale factor of the (i-1)th cyclic process based on a second step may refer to decreasing the scale factor of the (i-1)th cyclic process based on the second step.

[0204] The increase and decrease processes may be linear or nonlinear. For example, the sum of the scale factor of the (i-1)th cyclic process and the first step may be determined as the scale factor of the i-th cyclic process, or the difference between the scale factor of the (i-1)th cyclic process and the second step may be determined as the scale factor of the i-th cyclic process.

[0205] It should be noted that the first step and the second step may be preset, and the first step and the second step may be adjusted based on different requirements. In addition, the first step may be equal or unequal to the second step.

[0206] In a third implementation, when the initial number of coding bits is smaller than or equal to the target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the initial number of coding bits is greater than the target number of coding bits, the first variable scale factor may be determined according to the first implementation described above. Alternatively, the first variable scale factor may be determined in the second implementation described above when the initial number of coding bits is greater than the target number of coding bits.

[0207] Step 603: Obtain the entropy coding result of the second latent variable.

[0208] In the process of determining the first variable scale factor in step 602, the first variable scale factor may be the first initial scale factor, may be the scale factor of the i-th cyclic process, may be the third scale factor of the j-th cyclic process, or may be determined and obtained based on the first scale factor of the j-th cyclic process. In any case, the corresponding entropy coding result is determined in the aforementioned cyclic process. Therefore, the entropy coding result corresponding to the first variable scale factor, that is, the entropy coding result of the second latent variable, can be directly obtained from the entropy coding result determined in the aforementioned cyclic process.

[0209] Of course, the process may be performed again. In other words, the first latent variable is directly scaled based on the first variable scale factor to obtain the second latent variable. A quantization process is performed on the second latent variable to obtain a quantized second latent variable. Entropy coding is performed on the quantized second latent variable to obtain an entropy coding result of the second latent variable.

[0210] Based on the above description, the first initial scale factor may be a scalar or a vector, and the first variable scale factor is determined based on the first initial scale factor. Thus, the first variable scale factor may be a scalar or a vector.

[0211] In other words, the first variable scaling factor includes one or more elements, and further, when the first variable scaling factor includes multiple elements, one element in the first variable scaling factor corresponds to one or more elements in the first latent variable.

[0212] Step 604: Write the entropy coding result of the second latent variable and the encoding result of the first variable scale factor into a bitstream.

[0213] Based on the above description, the first variable scale factor may be a non-quantized value or a quantized value. When the first variable scale factor is a non-quantized value, the implementation process of determining the encoding result of the first variable scale factor includes performing a quantization and encoding process on the first variable scale factor to obtain the encoding result of the first variable scale factor. When the first variable scale factor is a quantized value, the implementation process of determining the encoding result of the first variable scale factor includes encoding the first variable scale factor to obtain the encoding result of the first variable scale factor.

[0214] The first variable scale factor may be encoded in any encoding manner, which is not limited in this embodiment of the present application.

[0215] In addition, in some encoders, the quantization process and the encoding process are implemented in one process, that is, the quantization result and the encoding result can be obtained in one process. Therefore, for the first variable scale factor, the encoding result of the first variable scale factor can also be obtained in the above process of determining the first variable scale factor. Therefore, the encoding result of the first variable scale factor can be directly obtained. In other words, the encoding result of the first variable scale factor can also be directly obtained in the process of determining the first variable scale factor.

[0216] Optionally, when the first variable scale factor is a non-quantized value, in this embodiment of the present application, a quantization index corresponding to a quantization step of the first variable scale factor may be further determined to obtain a first quantization index. Optionally, in this embodiment of the present application, a quantization index corresponding to a quantization step of the second latent variable may be further determined to obtain a second quantization index. The first quantization index and the second quantization index are encoded into the bitstream. The quantization index indicates a corresponding quantization step, i.e., the first quantization index indicates the quantization step of the first variable scale factor, and the second quantization index indicates the quantization step of the second latent variable.

[0217] In this embodiment of the present application, the first latent variable is scaled according to a first variable scale factor to obtain a second latent variable, and the number of coding bits of the entropy coding result of the second latent variable meets a preset encoding rate condition. This ensures that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can meet the preset encoding rate condition, that is, the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can basically keep consistency instead of changing dynamically, thereby meeting the encoder's requirement for a stable encoding rate. Furthermore, when side information (e.g., window type, Temporal Noise Shaping (TNS) parameters, Frequency-domain noise shaping (FDNS) parameters, and / or Bandwidth extension (BWE) parameters) needs to be transmitted, it can be guaranteed that the number of coding bits of the entropy coding result of the latent variables corresponding to each frame of media data is essentially consistent with the number of coding bits of the side information, thereby meeting the encoder's requirement for a stable encoding rate.

[0218] 9 is a flowchart of a first decoding method according to an embodiment of the present application. The method is applied on the decoder side. The method corresponds to the encoding method shown in FIG. 6. The method includes the following steps:

[0219] Step 901: Determine a reconstructed second latent variable and a reconstructed first variable scale factor based on a bitstream.

[0220] In some embodiments, entropy decoding may be performed on the entropy coding result of the second latent variable in the bitstream, and the encoding result of the first variable scale factor in the bitstream may be decoded to obtain the quantized second latent variable and the quantized first variable scale factor. An inverse quantization process is performed on the quantized second latent variable and the quantized first variable scale factor to obtain the reconstructed second latent variable and the reconstructed first variable scale factor.

[0221] The decoding method of this step corresponds to the encoding method on the encoder side, and the inverse quantization process of this step corresponds to the quantization process on the encoder side, in other words, the decoding method is the inverse process of the encoding method, and the inverse quantization process is the inverse process of the quantization process.

[0222] For example, the bitstream may be parsed to obtain a first quantization index and a second quantization index, the first quantization index indicating a quantization step of the first variable scale factor, and the second quantization index indicating a quantization step of the second latent variable, an inverse quantization process is performed on the quantized first variable scale factor based on the quantization step indicated by the first quantization index to obtain a reconstructed first variable scale factor, and an inverse quantization process is performed on the quantized second latent variable based on the quantization step indicated by the second quantization index to obtain a reconstructed second latent variable.

[0223] Step 902: Scale the reconstructed second latent variable based on the reconstructed first variable scale factor to obtain a reconstructed first latent variable, where the reconstructed first latent variable indicates a characteristic of the media data to be decoded.

[0224] Since the second latent variable is obtained by scaling the first latent variable based on the first variable scaling factor, the reconstructed second latent variable may be scaled based on the reconstructed first variable scaling factor to obtain the reconstructed first latent variable.

[0225] The process of scaling the reconstructed second latent variable based on the reconstructed first variable scale factor is the inverse process of scaling the first latent variable by the encoder side. For example, when the encoder side multiplies each element in the first latent variable by the corresponding element in the first variable scale factor, each element in the reconstructed second latent variable may be divided by the corresponding element in the reconstructed first variable scale factor to obtain the reconstructed first latent variable. When the encoder side divides each element in the first latent variable by the corresponding element in the first variable scale factor, each element in the reconstructed second latent variable may be multiplied by the corresponding element in the reconstructed first variable scale factor to obtain the reconstructed first latent variable.

[0226] Based on the above description, the first initial scale factor may be a scalar or a vector. Therefore, the finally obtained first variable scale factor may be a scalar or a vector. For example, it is assumed that the network structure of the first encoding neural network model is a fully connected network, the second latent variable is a vector, and the number of dimensions M of the vector is the size of the second latent variable. If the first variable scale factor is a scalar, the scale factor values ​​corresponding to all elements in the second latent variable, the number of dimensions of which is M and which is a vector, are the same, i.e., the first variable scale factor includes one element. If the first variable scale factor is a vector, the scale factor values ​​corresponding to all elements in the second latent variable vector, the number of dimensions of which is M and which is a vector, are not the same, and multiple elements can share one scale factor value, i.e., the first variable scale factor includes multiple elements, and each element corresponds to one or more elements in the second latent variable.

[0227] Similarly, it is assumed that the network structure of the first encoding neural network model is a CNN network, and the second latent variable is an N*M dimensional matrix, where N is the number of channels of the CNN network, and M is the size of the latent variable of each channel of the CNN network. If the first variable scale factor is a scalar, the scale factor values ​​corresponding to all elements in the N*M dimensional second latent variable matrix are the same, i.e., the first variable scale factor includes one element. If the first variable scale factor is a vector, the scale factor values ​​corresponding to all elements in the N*M dimensional second latent variable matrix are not the same, and elements of the latent variable that fall in the same channel can correspond to the same scale factor value, i.e., the first variable scale factor includes N elements, and each element corresponds to M elements with the same channel sequence number in the second latent variable.

[0228] Step 903: Process the reconstructed first latent variables by using a first decoding neural network model to obtain reconstructed media data.

[0229] In some embodiments, the reconstructed first latent variables may be input to the first decoding neural network model to obtain the reconstructed media data output by the first decoding neural network model. Alternatively, the reconstructed first latent variables are post-processed and the post-processed first latent variables are input to the first decoding neural network model to obtain the reconstructed media data output by the first decoding neural network model.

[0230] In other words, the reconstructed first latent variables may be used as inputs of a first decoding neural network model to determine the reconstructed media data, or the reconstructed first latent variables may be post-processed and then used as inputs of a first decoding neural network model to determine the reconstructed media data.

[0231] The first decoding neural network model corresponds to the first encoding neural network model, and both are pre-trained. The network structure and training method of the first decoding neural network model are not limited in this embodiment of the application. For example, the network structure of the first decoding neural network model may be a fully connected network or a CNN network. In addition, the number of layers included in the network structure of the first decoding neural network model and the number of nodes in each layer are not limited in this embodiment of the application.

[0232] When the media data are audio and video signals, the output of the first decoding neural network model may be reconstructed time-domain media data or may be reconstructed frequency-domain media data. If the output is frequency-domain media data, a frequency-domain to time-domain transformation needs to be performed to obtain the time-domain media data. Alternatively, the output of the first decoding neural network model may be a residual signal. In this case, other corresponding processing needs to be performed to obtain the audio or video signal.

[0233] In this embodiment of the present application, the number of coding bits of the entropy coding result of the second latent variable meets the preset encoding rate condition. This ensures that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can meet the preset encoding rate condition, that is, the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can basically keep consistency instead of dynamically changing, thereby meeting the encoder's requirement for a stable encoding rate. Furthermore, when side information (e.g., window type, Temporal Noise Shaping (TNS) parameter, Frequency-domain noise shaping (FDNS) parameter, and / or Bandwidth Extension (BWE) parameter) needs to be transmitted, it can be guaranteed that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data is basically consistent with the number of coding bits of the side information, thereby meeting the encoder's requirement for a stable encoding rate.

[0234] 10 is a flowchart of a second encoding method according to an embodiment of the present application. This method includes a context model, but only the latent variables generated by using the media data to be encoded are scaled based on a scale factor. The encoding method is applied to an encoder-side device and includes the following steps:

[0235] Step 1001: Process media data to be encoded by using a first encoding neural network model to obtain a first latent variable, where the first latent variable indicates a feature of the media data to be encoded.

[0236] For the implementation process of step 1001, please refer to the implementation process of step 601. The details will not be repeated here.

[0237] Step 1002: Determine a first variable scale factor based on a first latent variable.

[0238] In some embodiments, the initial number of coding bits may be determined based on the first latent variable, and the first variable scale factor may be determined based on the initial number of coding bits and the target number of coding bits. For a related description of the target number of coding bits, please refer to the content of step 602. The details will not be repeated here. The following will separately describe and describe in detail the process of determining the initial number of coding bits and the first variable scale factor.

[0239] Determine the initial number of coding bits There can be two ways to determine the initial number of coding bits based on the first latent variable, which are described separately below.

[0240] In a first implementation form, a corresponding initial number of context coding bits and initial entropy coding model parameters are determined based on a first latent variable by using a context model, a number of coding bits of an entropy coding result of the first latent variable is determined based on the initial entropy coding model parameters to obtain a basic initial number of coding bits, and the initial number of coding bits is determined based on the initial number of context coding bits and the basic initial number of coding bits.

[0241] The context model includes a context encoding neural network model and a context decoding neural network model. An implementation process of determining a corresponding initial number of context coding bits and initial entropy coding model parameters based on a first latent variable by using the context model includes: processing the first latent variable by using the context encoding neural network model to obtain a fifth latent variable, where the fifth latent variable indicates a probability distribution of the first latent variable; determining an entropy coding result of the fifth latent variable, and using a number of coding bits of the entropy coding result of the fifth latent variable as the initial number of context coding bits; and processing the fifth latent variable obtained through the reconstruction by using the context decoding neural network model to reconstruct the fifth latent variable based on the entropy coding result of the fifth latent variable and obtain the initial entropy coding model parameters.

[0242] In one example, the first latent variable is processed by using a context encoding neural network model to obtain a fifth latent variable. A quantization process is performed on the fifth latent variable to obtain a quantized fifth latent variable. An entropy coding is performed on the quantized fifth latent variable, and a number of coding bits of the entropy coding result is counted to obtain an initial number of context coding bits. Then, an entropy decoding is performed on the entropy coding result of the fifth latent variable to obtain a quantized fifth latent variable, and an inverse quantization process is performed on the quantized fifth latent variable to obtain a reconstructed fifth latent variable. The reconstructed fifth latent variable is input to the context decoding neural network model to obtain initial entropy coding model parameters output by the context decoding neural network model.

[0243] An implementation process of processing the first latent variable by using the context encoding neural network model is as follows: inputting the first latent variable into the context encoding neural network model to obtain a fifth latent variable output by the context encoding neural network model; or obtaining the absolute value of each element in the first latent variable, and then inputting the absolute value of each element in the first latent variable into the context encoding neural network model to obtain the fifth latent variable output by the context encoding neural network model.

[0244] An implementation process of determining a number of coding bits of an entropy coding result of a first latent variable based on initial entropy coding model parameters to obtain a basic initial number of coding bits is as follows: determining an entropy coding model corresponding to the initial entropy coding model parameters from an entropy coding model having scalable encoding model parameters; performing a quantization process on the first latent variable to obtain a quantized first latent variable; performing entropy coding on the quantized first latent variable based on the entropy coding model corresponding to the initial entropy coding model parameters to obtain an initial encoding result of the first latent variable; and counting the number of coding bits of the initial encoding result of the first latent variable to obtain a basic initial number of coding bits.

[0245] For the quantization processing manner and the entropy coding manner in step 1002, please refer to the quantization processing manner and the entropy coding manner in step 602. The details will not be repeated here.

[0246] The implementation process of determining the initial number of coding bits based on the initial number of context coding bits and the basic initial number of coding bits includes: determining the sum of the initial number of context coding bits and the basic initial number of coding bits as the initial number of coding bits. Of course, the initial number of coding bits may be further determined in another implementation.

[0247] In a second implementation, the first latent variable is scaled based on a first initial scale factor, and a corresponding initial number of context coding bits and an initial entropy coding model parameter are determined based on the scaled first latent variable by using a context model, and a number of coding bits of an entropy coding result of the scaled first latent variable is determined based on the initial entropy coding model parameter to obtain a basic initial number of coding bits, and the initial number of coding bits is determined based on the initial number of context coding bits and the basic initial number of coding bits. The first initial scale factor may be a first preset scale factor.

[0248] The context model includes a context encoding neural network model and a context decoding neural network model. An implementation process of determining a corresponding initial number of context coding bits and initial entropy coding model parameters based on the scaled first latent variable by using the context model includes: processing the scaled first latent variable by using the context encoding neural network model to obtain a sixth latent variable, where the sixth latent variable indicates a probability distribution of the scaled first latent variable; determining an entropy coding result of the sixth latent variable, and using a number of coding bits of the entropy coding result of the sixth latent variable as the initial number of context coding bits; and processing the sixth latent variable obtained through the reconstruction by using the context decoding neural network model to reconstruct the sixth latent variable based on the entropy coding result of the sixth latent variable and obtain the initial entropy coding model parameters.

[0249] In one example, the scaled first latent variable is processed by using a context encoding neural network model to obtain a sixth latent variable. A quantization process is performed on the sixth latent variable to obtain a quantized sixth latent variable. An entropy coding is performed on the quantized sixth latent variable, and the number of coding bits of the entropy coding result is counted to obtain an initial number of context coding bits. Then, an entropy decoding is performed on the entropy coding result of the sixth latent variable to obtain a quantized sixth latent variable, and an inverse quantization process is performed on the quantized sixth latent variable to obtain a reconstructed sixth latent variable. The reconstructed sixth latent variable is input to the context decoding neural network model to obtain initial entropy coding model parameters output by the context decoding neural network model.

[0250] Please note that for other contents in the second implementation form, please refer to the above description of the corresponding steps, and the details will not be repeated here.

[0251] Determine the first variable scale factor The first variable scale factor is a final value of the scale factor of the first latent variable. The second latent variable is obtained by scaling the first latent variable based on the first variable scale factor, and the third latent variable is obtained by processing the second latent variable by using a context encoding neural network model, and a total number of coding bits of the entropy coding result of the second latent variable and the coding bits of the entropy coding result of the third latent variable meets a preset encoding rate condition. When the media data to be encoded is encoded at a constant bit rate, meeting the preset encoding rate condition includes: the number of coding bits is smaller than or equal to a target number of coding bits. Alternatively, meeting the preset encoding rate condition includes: the number of coding bits is smaller than or equal to a target number of coding bits, and a difference between the number of coding bits and the target number of coding bits is smaller than a bit number threshold. When the media data to be encoded is encoded at a variable bit rate, satisfying the preset encoding rate condition includes that the absolute value of the difference between the number of coding bits and the target number of coding bits is smaller than the bit number threshold. In other words, satisfying the preset encoding rate condition includes that the number of coding bits is smaller than or equal to the target number of coding bits, and that the difference between the target number of coding bits and the number of coding bits is smaller than the bit number threshold. Alternatively, satisfying the preset encoding rate condition includes that the number of coding bits is larger than or equal to the target number of coding bits, and that the difference between the number of coding bits and the target number of coding bits is smaller than the bit number threshold.

[0252] It should be noted that the bit number threshold may be preset, and the bit number threshold may be adjusted based on different requirements.

[0253] There may be several implementations for determining the first variable scale factor based on the initial number of coding bits and the target number of coding bits. Three implementations are described below.

[0254] In a first implementation, when the initial number of coding bits is equal to the target number of coding bits, the first initial scale factor is determined as the first variable scale factor.When the initial number of coding bits is not equal to the target number of coding bits, the first variable scale factor is determined in a first cyclic manner based on the initial number of coding bits and the target number of coding bits.

[0255] The i-th cyclic process of the first cyclic scheme includes the steps of: determining a scale factor of the i-th cyclic process, where i is a positive integer; scaling the first latent variable based on the scale factor of the i-th cyclic process to obtain an i-th scaled first latent variable; determining a corresponding i-th number of context coding bits and an i-th entropy coding model parameter based on the i-th scaled first latent variable by using a context model; and determining an i-th entropy coding model parameter based on the i-th scaled first latent variable by using a context model to obtain an i-th base number of coding bits. determining a number of coding bits of an entropy coding result of the i-th scaled first latent variable based on the parameters; determining the i-th number of coding bits based on the i-th number of context coding bits and the i-th base number of coding bits; and performing an (i+1)-th cyclic process of the first cyclic scheme when the i-th number of coding bits satisfies the continuous scaling condition, or terminating the execution of the first cyclic scheme and determining a first variable scale factor based on the scale factor of the i-th cyclic process when the i-th number of coding bits does not satisfy the continuous scaling condition.

[0256] For an implementation process of determining the scale factor of the i-th cyclic processing, please refer to the relevant description of step 602. Details will not be repeated here. For an implementation process of determining the i-th number of context coding bits and the i-th entropy coding model parameters based on the i-th scaled first latent variable by using a context model, please refer to the aforementioned process of determining the initial number of context coding bits and the initial entropy coding model parameters. Details will not be repeated here. For an implementation process of determining the number of coding bits of the entropy coding result of the i-th scaled first latent variable based on the i-th entropy coding model parameters, please refer to the aforementioned process of determining the basic initial number of coding bits based on the initial entropy coding model parameters. Details will not be repeated here.

[0257] For an implementation process of determining the first variable scale factor based on the scale factor of the i-th cyclic process, please refer to the description of step 602. The details will not be repeated here.

[0258] In the second implementation, when the initial number of coding bits is equal to the target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the initial number of coding bits is not equal to the target number of coding bits, the first variable scale factor is determined according to the above-mentioned first implementation. However, the difference from the above-mentioned first implementation is that when the initial number of coding bits is smaller than the target number of coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the first step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling conditions include that the i-th number of coding bits is smaller than the target number of coding bits. When the initial number of coding bits is larger than the target number of coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the second step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling conditions include that the i-th number of coding bits is larger than the target number of coding bits.

[0259] Scaling the scale factor of the (i-1)th cyclic process based on a first step may refer to increasing the scale factor of the (i-1)th cyclic process based on the first step, and scaling the scale factor of the (i-1)th cyclic process based on a second step may refer to decreasing the scale factor of the (i-1)th cyclic process based on the second step.

[0260] The increase and decrease processes may be linear or nonlinear. For example, the sum of the scale factor of the (i-1)th cyclic process and the first step may be determined as the scale factor of the i-th cyclic process, or the difference between the scale factor of the (i-1)th cyclic process and the second step may be determined as the scale factor of the i-th cyclic process.

[0261] It should be noted that the first step and the second step may be preset, and the first step and the second step may be adjusted based on different requirements. In addition, the first step may be equal or unequal to the second step.

[0262] In a third implementation, when the initial number of coding bits is smaller than or equal to the target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the initial number of coding bits is greater than the target number of coding bits, the first variable scale factor may be determined according to the first implementation described above. Alternatively, the first variable scale factor may be determined in the second implementation described above when the initial number of coding bits is greater than the target number of coding bits.

[0263] Step 1003: Obtain an entropy coding result of a second latent variable and an entropy coding result of a third latent variable, where the second latent variable is obtained by scaling the first latent variable based on a first variable scale factor, and the third latent variable indicates a probability distribution of the second latent variable, and the total number of coding bits of the entropy coding result of the second latent variable and the coding bits of the entropy coding result of the third latent variable satisfies a preset encoding rate condition.

[0264] In the process of determining the first variable scale factor in step 1002, the first variable scale factor may be the first initial scale factor, may be the scale factor of the i-th cyclic process, may be the third scale factor of the j-th cyclic process, or may be determined based on the first scale factor of the j-th cyclic process. In any case, the corresponding entropy coding result is determined in the aforementioned cyclic process. Therefore, the entropy coding result of the second latent variable and the entropy coding result of the third latent variable can be directly obtained from the entropy coding result determined in the aforementioned cyclic process.

[0265] Of course, the process may be performed again. In other words, the first latent variable is directly scaled based on the first variable scale factor to obtain the second latent variable. The entropy coding result of the third latent encoding and the first entropy coding model parameters are determined based on the second latent variable by using the context model. A quantization process is performed on the second latent variable to obtain a quantized second latent variable. The entropy coding result of the quantized second latent variable, i.e., the entropy coding result of the second latent variable, is determined based on the first entropy coding model parameters.

[0266] The context models include a context encoding neural network model and a context decoding neural network model. An implementation process of determining an entropy coding result of a third latent encoding and first entropy coding model parameters based on a second latent variable by using a context model includes: processing the second latent variable by using the context encoding neural network model to obtain a third latent variable; performing a quantization operation on the third latent variable to obtain a quantized third latent variable; performing entropy coding on the quantized third latent variable to obtain an entropy coding result of the third latent variable; performing entropy decoding on the entropy coding result of the third latent variable to obtain the quantized third latent variable; performing an inverse quantization operation on the quantized third latent variable to obtain a reconstructed third latent variable; and processing the reconstructed third latent variable by using the context decoding neural network model to obtain the first entropy coding model parameters.

[0267] An implementation process for determining an entropy coding result of a quantized second latent variable based on first entropy coding model parameters includes determining an entropy coding model corresponding to the first entropy coding model parameters from an entropy coding model having scalable encoding model parameters, and performing entropy coding on the quantized second latent variable based on the entropy coding model corresponding to the first entropy coding model parameters to obtain an entropy coding result of the second latent variable.

[0268] Step 1004: Write the entropy coding result of the second latent variable, the entropy coding result of the third latent variable, and the encoding result of the first variable scale factor into a bitstream.

[0269] For the relevant content of the encoding result of the first variable scale factor, please refer to the description of step 604. The details will not be repeated here.

[0270] Optionally, when the first variable scale factor is a non-quantized value, in this embodiment of the present application, a quantization index corresponding to the quantization step of the first variable scale factor may be further determined to obtain a first quantization index. Optionally, in this embodiment of the present application, a quantization index corresponding to the quantization step of the second latent variable may be further determined to obtain a second quantization index, and a quantization index corresponding to the quantization step of the third latent variable may be further determined to obtain a third quantization index. The first quantization index, the second quantization index, and the third quantization index are encoded into the bitstream. The quantization indexes indicate corresponding quantization steps, i.e., the first quantization index indicates the quantization step of the first variable scale factor, the second quantization index indicates the quantization step of the second latent variable, and the third quantization index indicates the quantization step of the third latent variable.

[0271] In this embodiment of the present application, a first latent variable is scaled based on a first variable scale factor to obtain a second latent variable, and the second latent variable is processed by using a context encoding neural network model to obtain a third latent variable, and the total number of coding bits of the entropy coding result of the second latent variable and the coding bits of the entropy coding result of the third latent variable meets a preset encoding rate condition. This ensures that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can meet the preset encoding rate condition, that is, the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can basically keep consistency instead of changing dynamically, thereby meeting the encoder's requirement for a stable encoding rate. Furthermore, when side information (e.g., window type, Temporal Noise Shaping (TNS) parameters, Frequency-domain noise shaping (FDNS) parameters, and / or Bandwidth extension (BWE) parameters) needs to be transmitted, it can be guaranteed that the number of coding bits of the entropy coding result of the latent variables corresponding to each frame of media data is essentially consistent with the number of coding bits of the side information, thereby meeting the encoder's requirement for a stable encoding rate.

[0272] Figure 11 is a flow chart of a second decoding method according to an embodiment of the present application. This method is applied to the decoder side. This method corresponds to the encoding method shown in Figure 10. This method includes the following steps:

[0273] Step 1101: Determine a reconstructed third latent variable and a reconstructed first variable scale factor based on a bitstream.

[0274] In some embodiments, entropy decoding may be performed on the entropy coding result of the third latent variable in the bitstream, and the encoding result of the first variable scale factor in the bitstream may be decoded to obtain a quantized third latent variable and a quantized first variable scale factor. An inverse quantization process is performed on the quantized third latent variable and the quantized first variable scale factor to obtain a reconstructed third latent variable and a reconstructed first variable scale factor.

[0275] The decoding method of this step corresponds to the encoding method on the encoder side, and the inverse quantization process of this step corresponds to the quantization process on the encoder side, in other words, the decoding method is the inverse process of the encoding method, and the inverse quantization process is the inverse process of the quantization process.

[0276] For example, the bitstream may be parsed to obtain a first quantization index and a third quantization index, where the first quantization index indicates a quantization step of the first variable scale factor and the third quantization index indicates a quantization step of the third latent variable, and an inverse quantization process is performed on the quantized first variable scale factor based on the quantization step indicated by the first quantization index to obtain a reconstructed first variable scale factor, and an inverse quantization process is performed on the quantized third latent variable based on the quantization step indicated by the third quantization index to obtain a reconstructed third latent variable.

[0277] Step 1102: Determine a reconstructed second latent variable based on the bitstream and the reconstructed third latent variable.

[0278] In some embodiments, the reconstructed third latent variables are processed by using a context decoding neural network model to obtain reconstructed first entropy coding model parameters, and the reconstructed second latent variables are determined based on the bitstream and the reconstructed first entropy coding model parameters.

[0279] In some embodiments, an entropy decoding model corresponding to the reconstructed first entropy coding model parameters may be determined. Entropy decoding is performed on an entropy coding result of a second latent variable in the bitstream based on the entropy decoding model corresponding to the reconstructed first entropy coding model parameters to obtain a quantized second latent variable. An inverse quantization process is performed on the quantized second latent variable to obtain the reconstructed second latent variable.

[0280] The decoding method of this step corresponds to the encoding method on the encoder side, and the inverse quantization process of this step corresponds to the quantization process on the encoder side, in other words, the decoding method is the inverse process of the encoding method, and the inverse quantization process is the inverse process of the quantization process.

[0281] For example, the bitstream may be parsed to obtain a second quantization index, where the second quantization index indicates a quantization step of the second latent variable, and an inverse quantization process is performed on the quantized second latent variable based on the quantization step indicated by the second quantization index to obtain a reconstructed second latent variable.

[0282] Step 1103: Scale the reconstructed second latent variable based on the reconstructed first variable scale factor to obtain a reconstructed first latent variable, where the reconstructed first latent variable indicates a characteristic of the media data to be decoded.

[0283] For the implementation process of step 1103, please refer to the implementation process of step 902. The details will not be repeated here.

[0284] Step 1104: Process the reconstructed first latent variables by using the first decoding neural network model to obtain reconstructed media data.

[0285] For the implementation process of step 1104, please refer to the implementation process of step 903. The details will not be repeated here.

[0286] In this embodiment of the present application, the total number of coding bits of the entropy coding result of the second latent variable and the coding bits of the entropy coding result of the third latent variable meets a preset encoding rate condition. This ensures that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of media data can meet the preset encoding rate condition, i.e., the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of media data can basically keep consistency instead of dynamically changing, thereby meeting the encoder's requirement for a stable encoding rate. Furthermore, when side information (e.g., window type, Temporal Noise Shaping (TNS) parameter, Frequency-domain noise shaping (FDNS) parameter, and / or Bandwidth Extension (BWE) parameter) needs to be transmitted, it can be guaranteed that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of media data is basically consistent with the number of coding bits of the side information, thereby meeting the encoder's requirement for a stable encoding rate.

[0287] FIG. 12 is an exemplary block diagram of an encoding method according to an embodiment of the present application. In FIG. 12, the encoding method shown in FIG. 10 is mainly described as an example. In FIG. 12, an audio signal is used as an example. A windowing process may be performed on the audio signal to obtain an audio signal of a current frame. An MDCT process is performed on the audio signal of a current frame to obtain a frequency domain signal of the current frame. A first latent variable is output by performing a process by a first encoding neural network model based on the frequency domain signal of the current frame. The first latent variable is scaled based on a first latent variable scale factor to obtain a second latent variable. The second latent variable is processed by using a context encoding neural network model to obtain a third latent variable. A quantization process and an entropy coding are performed on a third latent variable to obtain an entropy coding result of the third latent variable, and the entropy coding result of the third latent variable is written into a bitstream. In addition, entropy decoding is performed on the entropy coding result of the third latent variable to obtain a quantized third latent variable. An inverse quantization process is performed on the quantized third latent variable to obtain a reconstructed third latent variable. The reconstructed third latent variable is processed by using a context decoding neural network model to obtain a first entropy coding model parameter, and an entropy coding model corresponding to the first entropy coding model parameter is selected from the entropy coding models having scalable parameters. The second latent variable is quantized, and entropy coding is performed on the quantized second latent variable based on the selected entropy coding model to obtain an entropy coding result of the second latent variable, and further, the entropy coding result of the second latent variable is written into the bitstream.The encoding result of the first variable scale factor is then written to the bitstream.

[0288] FIG. 13 is an exemplary block diagram of a decoding method according to an embodiment of the present application. In FIG. 13, the decoding method shown in FIG. 11 is mainly taken as an example to describe. In FIG. 13, an audio signal is used as an example. Entropy decoding is performed on the entropy coding result of the third latent variable in the bitstream by using an entropy decoding model to obtain a quantized third latent variable. An inverse quantization process is performed on the quantized third latent variable to obtain a reconstructed third latent variable. The reconstructed third variable is processed by using a context decoding neural network model to obtain a reconstructed first entropy coding model parameter. A corresponding entropy decoding model is selected from the entropy decoding models with scalable parameters based on the reconstructed first entropy coding model parameter. Entropy decoding is performed on an entropy coding result of the second latent variable in the bitstream based on the selected entropy decoding model to obtain a quantized second latent variable. An inverse quantization process is performed on the quantized second latent variable to obtain a reconstructed second latent variable. An encoding result of the first variable scale factor in the bitstream is decoded to obtain a reconstructed first variable scale factor. The reconstructed second latent variable is scaled based on the reconstructed first variable scale factor to obtain a reconstructed first latent variable. The reconstructed first latent variable is processed by using the first decoding neural network model to obtain a reconstructed frequency domain signal of the current frame. An IMDCT process and a dewindowing process are performed on the reconstructed frequency domain signal of the current frame to obtain a reconstructed audio signal.

[0289] 14 is a flowchart of a third encoding method according to an embodiment of the present application. The method includes a context model, and not only the latent variables generated by using the media data to be encoded are scaled based on a scale factor, but also the latent variables determined by using the context model are scaled based on a scale factor. The encoding method is applied to an encoder-side device and includes the following steps:

[0290] Step 1401: Process media data to be encoded by using a first encoding neural network model to obtain a first latent variable, where the first latent variable indicates a feature of the media data to be encoded.

[0291] For the implementation process of step 1401, please refer to the implementation process of step 601. The details will not be repeated here.

[0292] Step 1402: Determine a first variable scale factor and a second variable scale factor based on a first latent variable.

[0293] In some embodiments, a corresponding initial number of context coding bits and initial entropy coding model parameters may be determined based on the first latent variable by using the context model, a number of coding bits of an entropy coding result of the first latent variable is determined based on the initial entropy coding model parameters to obtain a basic initial number of coding bits, and further, a first variable scale factor and a second variable scale factor are determined based on the initial number of context coding bits, the basic initial number of coding bits, and the target number of coding bits.

[0294] For an implementation process of determining a corresponding initial number of context coding bits and an initial entropy coding model parameter based on the first latent variable by using a context model, as well as an implementation process of determining a number of coding bits of the entropy coding result of the first latent variable based on the initial entropy coding model parameter to obtain a basic initial number of coding bits, please refer to the relevant description of step 1002. Details will not be repeated here.

[0295] In some other embodiments, the first preset scale factor may be used as the first initial scale factor, and the second preset scale factor may be used as the second initial scale factor. The first latent variable is scaled based on the first initial scale factor, and a corresponding initial number of context coding bits and an initial entropy coding model parameter are determined based on the scaled first latent variable and the second initial scale factor by using a context model, and further, a number of coding bits of the entropy coding result of the first latent variable is determined based on the initial entropy coding model parameter to obtain a basic initial number of coding bits. The first variable scale factor and the second variable scale factor are determined based on the initial number of context coding bits, the basic initial number of coding bits, and the target number of coding bits.

[0296] For the implementation process of scaling the first latent variable based on the first initial scale factor, please refer to the related description of step 1002. The details will not be repeated here.

[0297] The context models include a context encoding neural network model and a context decoding neural network model. An implementation process for determining a corresponding initial number of context coding bits and initial entropy coding model parameters based on a scaled first latent variable and a second initial scale factor by using a context model includes: processing the scaled first latent variable by using a context encoding neural network model to obtain a sixth latent variable, where the sixth latent variable indicates a probability distribution of the scaled first latent variable; scaling the sixth latent variable based on the second initial scale factor to obtain a seventh latent variable; determining an entropy coding result of the seventh latent variable and using a number of coding bits of the entropy coding result of the seventh latent variable as the initial number of context coding bits; reconstructing the seventh latent variable based on the entropy coding result of the seventh latent variable and processing the seventh latent variable obtained through the reconstruction by using a context decoding neural network model to obtain initial entropy coding model parameters.

[0298] In one example, the scaled first latent variable is processed by using a context encoding neural network model to obtain a sixth latent variable. The sixth latent variable is scaled based on a second initial scale factor to obtain a seventh latent variable. A quantization process is performed on the seventh latent variable to obtain a quantized seventh latent variable. An entropy coding is performed on the quantized seventh latent variable, and a number of coding bits of the entropy coding result is counted to obtain an initial number of context coding bits. Then, an entropy decoding is performed on the entropy coding result of the seventh latent variable to obtain a quantized seventh latent variable, and an inverse quantization process is performed on the quantized seventh latent variable to obtain a reconstructed seventh latent variable. The reconstructed seventh latent variable is input to the context decoding neural network model to obtain initial entropy coding model parameters output by the context decoding neural network model.

[0299] An implementation process for scaling the sixth latent variable based on the second initial scaling factor is to multiply each element in the sixth latent variable by the corresponding element in the second initial scaling factor to obtain a seventh latent variable as follows:

[0300] It should be noted that the above implementation process is just an example, and in practical application, other methods may be used for scaling. For example, each element in the sixth latent variable may be divided by the corresponding element in the second initial scale factor to obtain the seventh latent variable. The scaling method is not limited in this embodiment of the present application.

[0301] For the specific implementation process of the remaining steps, please refer to the above description of the corresponding steps, and the details will not be repeated here.

[0302] It should be noted that in this embodiment of the present application, an initial value of the scale factor is set for the third latent variable, and the initial value of the scale factor is usually equal to 1. The second preset scale factor may be greater than or equal to the initial value of the scale factor, and may be smaller than the initial value of the scale factor. For example, the second preset scale factor is a constant, such as 1 or 2. When the first variable scale factor and the second variable scale factor are determined in the first implementation, the second initial scale factor is the initial value of the scale factor. When the first variable scale factor and the second variable scale factor are determined in the second implementation, the second initial scale factor is the second preset scale factor.

[0303] In addition, the second initial scale factor may be a scalar or a vector. For a specific description, please refer to the related description of the first initial scale factor.

[0304] There may be two implementation processes for determining the first variable scale factor and the second variable scale factor based on the initial number of context coding bits, the basic initial number of coding bits, and the target number of coding bits. The implementation processes are described below individually.

[0305] In a first implementation, the second variable scale factor is set to a second initial scale factor, and the base target number of coding bits is determined based on the target number of coding bits and at least one of the base initial number of coding bits and the initial number of context coding bits. The first variable scale factor and the base actual number of coding bits are determined based on the second initial scale factor, the base target number of coding bits and the base initial number of coding bits, and the base actual number of coding bits is the number of coding bits of the entropy coding result of the first latent variable scaled based on the first variable scale factor. The target number of context coding bits is determined based on the target number of coding bits and the base actual number of coding bits. The second variable scale factor is determined based on the target number of context coding bits and the initial number of context coding bits.

[0306] The implementation process of determining the basic target number of coding bits based on the target number of coding bits and at least one of the basic initial number of coding bits and the initial number of context coding bits includes: subtracting the initial number of context coding bits from the target number of coding bits to obtain the basic target number of coding bits; determining a ratio of the basic initial number of coding bits to the initial number of context coding bits and determining the basic target number of coding bits based on the ratio and the target number of coding bits; or determining the basic target number of coding bits based on the ratio of the target number of coding bits to the basic initial number of coding bits. Of course, it may be further determined by using another implementation process.

[0307] For example, the ratio of the base initial number of coding bits to the initial number of context coding bits is determined to be 5:3. In this case, the target number of coding bits may be multiplied by 5 / 8 to obtain the base target number of coding bits.

[0308] In another example, a ratio of the target number of coding bits to the basic initial number of coding bits is determined. If the ratio of the target number of coding bits to the basic initial number of coding bits is greater than a first ratio threshold, the sum of the basic initial number of coding bits and the preset first ratio scaling step is determined as the basic target number of coding bits. If the ratio of the target number of coding bits to the basic initial number of coding bits is less than a second ratio threshold, the difference between the basic initial number of coding bits and the preset second ratio scaling step is determined as the basic target number of coding bits. The first ratio threshold is greater than the second ratio threshold. If the ratio of the target number of coding bits to the basic initial number of coding bits is greater than or equal to the second ratio threshold and is further less than or equal to the first ratio threshold, the basic initial number of coding bits is determined as the basic target number of coding bits.

[0309] It should be noted that the first ratio scaling step may be greater than the second ratio scaling step, or may be less than the second ratio scaling step, or may of course be equal to the second ratio scaling step. The value relationship between the first ratio scaling step and the second ratio scaling step is not limited in this embodiment of the present application.

[0310] In addition, there are three implementations for determining the first variable scale factor according to the second initial scale factor, the base target number of coding bits, and the base initial number of coding bits. The following describes the implementations individually.

[0311] Scheme 11: When the basic initial number of coding bits is equal to the basic target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the basic initial number of coding bits is not equal to the basic target number of coding bits, the first variable scale factor is determined in a first cyclic manner based on the basic initial number of coding bits and the basic target number of coding bits.

[0312] The i-th cyclic process of the first cyclic scheme includes the steps of: determining a scale factor of the i-th cyclic process, where i is a positive integer; scaling the first latent variable based on the scale factor of the i-th cyclic process to obtain an i-th scaled first latent variable; determining an i-th number of context coding bits and an i-th entropy coding model parameter based on the i-th scaled first latent variable and a second initial scale factor by using a context model; and determining an i-th base number of coding bits based on the i-th scaled first latent variable and a second initial scale factor by using a context model. determining a number of coding bits of an entropy coding result of the ith scaled first latent variable based on a scaling model parameter; determining the ith number of coding bits based on the ith number of context coding bits and the ith base number of coding bits; and performing an (i+1)th cyclic process of the first cyclic scheme when the ith number of coding bits satisfies the continuous scaling condition, or terminating the execution of the first cyclic scheme and determining a first variable scale factor based on the scale factor of the ith cyclic process when the ith number of coding bits does not satisfy the continuous scaling condition.

[0313] The content in the method 11 for determining the first variable scale factor is similar to the content in the first implementation for determining the first variable scale factor in step 1002. The difference is that in this step, the i-th number of context coding bits and the i-th entropy coding model parameters are determined based on the i-th scaled first latent variable and the second initial scale factor by using a context model. For the implementation process for determining the i-th number of context coding bits and the i-th entropy coding model parameters based on the i-th scaled first latent variable and the second initial scale factor by using a context model, please refer to the above implementation process for determining the corresponding initial number of context coding bits and the initial entropy coding model parameters based on the scaled first latent variable and the second initial scale factor by using a context model. The details will not be repeated here.

[0314] Scheme 12: When the basic initial number of coding bits is equal to the basic target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the basic initial number of coding bits is not equal to the basic target number of coding bits, the first variable scale factor is determined in the above scheme 11. However, the difference from the above scheme 11 is that when the basic initial number of coding bits is smaller than the basic target number of coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the first step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling conditions include that the i-th number of coding bits is smaller than the basic target number of coding bits. When the basic initial number of coding bits is larger than the basic target number of coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the second step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling conditions include that the i-th number of coding bits is larger than the basic target number of coding bits.

[0315] For related content in Method 12, please refer to the content in the second implementation form of step 1002. Details will not be repeated here.

[0316] Manner 13: When the basic initial number of coding bits is less than or equal to the basic target number of coding bits, the first initial scale factor is determined as the first variable scale factor. When the basic initial number of coding bits is greater than the basic target number of coding bits, the first variable scale factor may be determined in the above manner 11. Alternatively, when the basic initial number of coding bits is greater than the basic target number of coding bits, the first variable scale factor may be determined in the above manner 12.

[0317] An implementation process for determining a target number of context coding bits based on a target number of coding bits and a base actual number of coding bits includes subtracting the base actual number of coding bits from the target number of coding bits to obtain the target number of context coding bits.

[0318] The implementation process of determining the second variable scale factor based on the target number of context coding bits and the initial number of context coding bits is similar to the implementation process of determining the first variable scale factor based on the base initial number of coding bits and the base target number of coding bits. The following describes the three schemes individually.

[0319] Scheme 21: When the initial number of context coding bits is equal to the target number of context coding bits, the second initial scale factor is determined as the second variable scale factor. When the initial number of context coding bits is not equal to the target number of context coding bits, the first latent variable is scaled based on the first variable scale factor to obtain the second latent variable. The second variable scale factor is determined in a first cyclic manner based on the initial number of context coding bits, the target number of context coding bits, and the second latent variable.

[0320] The ith cyclic process of the first cyclic scheme includes the following steps: determining a scale factor of the ith cyclic process, where i is a positive integer; determining an ith number of context coding bits and an ith entropy coding model parameter based on the second latent variable and the scale factor of the ith cyclic process by using a context model; determining a number of coding bits of an entropy coding result of the second latent variable based on the ith entropy coding model parameter to obtain an ith base number of coding bits; determining the ith number of coding bits based on the ith number of context coding bits and the ith base number of coding bits; performing an (i+1)th cyclic process of the first cyclic scheme when the ith number of coding bits satisfies a continuous scaling condition, or terminating the execution of the first cyclic scheme and determining a second variable scale factor based on the scale factor of the ith cyclic process when the ith number of coding bits does not satisfy the continuous scaling condition.

[0321] An implementation process for determining the i-th number of context coding bits and the i-th entropy coding model parameters based on the second latent variable and the scale factor of the i-th cyclic process by using a context model includes processing the second latent variable by using a context encoding neural network model to obtain a third latent variable, where the third latent variable indicates a probability distribution of the second latent variable; scaling the third latent variable based on the scale factor of the i-th cyclic process to obtain an i-th scaled third latent variable; and determining an entropy coding result of the i-th scaled third latent variable. determining a number of coding bits of an entropy coding result of the i-th scaled third latent variable as the i-th number of context coding bits; reconstructing the i-th scaled third latent variable based on the entropy coding result of the i-th scaled third latent variable; scaling the reconstructed i-th scaled third latent variable based on a scale factor of the i-th cyclic processing to obtain a reconstructed third latent variable; and processing the reconstructed third latent variable by using a context decoding neural network model to obtain the i-th entropy coding model parameters.

[0322] For other steps, please refer to the above description of the corresponding contents, and the details will not be repeated here.

[0323] Scheme 22: When the initial number of context coding bits is equal to the target number of context coding bits, the second initial scale factor is determined as the second variable scale factor. When the initial number of context coding bits is not equal to the target number of context coding bits, the second variable scale factor is determined in the above-mentioned scheme 21. However, the difference from the above-mentioned scheme 21 is that when the initial number of context coding bits is smaller than the target number of context coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the first step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling condition includes that the i-th number of coding bits is smaller than the target number of context coding bits. When the initial number of context coding bits is larger than the target number of context coding bits, the scale factor of the (i-1)th cyclic process of the first cyclic scheme is scaled based on the second step to obtain the scale factor of the i-th cyclic process. In this case, the successive scaling condition includes that the i-th number of coding bits is greater than the target number of context coding bits.

[0324] Scheme 23: When the initial number of context coding bits is less than or equal to the target number of context coding bits, the second initial scale factor is determined as the second variable scale factor. When the initial number of context coding bits is greater than the target number of context coding bits, the second variable scale factor may be determined as in scheme 21 above. Alternatively, the second variable scale factor may be determined as in scheme 22 above when the initial number of context coding bits is greater than the target number of context coding bits.

[0325] In a second implementation, the target number of coding bits is divided into a base target number of coding bits and a target number of context coding bits, and a first variable scale factor is determined based on the base target number of coding bits and the base initial number of coding bits, and a second variable scale factor is determined based on the target number of context coding bits and the initial number of context coding bits.

[0326] When the first variable scale factor is determined based on the basic target number of coding bits and the basic initial number of coding bits, the second variable scale factor may be set as the second initial scale factor, and the first variable scale factor is determined based on the second initial scale factor, the basic target number of coding bits, and the basic initial number of coding bits. For a specific implementation process, please refer to the above description.

[0327] In addition, when the second variable scale factor is determined based on the target number of context coding bits and the initial number of context coding bits, the first variable scale factor has already been determined. In this case, the second variable scale factor can be directly determined based on the first variable scale factor, the target number of context coding bits, and the initial number of context coding bits. For a specific implementation process, please refer to the above description.

[0328] Step 1403: Obtain an entropy coding result of a second latent variable and an entropy coding result of a fourth latent variable, where the second latent variable is obtained by scaling the first latent variable based on a first variable scale factor, the fourth latent variable is obtained by scaling the third latent variable based on a second variable scale factor, the third latent variable is determined based on the second latent variable by using a context model, and the third latent variable indicates a probability distribution of the second latent variable, and the total number of coding bits of the entropy coding result of the second latent variable and the coding bits of the entropy coding result of the fourth latent variable satisfies a preset encoding rate condition.

[0329] In the process of determining the first and second variable scale factors in step 1402, the entropy coding results corresponding to the scale factors of each cyclic process are determined in the aforementioned cyclic process. Therefore, the entropy coding results of the second latent variable and the entropy coding results of the fourth latent variable can be directly obtained from the entropy coding results determined in the aforementioned cyclic process.

[0330] Of course, the process may be performed again. In other words, the first latent variable is directly scaled based on the first variable scale factor to obtain the second latent variable. The entropy coding result of the fourth latent encoding and the second entropy coding model parameters are determined based on the second latent variable by using the context model. A quantization process is performed on the second latent variable to obtain a quantized second latent variable. The entropy coding result of the quantized second latent variable, i.e., the entropy coding result of the second latent variable, is determined based on the second entropy coding model parameters.

[0331] The context model includes a context encoding neural network model and a context decoding neural network model. An implementation process for determining an entropy coding result of a fourth latent encoding and second entropy coding model parameters based on a second latent variable by using the context model includes processing the second latent variable by using the context encoding neural network model to obtain a third latent variable, scaling the third latent variable based on a second variable scale factor to obtain a fourth latent variable, performing a quantization process on the fourth latent variable to obtain a quantized fourth latent variable, and decoding the quantized fourth latent variable to obtain an entropy coding result of the fourth latent variable. performing entropy coding on the fourth latent variable; performing entropy decoding on a result of the entropy coding of the fourth latent variable to obtain a quantized fourth latent variable; performing an inverse quantization process on the quantized fourth latent variable to obtain a reconstructed fourth latent variable; scaling the reconstructed fourth latent variable based on a second variable scale factor to obtain a reconstructed third latent variable; and processing the reconstructed third latent variable by using a context decoding neural network model to obtain second entropy coding model parameters.

[0332] An implementation process for determining an entropy coding result of a quantized second latent variable based on second entropy coding model parameters includes determining an entropy coding model corresponding to the second entropy coding model parameters from an entropy coding model having scalable encoding model parameters, and performing entropy coding on the quantized second latent variable based on the entropy coding model corresponding to the second entropy coding model parameters to obtain an initial encoding result of the second latent variable.

[0333] Step 1404: Write the entropy coding result of the second latent variable, the entropy coding result of the fourth latent variable, the encoding result of the first variable scale factor, and the encoding result of the second variable scale factor into a bitstream.

[0334] For the relevant content of the encoding result of the first variable scale factor, please refer to the description of step 604. The details will not be repeated here.

[0335] Based on the above description, the second variable scale factor may be a non-quantized value or a quantized value. When the second variable scale factor is a non-quantized value, the implementation process of determining the encoding result of the second variable scale factor includes performing a quantization and encoding process on the second variable scale factor to obtain the encoding result of the second variable scale factor. When the second variable scale factor is a quantized value, the implementation process of determining the encoding result of the second variable scale factor includes encoding the second variable scale factor to obtain the encoding result of the second variable scale factor. The second variable scale factor may be encoded by any encoding scheme. This is not limited in this embodiment of the present application.

[0336] In addition, in some encoders, the quantization process and the encoding process are implemented in one process, that is, the quantization result and the encoding result can be obtained in one process. Therefore, for the second variable scale factor, the encoding result of the second variable scale factor can also be obtained in the above process of determining the second variable scale factor. Therefore, the encoding result of the second variable scale factor can be obtained directly. In other words, the encoding result of the second variable scale factor can also be obtained directly in the process of determining the second variable scale factor.

[0337] Optionally, when the first variable scale factor and the second variable scale factor are non-quantized values, in this embodiment of the present application, a quantization index corresponding to a quantization step of the first variable scale factor may be further determined to obtain a first quantization index, and a quantization step of the second variable scale factor may be further determined to obtain a fifth quantization index. Optionally, in this embodiment of the present application, a quantization index corresponding to a quantization step of the second latent variable may be further determined to obtain a second quantization index, and a quantization index corresponding to a quantization step of the fourth latent variable may be further determined to obtain a fourth quantization index, and the first quantization index, the second quantization index, the fourth quantization index, and the fifth quantization index are encoded into the bitstream. The quantization indexes indicate corresponding quantization steps, i.e., the first quantization index indicates the quantization step of the first variable scale factor, the second quantization index indicates the quantization step of the second latent variable, the fourth quantization index indicates the quantization step of the fourth latent variable, and the fifth quantization index indicates the quantization step of the second variable scale factor.

[0338] In this embodiment of the present application, the first latent variable is scaled based on a first variable scale factor to obtain a second latent variable, and the second latent variable is processed by using a context encoding neural network model to obtain a third latent variable. The third latent variable is scaled based on a second variable scale factor to obtain a fourth latent variable. In addition, the total number of coding bits of the entropy coding result of the second latent variable and the coding bits of the entropy coding result of the fourth latent variable meets a preset encoding rate condition. This ensures that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can meet the preset encoding rate condition, i.e., the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can basically keep consistency instead of changing dynamically, thereby meeting the encoder's requirement for a stable encoding rate. Furthermore, when side information (e.g., window type, Temporal Noise Shaping (TNS) parameters, Frequency-domain noise shaping (FDNS) parameters, and / or Bandwidth extension (BWE) parameters) needs to be transmitted, it can be guaranteed that the number of coding bits of the entropy coding result of the latent variables corresponding to each frame of media data is essentially consistent with the number of coding bits of the side information, thereby meeting the encoder's requirement for a stable encoding rate.

[0339] Figure 15 is a flow chart of a third decoding method according to an embodiment of the present application. This method is applied to the decoder side. This method corresponds to the encoding method shown in Figure 14. This method includes the following steps:

[0340] Step 1501: Determine a reconstructed fourth latent variable, a reconstructed first variable scale factor, and a reconstructed second variable scale factor based on a bitstream.

[0341] In some embodiments, entropy decoding may be performed on the entropy coding result of the fourth latent variable in the bitstream, and the encoding result of the first variable scale factor and the second variable scale factor in the bitstream may be decoded to obtain the quantized fourth latent variable, the quantized first variable scale factor, and the quantized second variable scale factor. An inverse quantization process is performed on the quantized fourth latent variable, the quantized first variable scale factor, and the quantized second variable scale factor to obtain the reconstructed fourth latent variable, the reconstructed first variable scale factor, and the reconstructed second variable scale factor.

[0342] The decoding method of this step corresponds to the encoding method on the encoder side, and the inverse quantization process of this step corresponds to the quantization process on the encoder side, in other words, the decoding method is the inverse process of the encoding method, and the inverse quantization process is the inverse process of the quantization process.

[0343] For example, the bitstream may be parsed to obtain a first quantization index, a fourth quantization index, and a fifth quantization index, where the first quantization index indicates a quantization step of the first variable scale factor, the fourth quantization index indicates a quantization step of the fourth latent variable, and the fifth quantization index indicates a quantization step of the second variable scale factor. An inverse quantization process is performed on the quantized first variable scale factor based on the quantization step indicated by the first quantization index to obtain a reconstructed first variable scale factor. An inverse quantization process is performed on the quantized fourth latent variable based on the quantization step indicated by the fourth quantization index to obtain a reconstructed fourth latent variable. An inverse quantization process is performed on the quantized second variable scale factor based on the quantization step indicated by the fifth quantization index to obtain a reconstructed second variable scale factor.

[0344] Step 1502: Determine a reconstructed second latent variable based on the bitstream, the reconstructed fourth latent variable, and the reconstructed second variable scale factor.

[0345] In some embodiments, the reconstructed fourth latent variable is scaled based on the reconstructed second variable scale factor to obtain a reconstructed third latent variable. The reconstructed third latent variable is processed by using a context decoding neural network model to obtain reconstructed second entropy coding model parameters, and the reconstructed second latent variable is determined based on the bitstream and the reconstructed second entropy coding model parameters.

[0346] In some embodiments, an entropy decoding model corresponding to the reconstructed second entropy coding model parameters may be determined. Entropy decoding is performed on the entropy coding result of the second latent variable in the bitstream based on the entropy decoding model corresponding to the second entropy coding model parameters to obtain a quantized second latent variable. An inverse quantization process is performed on the quantized second latent variable to obtain a reconstructed second latent variable.

[0347] The decoding method of this step corresponds to the encoding method on the encoder side, and the inverse quantization process of this step corresponds to the quantization process on the encoder side, in other words, the decoding method is the inverse process of the encoding method, and the inverse quantization process is the inverse process of the quantization process.

[0348] For example, the bitstream may be parsed to obtain a second quantization index, where the second quantization index indicates a quantization step of the second latent variable, and an inverse quantization process is performed on the quantized second latent variable based on the quantization step indicated by the second quantization index to obtain a reconstructed second latent variable.

[0349] Step 1503: Scale the reconstructed second latent variable based on the reconstructed first variable scale factor to obtain a reconstructed first latent variable.

[0350] For the implementation process of step 1503, please refer to the implementation process of step 902. The details will not be repeated here.

[0351] Step 1504: Process the reconstructed first latent variables by using the first decoding neural network model to obtain reconstructed media data.

[0352] For the implementation process of step 1504, please refer to the implementation process of step 903. The details will not be repeated here.

[0353] In this embodiment of the present application, the total number of coding bits of the entropy coding result of the second latent variable and the coding bits of the entropy coding result of the fourth latent variable meets the preset encoding rate condition. This ensures that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of media data can meet the preset encoding rate condition, i.e., the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of media data can basically keep consistency instead of dynamically changing, thereby meeting the encoder's requirement for a stable encoding rate. Furthermore, when side information (e.g., window type, Temporal Noise Shaping (TNS) parameter, Frequency-domain noise shaping (FDNS) parameter, and / or Bandwidth Extension (BWE) parameter) needs to be transmitted, it can be guaranteed that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of media data is basically consistent with the number of coding bits of the side information, thereby meeting the encoder's requirement for a stable encoding rate.

[0354] 16A and 16B are exemplary block diagrams of an encoding method according to an embodiment of the present application. FIG. 16A and 16B mainly describe the encoding method shown in FIG. 14 as an example. In FIG. 16A and 16B, an audio signal is used as an example. A windowing process may be performed on the audio signal to obtain an audio signal of a current frame. An MDCT process is performed on the audio signal of a current frame to obtain a frequency domain signal of the current frame. A first latent variable is output by performing a process by a first encoding neural network model based on the frequency domain signal of the current frame. The first latent variable is scaled based on a first latent variable scale factor to obtain a second latent variable. The second latent variable is processed by using a context encoding neural network model to obtain a third latent variable. The third latent variable is scaled based on a second variable scale factor to obtain a fourth latent variable. A quantization process and entropy coding are performed on the fourth latent variable to obtain an entropy coding result of the fourth latent variable, and the entropy coding result of the fourth latent variable is written to the bitstream. In addition, entropy decoding is performed on the entropy coding result of the fourth latent variable to obtain a quantized fourth latent variable. An inverse quantization process is performed on the quantized fourth latent variable to obtain a reconstructed fourth latent variable. The reconstructed fourth latent variable is scaled based on a second variable scale factor to obtain a reconstructed third latent variable. The reconstructed third latent variable is processed using a context decoding neural network model to obtain second entropy coding model parameters. An entropy coding model corresponding to the second entropy coding model parameters is selected from the entropy coding models having scalable parameters.The second latent variable is quantized, and entropy coding is performed on the quantized second latent variable based on the selected entropy coding model to obtain an entropy coding result of the second latent variable, and the entropy coding result of the second latent variable is written into the bitstream. And the encoding result of the first variable scale factor is written into the bitstream.

[0355] FIG. 17 is an exemplary block diagram of a decoding method according to an embodiment of the present application. In FIG. 17, the decoding method shown in FIG. 15 is mainly taken as an example to describe. In FIG. 17, an audio signal is used as an example. Entropy decoding is performed on the entropy coding result of the fourth latent variable in the bitstream by using an entropy decoding model to obtain a quantized fourth latent variable. An inverse quantization process is performed on the quantized fourth latent variable to obtain a reconstructed fourth latent variable. The encoding result of the second variable scale factor in the bitstream is decoded to obtain a quantized second variable scale factor. An inverse quantization process is performed on the quantized second variable scale factor to obtain a reconstructed second variable scale factor. The reconstructed fourth latent variable is scaled based on the reconstructed second variable scale factor to obtain a reconstructed third latent variable. The reconstructed third variable is processed by using a context decoding neural network model to obtain a reconstructed second entropy coding model parameter. A corresponding entropy decoding model is selected from the entropy decoding models having scalable parameters based on the reconstructed second entropy coding model parameter. Entropy decoding is performed on the entropy coding result of the second latent variable in the bitstream based on the selected entropy decoding model to obtain a quantized second latent variable. An inverse quantization process is performed on the quantized second latent variable to obtain a reconstructed second latent variable. The encoding result of the first variable scale factor in the bitstream is decoded to obtain a reconstructed first variable scale factor. The reconstructed second latent variable is scaled based on the reconstructed first variable scale factor to obtain a reconstructed first latent variable.The reconstructed first latent variable is processed by using a first decoding neural network model to obtain a reconstructed frequency domain signal of the current frame, and an IMDCT processing and a dewindowing processing are performed on the reconstructed frequency domain signal of the current frame to obtain a reconstructed audio signal.

[0356] 18 is a schematic diagram of the structure of an encoding device according to an embodiment of the present application. The encoding device may be implemented as a part or whole of an encoder-side device by using software, hardware, or a combination thereof. The encoder-side device may be the source device shown in FIG. 1. As shown in FIG. 18, the device includes a data processing module 1801, a scale factor determination module 1802, a first encoding result acquisition module 1803, and a first encoding result writing module 1804.

[0357] The data processing module 1801 is configured to process the media data to be encoded by using a first encoding neural network model to obtain a first latent variable, and the first latent variable indicates the characteristics of the media data to be encoded. For detailed implementation process, please refer to the corresponding content of the preceding embodiment. The details will not be repeated here.

[0358] The scale factor determination module 1802 is configured to determine a first variable scale factor based on a first latent variable, the first variable scale factor is used to enable the number of coding bits of the entropy coding result of the second latent variable to meet a preset encoding rate condition, and the second latent variable is obtained by scaling the first latent variable based on the first variable scale factor. For detailed implementation process, please refer to the corresponding content of the preceding embodiment. The details will not be repeated here.

[0359] The first encoding result obtaining module 1803 is configured to obtain the entropy coding result of the second latent variable.For detailed implementation process, please refer to the corresponding content of the above embodiment.The details will not be described repeatedly here.

[0360] The first encoding result writing module 1804 is configured to write the entropy coding result of the second latent variable and the encoding result of the first variable scale factor into a bitstream.For detailed implementation process, please refer to the corresponding content of the above embodiment.The details will not be repeated here.

[0361] Optionally, when the media data to be encoded is encoded at a constant bit rate, satisfying the preset encoding rate condition includes the number of coding bits being less than or equal to the target number of coding bits, or satisfying the preset encoding rate condition includes the number of coding bits being less than or equal to the target number of coding bits and the difference between the number of coding bits and the target number of coding bits being less than a bit number threshold.

[0362] Optionally, when the media data to be encoded is encoded at a variable bit rate, satisfying the preset encoding rate condition includes: an absolute value of a difference between the number of coding bits and a target number of coding bits being less than a bit number threshold.

[0363] Optionally, the scale factor determination module 1802 comprises: a bit number determination submodule configured to determine an initial number of coding bits based on a first latent variable; and a first factor determination sub-module configured to determine a first variable scale factor based on the initial number of coding bits and the target number of coding bits.

[0364] Optionally, the initial number of coding bits is a number of coding bits of an entropy coding result of the first latent variable; or The initial number of coding bits is the number of coding bits of the entropy coding result of the first latent variable scaled based on the first initial scale factor.

[0365] Optionally, when the initial number of coding bits is not equal to the target number of coding bits, the first factor determination submodule: The method is particularly configured to determine a first variable scale factor in a first cyclic manner based on an initial number of coding bits and a target number of coding bits, and further comprising: The i-th cyclic process of the first cyclic method includes the following steps: determining a scale factor for an i-th cyclic process, where i is a positive integer; Scaling the first latent variable based on a scale factor of the i-th cyclic process to obtain an i-th scaled first latent variable; determining a number of coding bits of an entropy coding result of the i-th scaled first latent variable to obtain an i-th number of coding bits; performing an (i+1)th cyclic process of the first cyclic scheme if the i-th number of coding bits satisfies a continuous scaling condition; or If the i-th number of coding bits does not satisfy the continuous scaling condition, terminating the execution of the first cyclic scheme and determining a first variable scale factor based on the scale factor of the i-th cyclic process.

[0366] Optionally, the apparatus further comprises: a second encoding result acquisition module configured to acquire an entropy coding result of a third latent variable, where the third latent variable is determined based on the second latent variable by using a context model; and and a second encoding result writing module configured to write an entropy coding result of the third latent variable into the bitstream.

[0367] The total number of coding bits of the entropy coding result of the second latent variable and the coding bits of the entropy coding result of the third latent variable satisfies a preset encoding rate condition.

[0368] Optionally, the scale factor determination module 1802 comprises: a bit number determination submodule configured to determine an initial number of coding bits based on a first latent variable; and a first factor determination sub-module configured to determine a first variable scale factor based on the initial number of coding bits and the target number of coding bits.

[0369] Optionally, the bit number determination submodule comprises: determining a corresponding initial number of context coding bits and initial entropy coding model parameters based on the first latent variable by using a context model; Determine a number of coding bits of the entropy coding result of the first latent variable based on the initial entropy coding model parameters to obtain a basic initial number of coding bits; Determine the initial number of coding bits based on the initial number of context coding bits and the basic initial number of coding bits. The device is specifically configured as follows.

[0370] Optionally, when the initial number of coding bits is not equal to the target number of coding bits, the first factor determination submodule: The method is particularly configured to determine a first variable scale factor in a first cyclic manner based on an initial number of coding bits and a target number of coding bits, and further comprising: The i-th cyclic process of the first cyclic method includes the following steps: determining a scale factor for an i-th cyclic process, where i is a positive integer; Scaling the first latent variable based on a scale factor of the i-th cyclic process to obtain an i-th scaled first latent variable; determining an i-th number of context coding bits and an i-th entropy coding model parameter based on the i-th scaled first latent variable by using a context model; determining a number of coding bits of an entropy coding result of the i-th scaled first latent variable based on the i-th entropy coding model parameter to obtain an i-th base number of coding bits; determining an i-th number of coding bits based on the i-th number of context coding bits and the i-th base number of coding bits; performing an (i+1)th cyclic process of the first cyclic scheme if the i-th number of coding bits satisfies a continuous scaling condition; or If the i-th number of coding bits does not satisfy the continuous scaling condition, terminating the execution of the first cyclic scheme and determining a first variable scale factor based on the scale factor of the i-th cyclic process.

[0371] Optionally, the first factor determination submodule comprises: It is particularly configured to determine a scale factor for the i-th cyclic process based on the scale factor of the (i-1)-th cyclic process of the first cyclic scheme, the (i-1)-th number of coding bits, and the target number of coding bits.

[0372] When i=1, the scale factor of the (i-1)th cyclic process is the first initial scale factor, and the (i-1)th number of coding bits is the initial number of coding bits.

[0373] The continuous scaling condition includes: both the (i-1)th number of coding bits and the i-th number of coding bits are less than the target number of coding bits; or the continuous scaling condition includes both the (i-1)th number of coding bits and the i-th number of coding bits are greater than the target number of coding bits.

[0374] Optionally, when the initial number of coding bits is smaller than the target number of coding bits, the first factor determination submodule: It is particularly configured to scale a scale factor of the (i-1)th cyclic process of the first cyclic scheme based on the first step to obtain a scale factor of the i-th cyclic process.

[0375] When i=1, the scale factor of the (i-1)th cyclic process is the first initial scale factor.

[0376] The successive scaling conditions include that the i-th number of coding bits is less than the target number of coding bits.

[0377] Optionally, when the initial number of coding bits is greater than the target number of coding bits, the first factor determination submodule: It is particularly configured to scale the scale factor of the (i-1)th cyclic process of the first cyclic scheme based on the second step to obtain the scale factor of the i-th cyclic process.

[0378] When i=1, the scale factor of the (i-1)th cyclic process is the first initial scale factor.

[0379] The successive scaling conditions include that the i-th number of coding bits is greater than the target number of coding bits.

[0380] Optionally, the first factor determination submodule comprises: and determining the first initial scale factor as the first variable scale factor when the initial number of coding bits is smaller than the target number of coding bits.

[0381] Optionally, the first factor determination submodule comprises: When the i-th number of coding bits is equal to the target number of coding bits, determine the scale factor of the i-th cyclic process as a first variable scale factor; or It is particularly configured to determine the first variable scale factor based on the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process of the first cyclic scheme when the i-th number of coding bits is not equal to the target number of coding bits.

[0382] Optionally, the first factor determination submodule comprises: determining an average of the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process; Determine the first variable scale factor based on the average value The device is specifically configured as follows.

[0383] Optionally, the first factor determination submodule comprises: The scaling factor determining unit is particularly configured to determine a first variable scaling factor in a second cyclic manner based on the scaling factor of the i-th cyclic process and the scaling factor of the (i-1)-th cyclic process.

[0384] The j-th cyclic process of the second cyclic method includes the following steps: determining a third scale factor of the jth cyclic process based on the first scale factor of the jth cyclic process and the second scale factor of the jth cyclic process, where when j is equal to 1, the first scale factor of the jth cyclic process is one of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, and the second scale factor of the jth cyclic process is the other of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, the first scale factor of the jth cyclic process corresponds to the jth first number of coding bits and the second scale factor of the jth cyclic process corresponds to the jth second number of coding bits, the jth first number of coding bits is smaller than the jth second number of coding bits, and j is a positive integer; obtaining a j-th third number of coding bits, where the j-th third number of coding bits is a number of coding bits of an entropy coding result of the first latent variable scaled based on a third scale factor of the j-th cyclic process; if the j-th third number of coding bits does not satisfy the consecutive cycle condition, terminating the execution of the second cyclic scheme and determining the third scale factor of the j-th cyclic process as the first variable scale factor; If the jth third number of coding bits satisfies a consecutive cycle condition, is greater than the target number of coding bits and is less than the jth second number of coding bits, performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as the second scale factor of the (j+1)th cyclic process and using the first scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process; or If the jth third number of coding bits satisfies a consecutive cycle condition, is smaller than the target number of coding bits, and is larger than the jth first number of coding bits, performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process and using the second scale factor of the jth cyclic process as the second scale factor of the (j+1)th cyclic process.

[0385] Optionally, the first factor determination submodule comprises: The scaling factor determining unit is particularly configured to determine a first variable scaling factor in a second cyclic manner based on the scaling factor of the i-th cyclic process and the scaling factor of the (i-1)-th cyclic process.

[0386] The j-th cyclic process of the second cyclic method includes the following steps: determining a third scale factor of the jth cyclic process based on the first scale factor of the jth cyclic process and the second scale factor of the jth cyclic process, where when j is equal to 1, the first scale factor of the jth cyclic process is one of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, and the second scale factor of the jth cyclic process is the other of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, the first scale factor of the jth cyclic process corresponds to the jth first number of coding bits and the second scale factor of the jth cyclic process corresponds to the jth second number of coding bits, the jth first number of coding bits is smaller than the jth second number of coding bits, and j is a positive integer; obtaining a j-th third number of coding bits, where the j-th third number of coding bits is a number of coding bits of an entropy coding result of the first latent variable scaled based on a third scale factor of the j-th cyclic process; if the j-th third number of coding bits does not satisfy the consecutive cycle condition, terminating the execution of the second cyclic scheme and determining the third scale factor of the j-th cyclic process as the first variable scale factor; terminating execution of the second cyclic scheme when j reaches a maximum number of cycles and the j-th third number of coding bits satisfies a consecutive cycle condition, and determining a first variable scale factor based on the first scale factor of the j-th cyclic process; If j has not reached the maximum number of cycles and the jth third number of coding bits satisfies a consecutive cycle condition, is greater than the target number of coding bits and is less than the jth second number of coding bits, performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as the second scale factor of the (j+1)th cyclic process and using the first scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process; or If j has not reached the maximum number of cycles and the jth third number of coding bits satisfies a consecutive cycle condition, is smaller than the target number of coding bits and is larger than the jth first number of coding bits, performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as the first scale factor of the (j+1)th cyclic process and using the second scale factor of the jth cyclic process as the second scale factor of the (j+1)th cyclic process.

[0387] Optionally, the first factor determination submodule comprises: It is particularly configured to determine the first scale factor of the j-th cyclic process as the first variable scale factor when the media data to be encoded is encoded at a constant bit rate.

[0388] Optionally, the first factor determination submodule comprises: determining a first difference between a target number of coding bits and a j-th first number of coding bits, determining a second difference between the j-th second number of coding bits and the target number of coding bits when the media data to be encoded is encoded at a variable bit rate; If the first difference is smaller than the second difference, determine the first scale factor of the j-th cyclic process as a first variable scale factor. If the second difference is smaller than the first difference, determine the second scale factor of the j-th cyclic process as the first variable scale factor; or If the first difference is equal to the second difference, determine the first scale factor of the j-th cyclic process as the first variable scale factor, or determine the second scale factor of the j-th cyclic process as the first variable scale factor. The device is specifically configured as follows.

[0389] Optionally, when the media data to be encoded is encoded at a constant bit rate, the successive cycle condition includes: the jth third number of coding bits is greater than a target number of coding bits; or the successive cycle condition includes: the jth third number of coding bits is smaller than the target number of coding bits, and a difference between the target number of coding bits and the jth third number of coding bits is greater than a bit number threshold.

[0390] Optionally, when the media data to be encoded is encoded at a variable bit rate, the successive cycle condition includes that the absolute value of the difference between the target number of coding bits and the jth third number of coding bits is greater than a bit number threshold.

[0391] Optionally, the first factor determination submodule comprises: It is particularly configured to determine the first initial scale factor as the first variable scale factor when the initial number of coding bits is equal to the target number of coding bits.

[0392] Optionally, the apparatus further comprises: a scale factor determination module 1802 further configured to determine a second variable scale factor based on the first latent variable; a third encoding result acquisition module configured to acquire an entropy coding result of a fourth latent variable, where the fourth latent variable is acquired by scaling the third latent variable based on a second variable scale factor, and the third latent variable is determined based on the second latent variable by using a context model; and and a third encoding result writing module configured to write the entropy coding result of the fourth latent variable and the encoding result of the second variable scale factor into the bitstream.

[0393] The total number of coding bits of the entropy coding result of the second latent variable and the coding bits of the entropy coding result of the fourth latent variable satisfies a preset encoding rate condition.

[0394] Optionally, the scale factor determination module 1802 comprises: a first determining sub-module configured to determine a corresponding initial number of context coding bits and initial entropy coding model parameters based on the first latent variable by using a context model; a second determining sub-module configured to determine a number of coding bits of an entropy coding result of the first latent variable based on the initial entropy coding model parameters to obtain a basic initial number of coding bits; and a second factor determination submodule configured to determine a first variable scale factor and a second variable scale factor based on the initial number of context coding bits, the base initial number of coding bits, and the target number of coding bits.

[0395] Optionally, the second factor determination submodule comprises: determining a base target number of coding bits based on the target number of coding bits and at least one of a base initial number of coding bits and an initial number of context coding bits; Determine a first variable scale factor and a base actual number of coding bits according to the second initial scale factor, the base target number of coding bits, and the base initial number of coding bits, where the base actual number of coding bits is a number of coding bits of an entropy coding result of the first latent variable scaled according to the first variable scale factor; determining a target number of context coding bits based on the target number of coding bits and the base actual number of coding bits; Determine a second variable scale factor based on the target number of context coding bits and the initial number of context coding bits. The device is specifically configured as follows.

[0396] Optionally, the second factor determination submodule comprises: Dividing the target number of coding bits into a base target number of coding bits and a target number of context coding bits; determining a first variable scale factor based on a base target number of coding bits and a base initial number of coding bits; determining a second variable scale factor based on the target number of context coding bits and the initial number of context coding bits; The device is specifically configured as follows.

[0397] Optionally, the media data is an audio signal, a video signal or an image.

[0398] The number of coding bits of the entropy coding result of the second latent variable satisfies a preset encoding rate condition. This ensures that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can satisfy the preset encoding rate condition, i.e., the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can be basically kept consistent instead of dynamically changing, thereby meeting the encoder's requirement for a stable encoding rate. Furthermore, when side information (e.g., a window type, a Temporal Noise Shaping (TNS) parameter, a Frequency-domain noise shaping (FDNS) parameter, and / or a bandwidth extension (BWE) parameter) needs to be transmitted, it can be ensured that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data is basically consistent with the number of coding bits of the side information, thereby meeting the encoder's requirement for a stable encoding rate.

[0399] It should be noted that when the encoding device provided in the above embodiment performs encoding, the division of the above function modules is only used as an example for explanation. In practical application, the above functions may be assigned to different function modules of implementation based on requirements, that is, the internal structure of the device is divided into different function modules to implement all or part of the above functions. In addition, the encoding device provided in the above embodiment relates to the same concept as the encoding method embodiment. For the specific implementation process of the encoding device, please refer to the method embodiment. The details will not be repeated here.

[0400] 19 is a schematic diagram of the structure of a decoding device according to an embodiment of the present application. The decoding device may be implemented as a part or whole of a decoder-side device by using software, hardware, or a combination thereof. The decoder-side device may be the destination device shown in FIG. 1. As shown in FIG. 19, the device includes: a first determination module 1901, a variable scale module 1902, and a variable processing module 1903.

[0401] The first determining module 1901 is configured to determine a reconstructed second latent variable and a reconstructed first variable scale factor according to the bit stream. For detailed implementation process, please refer to the corresponding content of the above embodiment. The details will not be repeated here.

[0402] The variable scale module 1902 is configured to scale the reconstructed second latent variable according to the reconstructed first variable scale factor to obtain a reconstructed first latent variable, and the reconstructed first latent variable indicates the characteristics of the media data to be decoded. For detailed implementation process, please refer to the corresponding content of the above embodiment. The details will not be repeated here.

[0403] The variable processing module 1903 is configured to process the reconstructed first latent variables by using a first decoding neural network model to obtain reconstructed media data. For detailed implementation process, please refer to the corresponding content of the above-mentioned embodiment. The details will not be repeated here.

[0404] Optionally, the first determination module 1901: a first determination submodule configured to determine a reconstructed third latent variable based on the bitstream; and a second determining submodule configured to determine a reconstructed second latent variable based on the bitstream and the reconstructed third latent variable.

[0405] Optionally, the second determination submodule: Processing the reconstructed third latent variables by using a context decoding neural network model to obtain reconstructed first entropy coding model parameters; determining a reconstructed second latent variable based on the bitstream and the reconstructed first entropy coding model parameters; The device is specifically configured as follows.

[0406] Optionally, the first determination module 1901: a third determining sub-module configured to determine a reconstructed fourth latent variable and a reconstructed second variable scale factor based on the bitstream; and a fourth determination submodule configured to determine a reconstructed second latent variable based on the bitstream, the reconstructed fourth latent variable, and the reconstructed second variable scale factor.

[0407] Optionally, the fourth determination submodule: Scale the reconstructed fourth latent variable based on the reconstructed second variable scale factor to obtain a reconstructed third latent variable; Processing the reconstructed third latent variables by using a context decoding neural network model to obtain reconstructed second entropy coding model parameters; determining a reconstructed second latent variable based on the bitstream and the reconstructed second entropy coding model parameters; The device is specifically configured as follows.

[0408] Optionally, the media data is an audio signal, a video signal or an image.

[0409] In this embodiment of the present application, the number of coding bits of the entropy coding result of the second latent variable meets the preset encoding rate condition. This ensures that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can meet the preset encoding rate condition, that is, the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data can basically keep consistency instead of dynamically changing, thereby meeting the encoder's requirement for a stable encoding rate. Furthermore, when side information (e.g., window type, Temporal Noise Shaping (TNS) parameter, Frequency-domain noise shaping (FDNS) parameter, and / or Bandwidth Extension (BWE) parameter) needs to be transmitted, it can be guaranteed that the number of coding bits of the entropy coding result of the latent variable corresponding to each frame of the media data is basically consistent with the number of coding bits of the side information, thereby meeting the encoder's requirement for a stable encoding rate.

[0410] It should be noted that when the decoding device provided in the above embodiment performs decoding, the division of the above function modules is only used as an example for explanation. In practical application, the above functions may be assigned to different function modules of implementation based on requirements, that is, the internal structure of the device is divided into different function modules to implement all or part of the above functions. In addition, the decoding device provided in the above embodiment relates to the same concept as the embodiment of the decoding method. For the specific implementation process of the encoding device, please refer to the embodiment of the method. The details will not be repeated here.

[0411] 20 is a schematic block diagram of an encoding and decoding device 2000 according to an embodiment of the present application. The encoding and decoding device 2000 may include a processor 2001, a memory 2002, and a bus system 2003. The processor 2001 and the memory 2002 are connected through the bus system 2003. The memory 2002 is configured to store instructions. The processor 2001 is configured to execute instructions stored in the memory 2002 to implement various encoding or decoding methods described in the embodiments of the present application. To avoid repetition, details will not be described again here.

[0412] In this embodiment of the application, the processor 2001 may be a central processing unit (CPU), or the processor 2001 may be another general purpose processor, a DSP, an ASIC, an FPGA, or another programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0413] The memory 2002 may include a ROM device, or a RAM device. Any other suitable type of storage device may be used as the memory 2002. The memory 2002 may include code and data 20021 that are accessed by the processor 2001 through the bus 2003. The memory 2002 may further include an operating system 20023 and an application program 20022. The application program 20022 includes at least one program that enables the processor 2001 to perform the encoding or decoding method described in the embodiments of the present application. For example, the application program 20022 may include applications 1 to N, and may further include an encoding or decoding application (shortly referred to as encoding and decoding application) that performs the encoding or decoding method described in the embodiments of the present application.

[0414] In addition to a data bus, the bus system 2003 may further include a power bus, a control bus, a status signal bus, etc. However, for purposes of clarity, various types of buses are illustrated in the figures as the bus system 2003.

[0415] Optionally, the encoding and decoding apparatus 2000 may further include one or more output devices, such as a display 2004. In one example, the display 2004 may be a touch display that combines a display and a touch unit that operatively senses touch input. The display 2004 may be connected to the processor 2001 through a bus 2003.

[0416] It should be noted that the encoding and decoding device 2000 may implement the encoding method in the embodiment of the present application, and may also implement the decoding method in the embodiment of the present application.

[0417] Those skilled in the art may appreciate that the functions described with reference to the various exemplary logical blocks, modules, and algorithmic steps disclosed and described herein may be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, the functions described in connection with the exemplary logical blocks, modules, and steps may be stored on or transmitted via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium, such as a data storage medium, or any communication medium that facilitates the transmission of a computer program from any place to another (e.g., according to a communication protocol). In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers, or one or more processors, to incorporate instructions, codes, and / or data structures for implementing the techniques described in this application. A computer program product may include a computer-readable medium.

[0418] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that may store the necessary program code in the form of instructions or data structures and be accessed by a computer. In addition, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source through coaxial cable, fiber optic, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio waves, or microwave, the coaxial cable, fiber optic, twisted pair, DSL, or wireless technologies such as infrared, radio waves, or microwave are included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, and in fact refer to non-transitory tangible storage media. As used herein, discs and optical discs include compact discs (CDs), laser discs, optical discs, DVDs, and Blu-ray discs, where discs generally reproduce data magnetically and optical discs reproduce data optically by using a laser. Combinations of the above should also be included within the scope of computer-readable media.

[0419] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to the aforementioned structures, or any other structures that may be applied to the implementation of the techniques described herein. In addition, in some aspects, the functions described in connection with the exemplary logical blocks, modules, and steps described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a composite codec. In addition, these techniques may be fully implemented in one or more circuits or logic elements. In one example, the various exemplary logical blocks, units, and modules in the encoder 100 and decoder 200 may be understood as corresponding circuit devices or logic elements.

[0420] The techniques in the embodiments of the present application may be implemented in a variety of apparatuses or devices, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). In the present embodiments of the present application, various components, modules, or units are described to highlight functional aspects of an apparatus configured to implement the disclosed techniques, but are not necessarily implemented by different hardware units. In practice, as previously described, the various units may be combined into a codec hardware unit in combination with appropriate software and / or firmware, or may be effected by interoperable hardware units (including one or more processors as previously described).

[0421] In other words, all or part of the above-mentioned embodiments may be implemented by using software, hardware, firmware, or any combination thereof. When software is used to implement the embodiments, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the procedures or functions according to the embodiments of the present application are generated completely or partially. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or another programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired (e.g., coaxial cable, optical fiber, or digital subscriber line (DSL)) or wireless (e.g., infrared, radio wave, or microwave) manner. The computer-readable storage medium may be any available medium accessible by a computer, or may be a data storage device, such as a server or a data center, that integrates one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), a semiconductor medium (e.g., a solid state disk (SSD)), etc. It should be noted that the computer-readable storage medium referred to in the embodiments of the present application may be a non-volatile storage medium, or in other words, a non-transitory storage medium.

[0422] It should be understood that "multiple" in this specification means two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may represent A or B. In this specification, "and / or" describes only the association relationship to describe the associated object, and represents that three relationships may exist. For example, A and / or B represents the following three cases, namely, when only A exists, when both A and B exist, and when only B exists. In addition, in order to clearly describe the technical solutions in the embodiments of the present application, terms such as "first" and "second" are used in the embodiments of the present application to distinguish the same or similar items that basically provide the same function or purpose. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity or execution order, and terms such as "first" and "second" do not indicate clear differences.

[0423] The above description is merely an embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, or improvement made without departing from the spirit and principle of the present application should be included within the protection scope of the present application. [Explanation of symbols]

[0424] 10 Source Device 20 Destination Device 30 Links 40 Storage device 100 Encoder 101 First Terminal 120 Data Sources 140 Output Interface 200 Decoder 201 Second Terminal 220 Display device 240 Input Interface 1801 Data Processing Module 1802 Scale factor determination module 1803 First encoding result acquisition module 1804 first encoding result writing module 1901 First Decision Module 1902 Variable Scale Module 1903 Variable Processing Module 2000 Encoding and Decoding Device 2001 Processor 2002 Memory 2003 Bus System 2004 Display 20021 Code and Data 20022 Application Program 20023 Operating System

Claims

1. 1. A method of encoding, comprising: processing the media data to be encoded to obtain a first latent variable, the first latent variable indicative of a characteristic of the media data to be encoded; determining a first variable scale factor based on the first latent variable, the first variable scale factor being used to enable a number of coding bits of a coding result of a second latent variable to meet a preset encoding rate condition for ensuring consistency based on a target number of coding bits, the second latent variable being obtained by scaling the first latent variable by multiplying or dividing it by the first variable scale factor; encoding the second latent variable into a bitstream; encoding the first variable scale factor into the bitstream; A method comprising:

2. 2. The method of claim 1, wherein when the media data to be encoded is encoded at a constant bit rate, the satisfying of a preset encoding rate condition includes: the number of coding bits being smaller than or equal to a target number of coding bits; or the satisfying of a preset encoding rate condition includes: the number of coding bits being smaller than or equal to a target number of coding bits and a difference between the number of coding bits and the target number of coding bits being smaller than a bit number threshold.

3. 2. The method of claim 1, wherein when the media data to be encoded is encoded at a variable bit rate, the satisfying of a preset encoding rate condition comprises: an absolute value of a difference between the number of coding bits and a target number of coding bits being less than a bit number threshold.

4. said step of determining a first variable scale factor based on said first latent variable further comprising: determining an initial number of coding bits based on the first latent variable; determining the first variable scale factor based on the initial number of coding bits and a target number of coding bits; 2. The method of claim 1, comprising:

5. The initial number of coding bits is the number of coding bits of the entropy coding result of the first latent variable; or The method of claim 4 , wherein the initial number of coding bits is a number of coding bits of an entropy coding result of a first latent variable scaled based on a first initial scale factor.

6. When the initial number of coding bits is not equal to the target number of coding bits, the step of determining the first variable scale factor based on the initial number of coding bits and the target number of coding bits comprises: determining the first variable scale factor in a first cyclic manner based on the initial number of coding bits and the target number of coding bits; and The i-th cyclic process of the first cyclic method is determining a scale factor for the i-th cyclic process, where i is a positive integer; scaling the first latent variable based on the scale factor of the i-th cyclic process to obtain an i-th scaled first latent variable; determining a number of coding bits of an entropy coding result of the i-th scaled first latent variable to obtain an i-th number of coding bits; performing an (i+1)th cyclic process of the first cyclic scheme if the i-th number of coding bits satisfies a continuous scaling condition; or and if the i-th number of coding bits does not satisfy a continuous scaling condition, terminating execution of the first cyclic scheme and determining the first variable scale factor based on the scale factor of the i-th cyclic operation.

7. The method further comprising: obtaining an entropy coding result of a third latent variable, the third latent variable being determined based on the second latent variable by using a context model; encoding the entropy coding result of the third latent variable into the bitstream; Further comprising: The method of claim 1 , wherein a total number of coding bits of the entropy coding result of the second latent variable and the entropy coding result of the third latent variable satisfies the preset encoding rate condition.

8. said step of determining a first variable scale factor based on said first latent variable further comprising: determining an initial number of coding bits based on the first latent variable; determining the first variable scale factor based on the initial number of coding bits and a target number of coding bits; 8. The method of claim 7, comprising:

9. determining an initial number of coding bits based on the first latent variable, determining a corresponding initial number of context coding bits and initial entropy coding model parameters based on the first latent variables by using the context model; determining a number of coding bits of an entropy coding result of the first latent variable based on the initial entropy coding model parameters to obtain a basic initial number of coding bits; determining an initial number of coding bits based on an initial number of context coding bits and the basic initial number of coding bits; 9. The method of claim 8, comprising:

10. When the initial number of coding bits is not equal to the target number of coding bits, the step of determining the first variable scale factor based on the initial number of coding bits and the target number of coding bits comprises: determining the first variable scale factor in a first cyclic manner based on the initial number of coding bits and the target number of coding bits; and The i-th cyclic process of the first cyclic method is determining a scale factor for the i-th cyclic process, where i is a positive integer; scaling the first latent variable based on the scale factor of the i-th cyclic process to obtain an i-th scaled first latent variable; determining an i-th number of context coding bits and an i-th entropy coding model parameter based on the i-th scaled first latent variable by using the context model; determining a number of coding bits of an entropy coding result of the i-th scaled first latent variable based on the i-th entropy coding model parameter to obtain an i-th base number of coding bits; determining an i-th number of coding bits based on the i-th number of context coding bits and the i-th base number of coding bits; performing an (i+1)th cyclic process of the first cyclic scheme if the i-th number of coding bits satisfies a continuous scaling condition; or if the i-th number of coding bits does not satisfy a continuous scaling condition, terminating execution of the first cyclic scheme and determining the first variable scale factor based on the scale factor of the i-th cyclic process.

9. The method of claim 8, comprising:

11. said step of determining a scale factor for said i-th cyclic process comprising: determining the scale factor for the i-th cyclic process based on a scale factor for the (i-1)-th cyclic process of the first cyclic scheme, the (i-1)-th number of coding bits, and a target number of coding bits; When i=1, the scale factor of the (i-1)th cyclic process is a first initial scale factor, and the (i-1)th number of coding bits is the initial number of coding bits; and 7. The method of claim 6, wherein the successive scaling condition includes: both the (i-1)th number of coding bits and the i-th number of coding bits are smaller than the target number of coding bits; or the successive scaling condition includes both the (i-1)th number of coding bits and the i-th number of coding bits are greater than the target number of coding bits.

12. When the initial number of coding bits is less than the target number of coding bits, the step of determining a scale factor for the i-th cyclic process comprises: Scaling a scale factor of an (i-1)th cyclic process of the first cyclic scheme based on a first step to obtain the scale factor of the i-th cyclic process; When i=1, the scale factor of the (i-1)th cyclic process is a first initial scale factor; and The method of claim 6 , wherein the successive scaling condition comprises: the i-th number of coding bits being less than the target number of coding bits.

13. When the initial number of coding bits is greater than the target number of coding bits, the step of determining a scale factor for the i-th cyclic process comprises: Scaling a scale factor of the (i-1)th cyclic process of the first cyclic scheme based on the second step to obtain the scale factor of the i-th cyclic process; When i=1, the scale factor of the (i-1)th cyclic process is a first initial scale factor; and The method of claim 6 , wherein the successive scaling condition comprises: the i-th number of coding bits being greater than the target number of coding bits.

14. determining the first variable scale factor based on the initial number of coding bits and a target number of coding bits; The method of claim 4 , comprising: determining a first initial scale factor as the first variable scale factor when the initial number of coding bits is less than the target number of coding bits.

15. determining the first variable scale factor based on the scale factor of the i-th cyclic process; determining the scale factor of the i-th cyclic process as the first variable scale factor when the i-th number of coding bits is equal to the target number of coding bits; or 7. The method of claim 6, further comprising: determining the first variable scale factor based on the scale factor of the i-th cyclic process and a scale factor of an (i-1)-th cyclic process of the first cyclic scheme when the i-th number of coding bits is not equal to the target number of coding bits.

16. determining the first variable scale factor based on the scale factor of the i-th cyclic process and a scale factor of an (i-1)-th cyclic process of the first cyclic scheme, determining an average of the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process; determining the first variable scale factor based on the average value; 16. The method of claim 15, comprising:

17. determining the first variable scale factor based on the scale factor of the i-th cyclic process and a scale factor of an (i-1)-th cyclic process of the first cyclic scheme, determining the first variable scale factor in a second cyclic manner based on the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process; and The j-th cyclic process of the second cyclic method is determining a third scale factor of the jth cyclic process based on the first scale factor of the jth cyclic process and the second scale factor of the jth cyclic process, where when j is equal to 1, the first scale factor of the jth cyclic process is one of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, and the second scale factor of the jth cyclic process is the other of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, the first scale factor of the jth cyclic process corresponds to a jth first number of coding bits, the second scale factor of the jth cyclic process corresponds to a jth second number of coding bits, the jth first number of coding bits is smaller than the jth second number of coding bits, and j is a positive integer; obtaining a j-th third number of coding bits, the j-th third number of coding bits being a number of coding bits of an entropy coding result of a first latent variable scaled based on the third scale factor of the j-th cyclic process; if the j-th third number of coding bits does not satisfy a consecutive cycle condition, terminating execution of the second cyclic scheme and determining the third scale factor of the j-th cyclic scheme as the first variable scale factor; performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as a second scale factor of the (j+1)th cyclic process and using the first scale factor of the jth cyclic process as a first scale factor of the (j+1)th cyclic process if the jth third number of coding bits satisfies a consecutive cycle condition, is greater than the target number of coding bits and is less than the jth second number of coding bits; or performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as a first scale factor of the (j+1)th cyclic process and using the second scale factor of the jth cyclic process as a second scale factor of the (j+1)th cyclic process if the jth third number of coding bits satisfies a consecutive cycle condition, is less than the target number of coding bits and is greater than the jth first number of coding bits.

16. The method of claim 15, comprising:

18. determining the first variable scale factor based on the scale factor of the i-th cyclic process and a scale factor of an (i-1)-th cyclic process of the first cyclic scheme, determining the first variable scale factor in a second cyclic manner based on the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process; and The j-th cyclic process of the second cyclic method is determining a third scale factor of the jth cyclic process based on the first scale factor of the jth cyclic process and the second scale factor of the jth cyclic process, where when j is equal to 1, the first scale factor of the jth cyclic process is one of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, and the second scale factor of the jth cyclic process is the other of the scale factor of the ith cyclic process and the (i-1)th cyclic process, the first scale factor of the jth cyclic process corresponds to a jth first number of coding bits, the second scale factor of the jth cyclic process corresponds to a jth second number of coding bits, the jth first number of coding bits is smaller than the jth second number of coding bits, and j is a positive integer; obtaining a j-th third number of coding bits, the j-th third number of coding bits being a number of coding bits of an entropy coding result of a first latent variable scaled based on the third scale factor of the j-th cyclic process; if the j-th third number of coding bits does not satisfy a consecutive cycle condition, terminating execution of the second cyclic scheme and determining the third scale factor of the j-th cyclic scheme as the first variable scale factor; if j reaches a maximum number of cycles and the j-th third number of coding bits satisfies a consecutive cycle condition, terminating execution of the second cyclic scheme and determining the first variable scale factor based on the first scale factor of the j-th cyclic process; performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as a second scale factor of the (j+1)th cyclic process and using the first scale factor of the jth cyclic process as a first scale factor of the (j+1)th cyclic process, if j has not reached a maximum number of cycles and the jth third number of coding bits satisfies a consecutive cycle condition, is greater than the target number of coding bits and is less than the jth second number of coding bits; or performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as a first scale factor of the (j+1)th cyclic process and using the second scale factor of the jth cyclic process as a second scale factor of the (j+1)th cyclic process if j has not reached a maximum number of cycles and the jth third number of coding bits satisfies a consecutive cycle condition, is smaller than the target number of coding bits and is larger than the jth first number of coding bits; 16. The method of claim 15, comprising:

19. determining the first variable scale factor based on the first scale factor of the j-th cyclic process, 20. The method of claim 18, further comprising: determining the first scale factor of the j-th cyclic process as the first variable scale factor when the media data to be encoded is encoded at a constant bit rate.

20. determining the first variable scale factor based on the first scale factor of the j-th cyclic process, determining a first difference between the target number of coding bits and a j-th first number of coding bits, and determining a second difference between the j-th second number of coding bits and the target number of coding bits when the media data to be encoded is encoded at a variable bit rate; determining the first scale factor of the j-th cyclic process as the first variable scale factor if the first difference is smaller than the second difference; if the second difference is smaller than the first difference, determining the second scale factor of the j-th cyclic process as the first variable scale factor; or determining the first scale factor of the j-th cyclic process as the first variable scale factor if the first difference is equal to the second difference, or determining the second scale factor of the j-th cyclic process as the first variable scale factor; 20. The method of claim 18, comprising:

21. 18. The method of claim 17, wherein when the media data to be encoded is encoded at a constant bit rate, the successive cycle condition includes: the jth third number of coding bits being greater than the target number of coding bits; or the successive cycle condition includes: the jth third number of coding bits being smaller than the target number of coding bits and a difference between the target number of coding bits and the jth third number of coding bits being greater than a bit number threshold.

22. 18. The method of claim 17, wherein when the media data to be encoded is encoded at a variable bit rate, the successive cycle condition includes: an absolute value of a difference between the target number of coding bits and the j-th third number of coding bits is greater than a bit number threshold.

23. determining the first variable scale factor based on the initial number of coding bits and a target number of coding bits; The method of claim 4 , comprising: determining a first initial scale factor as the first variable scale factor when the initial number of coding bits is equal to the target number of coding bits.

24. The method further comprising: determining a second variable scale factor based on the first latent variable; obtaining an entropy coding result of a fourth latent variable, the fourth latent variable being obtained by scaling a third latent variable based on the second variable scale factor, the third latent variable being determined based on the second latent variable by using a context model; encoding the entropy coding result of the fourth latent variable and the second variable scale factor into the bitstream; Further comprising: The method of claim 1 , wherein a total number of coding bits of the entropy coding result of the second latent variable and the entropy coding result of the fourth latent variable satisfies the preset encoding rate condition.

25. determining a first variable scale factor and a second variable scale factor based on the first latent variable; determining a corresponding initial number of context coding bits and initial entropy coding model parameters based on the first latent variables by using the context model; determining a number of coding bits of an entropy coding result of the first latent variable based on the initial entropy coding model parameters to obtain a basic initial number of coding bits; determining the first variable scale factor and the second variable scale factor based on an initial number of context coding bits, the base initial number of coding bits, and a target number of coding bits; 25. The method of claim 24, comprising:

26. determining the first variable scale factor and the second variable scale factor based on the initial number of context coding bits, the base initial number of coding bits, and a target number of coding bits; determining a basic target number of coding bits based on the target number of coding bits and at least one of the basic initial number of coding bits and the initial number of context coding bits; determining the first variable scale factor and a base actual number of coding bits based on a second initial scale factor, the base target number of coding bits, and the base initial number of coding bits, the base actual number of coding bits being a number of coding bits of an entropy coding result of a first latent variable scaled based on the first variable scale factor; determining a target number of context coding bits based on the target number of coding bits and the base actual number of coding bits; determining the second variable scale factor based on the target number of context coding bits and the initial number of context coding bits; 26. The method of claim 25, comprising:

27. determining the first variable scale factor and the second variable scale factor based on the initial number of context coding bits, the base initial number of coding bits, and a target number of coding bits; dividing the target number of coding bits into a base target number of coding bits and a target number of context coding bits; determining the first variable scale factor based on the base target number of coding bits and the base initial number of coding bits; determining the second variable scale factor based on the target number of context coding bits and the initial number of context coding bits; 26. The method of claim 25, comprising:

28. The method of claim 1 , wherein the media data is an audio signal, a video signal, or an image.

29. A decoding method comprising the steps of: determining a reconstructed second latent variable and a reconstructed first variable scale factor based on the bitstream, the first variable scale factor being used to enable a number of coding bits of a coding result of the second latent variable to satisfy a preset encoding rate condition for ensuring consistency based on a target number of coding bits; scaling the reconstructed second latent variable by dividing or multiplying by the reconstructed first variable scale factor to obtain a reconstructed first latent variable, the reconstructed first latent variable being indicative of a characteristic of the media data to be decoded; processing the reconstructed first latent variables by using a first decoding neural network model to obtain reconstructed media data; A method comprising:

30. determining a reconstructed second latent variable based on the bit stream; determining a reconstructed third latent variable based on the bitstream; determining the reconstructed second latent variable based on the bitstream and the reconstructed third latent variable; 30. The method of claim 29, comprising:

31. determining the reconstructed second latent variable based on the bitstream and the reconstructed third latent variable, processing the reconstructed third latent variables by using a context decoding neural network model to obtain reconstructed first entropy coding model parameters; determining the reconstructed second latent variables based on the bitstream and the reconstructed first entropy coding model parameters; 31. The method of claim 30, comprising:

32. determining a reconstructed second latent variable based on the bit stream; determining a reconstructed fourth latent variable and a reconstructed second variable scale factor based on the bitstream; determining the reconstructed second latent variable based on the bitstream, the reconstructed fourth latent variable, and the reconstructed second variable scale factor; 30. The method of claim 29, comprising:

33. determining the reconstructed second latent variable based on the bitstream, the reconstructed fourth latent variable, and the reconstructed second variable scale factor, scaling the reconstructed fourth latent variable based on the reconstructed second variable scale factor to obtain a reconstructed third latent variable; processing the reconstructed third latent variables by using a context decoding neural network model to obtain reconstructed second entropy coding model parameters; determining the reconstructed second latent variables based on the bitstream and the reconstructed second entropy coding model parameters; 33. The method of claim 32, comprising:

34. 30. The method of claim 29, wherein the media data is an audio signal, a video signal, or an image.

35. 1. An encoding device, comprising: a data processing module configured to process media data to be encoded to obtain a first latent variable, the first latent variable being indicative of a characteristic of the media data to be encoded; a scale factor determination module configured to determine a first variable scale factor based on the first latent variable, the first variable scale factor being used to enable a number of coding bits of a coding result of a second latent variable to satisfy a preset encoding rate condition for ensuring consistency based on a target number of coding bits, the second latent variable being obtained by scaling the first latent variable by multiplying or dividing it by the first variable scale factor; a first encoding result obtaining module configured to encode the second latent variable into a bitstream; a first encoding result writing module configured to encode the first variable scale factor into the bitstream; An apparatus comprising:

36. 36. The apparatus of claim 35, wherein when the media data to be encoded is encoded at a constant bit rate, the satisfying of a preset encoding rate condition includes: the number of coding bits being smaller than or equal to a target number of coding bits; or the satisfying of a preset encoding rate condition includes: the number of coding bits being smaller than or equal to a target number of coding bits and a difference between the number of coding bits and the target number of coding bits being smaller than a bit number threshold.

37. 36. The apparatus of claim 35, wherein when the media data to be encoded is encoded at a variable bit rate, the satisfying of a preset encoding rate condition comprises: an absolute value of a difference between the number of coding bits and a target number of coding bits being less than a bit number threshold.

38. the scale factor determination module: a bit number determination submodule configured to determine an initial number of coding bits based on the first latent variable; a first factor determination submodule configured to determine the first variable scale factor based on the initial number of coding bits and a target number of coding bits; 36. The apparatus of claim 35, comprising:

39. The initial number of coding bits is the number of coding bits of the entropy coding result of the first latent variable; or 39. The apparatus of claim 38, wherein the initial number of coding bits is a number of coding bits of an entropy coding result of a first latent variable scaled based on a first initial scale factor.

40. When the initial number of coding bits is not equal to the target number of coding bits, the first factor determination submodule specifically comprises: determining the first variable scale factor in a first cyclic manner based on the initial number of coding bits and the target number of coding bits; and The i-th cyclic process of the first cyclic method is determining a scale factor for the i-th cyclic process, where i is a positive integer; scaling the first latent variable based on the scale factor of the i-th cyclic process to obtain an i-th scaled first latent variable; determining a number of coding bits of an entropy coding result of the i-th scaled first latent variable to obtain an i-th number of coding bits; performing an (i+1)th cyclic operation of the first cyclic scheme if the i-th number of coding bits satisfies a continuous scaling condition; or if the i-th number of coding bits does not satisfy a continuous scaling condition, terminating execution of the first cyclic scheme and determining the first variable scale factor based on the scale factor of the i-th cyclic process.

39. The apparatus of claim 38, comprising:

41. The apparatus, a second encoding result acquisition module configured to acquire an entropy coding result of a third latent variable, the third latent variable being determined based on the second latent variable by using a context model; and a second encoding result writing module configured to encode the entropy coding result of the third latent variable into the bitstream; Further comprising:

36. The apparatus of claim 35, wherein a total number of coding bits of the entropy coding result of the second latent variable and the entropy coding result of the third latent variable satisfies the preset encoding rate condition.

42. the scale factor determination module: a bit number determination submodule configured to determine an initial number of coding bits based on the first latent variable; a first factor determination submodule configured to determine the first variable scale factor based on the initial number of coding bits and a target number of coding bits; 42. The apparatus of claim 41 , comprising:

43. The bit number determination submodule specifically includes: determining a corresponding initial number of context coding bits and initial entropy coding model parameters based on the first latent variable by using the context model; Determine a number of coding bits of an entropy coding result of the first latent variable based on the initial entropy coding model parameters to obtain a basic initial number of coding bits; determining an initial number of coding bits based on an initial number of context coding bits and a base initial number of coding bits; 43. The apparatus of claim 42, configured to:

44. When the initial number of coding bits is not equal to the target number of coding bits, the first factor determination submodule specifically comprises: determining the first variable scale factor in a first cyclic manner based on the initial number of coding bits and the target number of coding bits; and The i-th cyclic process of the first cyclic method is determining a scale factor for the i-th cyclic process, where i is a positive integer; scaling the first latent variable based on the scale factor of the i-th cyclic process to obtain an i-th scaled first latent variable; determining an i-th number of context coding bits and an i-th entropy coding model parameter based on the i-th scaled first latent variable by using the context model; determining a number of coding bits of an entropy coding result of the i-th scaled first latent variable based on the i-th entropy coding model parameter to obtain an i-th base number of coding bits; determining an i-th number of coding bits based on the i-th number of context coding bits and the i-th base number of coding bits; performing an (i+1)th cyclic operation of the first cyclic scheme if the i-th number of coding bits satisfies a continuous scaling condition; or if the i-th number of coding bits does not satisfy a continuous scaling condition, terminating execution of the first cyclic scheme and determining the first variable scale factor based on the scale factor of the i-th cyclic process.

43. The apparatus of claim 42, comprising:

45. The first factor determination submodule is specifically configured to determine the scale factor of the i-th cyclic process according to a scale factor of the (i-1)-th cyclic process of the first cyclic scheme, the (i-1)-th number of coding bits, and the target number of coding bits; When i=1, the scale factor of the (i-1)th cyclic process is a first initial scale factor, and the (i-1)th number of coding bits is the initial number of coding bits; and 45. The apparatus of claim 40 or 44, wherein the successive scaling condition includes both the (i-1)th number of coding bits and the i-th number of coding bits being smaller than the target number of coding bits, or the successive scaling condition includes both the (i-1)th number of coding bits and the i-th number of coding bits being greater than the target number of coding bits.

46. When the initial number of coding bits is smaller than the target number of coding bits, the first factor determination sub-module specifically comprises: scaling a scale factor of an (i-1)th cyclic process of the first cyclic scheme based on a first step to obtain the scale factor of the i-th cyclic process; When i=1, the scale factor of the (i-1)th cyclic process is a first initial scale factor; and 45. The apparatus of claim 40 or 44, wherein the successive scaling condition comprises: the i-th number of coding bits being less than the target number of coding bits.

47. When the initial number of coding bits is greater than the target number of coding bits, the first factor determination sub-module specifically comprises: scaling a scale factor of the (i-1)th cyclic process of the first cyclic scheme based on a second step to obtain the scale factor of the i-th cyclic process; When i=1, the scale factor of the (i-1)th cyclic process is a first initial scale factor; and 45. The apparatus of claim 40 or 44, wherein the successive scaling condition comprises: the i-th number of coding bits being greater than the target number of coding bits.

48. Specifically, the first factor determination submodule comprises:

43. The apparatus of claim 38 or 42, configured to determine a first initial scale factor as the first variable scale factor when the initial number of coding bits is smaller than the target number of coding bits.

49. Specifically, the first factor determination submodule comprises: When the i-th number of coding bits is equal to the target number of coding bits, determine the scale factor of the i-th cyclic process as the first variable scale factor; or determining the first variable scale factor based on the scale factor of the i-th cyclic process and a scale factor of a (i-1)-th cyclic process of the first cyclic scheme when the i-th number of coding bits is not equal to the target number of coding bits; 45. Apparatus according to claim 40 or 44, configured to:

50. Specifically, the first factor determination submodule comprises: determining an average value of the scale factor of the i-th first cyclic scheme and the scale factor of the (i-1)-th first cyclic scheme; determining the first variable scale factor based on the average value; 50. The apparatus of claim 49, configured to:

51. Specifically, the first factor determination submodule comprises: determining the first variable scale factor in a second cyclic scheme based on the scale factor of the i-th first cyclic scheme and the scale factor of the (i-1)-th first cyclic scheme; and The j-th cyclic process of the second cyclic method is determining a third scale factor of the jth cyclic process based on a first scale factor of the jth cyclic process and a second scale factor of the jth cyclic process, where when j is equal to 1, the first scale factor of the jth cyclic process is one of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, and the second scale factor of the jth cyclic process is the other of the scale factor of the ith cyclic process and the (i-1)th cyclic process, the first scale factor of the jth cyclic process corresponds to a jth first number of coding bits, the second scale factor of the jth cyclic process corresponds to a jth second number of coding bits, the jth first number of coding bits is smaller than the jth second number of coding bits, and j is a positive integer; obtaining a j-th third number of coding bits, the j-th third number of coding bits being a number of coding bits of an entropy coding result of a first latent variable scaled based on the third scale factor of the j-th cyclic process; if the j-th third number of coding bits does not satisfy a consecutive cycle condition, terminating execution of the second cyclic scheme and determining the third scale factor of the j-th cyclic scheme as the first variable scale factor; performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as a second scale factor of the (j+1)th cyclic process and using the first scale factor of the jth cyclic process as a first scale factor of the (j+1)th cyclic process if the jth third number of coding bits satisfies a consecutive cycle condition, is greater than the target number of coding bits and is less than the jth second number of coding bits; or performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as a first scale factor of the (j+1)th cyclic process and using the second scale factor of the jth cyclic process as a second scale factor of the (j+1)th cyclic process if the jth third number of coding bits satisfies a consecutive cycle condition, is less than the target number of coding bits and is greater than the jth first number of coding bits.

50. The apparatus of claim 49, comprising:

52. Specifically, the first factor determination submodule comprises: determining the first variable scale factor in a second cyclic manner based on the scale factor of the i-th cyclic process and the scale factor of the (i-1)-th cyclic process; and The j-th cyclic process of the second cyclic method is determining a third scale factor of the jth cyclic process based on a first scale factor of the jth cyclic process and a second scale factor of the jth cyclic process, where when j is equal to 1, the first scale factor of the jth cyclic process is one of the scale factor of the ith cyclic process and the scale factor of the (i-1)th cyclic process, and the second scale factor of the jth cyclic process is the other of the scale factor of the ith cyclic process and the (i-1)th cyclic process, the first scale factor of the jth cyclic process corresponds to a jth first number of coding bits, the second scale factor of the jth cyclic process corresponds to a jth second number of coding bits, the jth first number of coding bits is smaller than the jth second number of coding bits, and j is a positive integer; obtaining a j-th third number of coding bits, the j-th third number of coding bits being a number of coding bits of an entropy coding result of a first latent variable scaled based on the third scale factor of the j-th cyclic process; if the j-th third number of coding bits does not satisfy a consecutive cycle condition, terminating execution of the second cyclic scheme and determining the third scale factor of the j-th cyclic scheme as the first variable scale factor; terminating execution of the second cyclic scheme when j reaches a maximum number of cycles and the j-th third number of coding bits satisfies a consecutive cycle condition, and determining the first variable scale factor based on the first scale factor of the j-th cyclic process; If j has not reached a maximum number of cycles and the jth third number of coding bits satisfies a consecutive cycle condition, is greater than the target number of coding bits and is less than the jth second number of coding bits, performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as a second scale factor of the (j+1)th cyclic process and using the first scale factor of the jth cyclic process as a first scale factor of the (j+1)th cyclic process; or performing the (j+1)th cyclic process of the second cyclic scheme by using the third scale factor of the jth cyclic process as a first scale factor of the (j+1)th cyclic process and using the second scale factor of the jth cyclic process as a second scale factor of the (j+1)th cyclic process if j has not reached a maximum number of cycles and the jth third number of coding bits satisfies a consecutive cycle condition, is smaller than the target number of coding bits and is larger than the jth first number of coding bits; 50. The apparatus of claim 49, comprising:

53. Specifically, the first factor determination submodule comprises:

53. The apparatus of claim 52, configured to determine the first scale factor of the j-th cyclic process as the first variable scale factor when the media data to be encoded is encoded at a constant bit rate.

54. Specifically, the first factor determination submodule comprises: determining a first difference between the target number of coding bits and a j-th first number of coding bits when the to-be-encoded media data is encoded at a variable bit rate; determining a second difference between the j-th second number of coding bits and the target number of coding bits; If the first difference is smaller than the second difference, determining the first scale factor of the j-th cyclic process as the first variable scale factor. If the second difference is smaller than the first difference, determining the second scale factor of the j-th cyclic process as the first variable scale factor; or If the first difference is equal to the second difference, determine the first scale factor of the j-th cyclic process as the first variable scale factor, or determine the second scale factor of the j-th cyclic process as the first variable scale factor.

53. The apparatus of claim 52, configured to:

55. 52. The apparatus of claim 51, wherein when the media data to be encoded is encoded at a constant bit rate, the successive cycle condition includes: the jth third number of coding bits being greater than the target number of coding bits; or the successive cycle condition includes: the jth third number of coding bits being smaller than the target number of coding bits and a difference between the target number of coding bits and the jth third number of coding bits being greater than a bit number threshold.

56. 52. The apparatus of claim 51, wherein when the media data to be encoded is encoded at a variable bit rate, the successive cycle condition includes: an absolute value of a difference between the target number of coding bits and the jth third number of coding bits is greater than a bit number threshold.

57. Specifically, the first factor determination submodule comprises:

43. The apparatus of claim 38 or 42, configured to determine a first initial scale factor as the first variable scale factor when the initial number of coding bits is equal to the target number of coding bits.

58. The scale factor determination module is further configured to determine a second variable scale factor based on the first latent variable, the apparatus further comprising: a third encoding result acquisition module configured to acquire an entropy coding result of a fourth latent variable, the fourth latent variable being acquired by scaling a third latent variable based on the second variable scale factor, and the third latent variable being determined based on the second latent variable by using a context model; and a third encoding result writing module configured to encode the entropy coding result of the fourth latent variable and the second variable scale factor into the bitstream; Further comprising:

36. The apparatus of claim 35, wherein a total number of coding bits of the entropy coding result of the second latent variable and the entropy coding result of the fourth latent variable satisfies the preset encoding rate condition.

59. the scale factor determination module: a first determining sub-module configured to determine a corresponding initial number of context coding bits and initial entropy coding model parameters based on the first latent variables by using the context model; a second determining sub-module configured to determine a number of coding bits of an entropy coding result of the first latent variable based on the initial entropy coding model parameters to obtain a basic initial number of coding bits; a second factor determination submodule configured to determine the first variable scale factor and the second variable scale factor based on the initial number of context coding bits, the base initial number of coding bits, and a target number of coding bits; 59. The apparatus of claim 58, comprising:

60. Specifically, the second factor determination submodule comprises: determining a basic target number of coding bits based on the target number of coding bits and at least one of the basic initial number of coding bits and the initial number of context coding bits; determining a first variable scale factor and a base actual number of coding bits based on a second initial scale factor, the base target number of coding bits, and the base initial number of coding bits, the base actual number of coding bits being a number of coding bits of an entropy coding result of a first latent variable scaled based on the first variable scale factor; determining a target number of context coding bits based on the target number of coding bits and the base actual number of coding bits; determining the second variable scale factor based on the target number of context coding bits and the initial number of context coding bits; 60. The apparatus of claim 59, configured to:

61. Specifically, the second factor determination submodule comprises: Dividing the target number of coding bits into a base target number of coding bits and a target number of context coding bits; determining the first variable scale factor based on the base target number of coding bits and the base initial number of coding bits; determining the second variable scale factor based on the target number of context coding bits and the initial number of context coding bits; 60. The apparatus of claim 59, configured to:

62. 45. Apparatus according to any one of claims 35 to 44, wherein the media data is an audio signal, a video signal or an image.

63. A decoding device, comprising: a first determination module configured to determine a reconstructed second latent variable and a reconstructed first variable scale factor based on the bitstream, the first variable scale factor being used to enable a number of coding bits of a coding result of the second latent variable to satisfy a preset encoding rate condition for ensuring consistency based on a target number of coding bits; a variable scale module configured to scale the reconstructed second latent variable by dividing or multiplying by the reconstructed first variable scale factor to obtain a reconstructed first latent variable, the reconstructed first latent variable being indicative of a characteristic of the media data to be decoded; a variable processing module configured to process the reconstructed first latent variables to obtain reconstructed media data; An apparatus comprising:

64. The first determination module: a first determination submodule configured to determine a reconstructed third latent variable based on the bitstream; a second determination submodule configured to determine the reconstructed second latent variable based on the bitstream and the reconstructed third latent variable; 64. The apparatus of claim 63, comprising:

65. The second determination submodule specifically comprises: processing the reconstructed third latent variables by using a context decoding neural network model to obtain reconstructed first entropy coding model parameters; determining the reconstructed second latent variables based on the bitstream and the reconstructed first entropy coding model parameters; 65. The apparatus of claim 64, configured to:

66. The first determination module: a third determination sub-module configured to determine a reconstructed fourth latent variable and a reconstructed second variable scale factor based on the bitstream; a fourth determination submodule configured to determine the reconstructed second latent variable based on the bitstream, the reconstructed fourth latent variable, and the reconstructed second variable scale factor; 64. The apparatus of claim 63, comprising:

67. The fourth determination submodule specifically comprises: scaling the reconstructed fourth latent variable based on the reconstructed second variable scale factor to obtain a reconstructed third latent variable; processing the reconstructed third latent variables by using a context decoding neural network model to obtain reconstructed second entropy coding model parameters; determining the reconstructed second latent variables based on the bitstream and the reconstructed second entropy coding model parameters; 67. The apparatus of claim 66, configured to:

68. 68. Apparatus according to any one of claims 63 to 67, wherein the media data is an audio signal, a video signal or an image.

69. An encoder side device, the encoder side device comprising: a memory; and a processor; 29. An encoder side device, wherein the memory is configured to store a computer program and the processor is configured to execute the computer program stored in the memory to perform the encoding method of any one of claims 1 to 28.

70. A decoder-side device, the decoder-side device comprising: a memory; a processor; and 35. A decoder-side device, wherein the memory is configured to store a computer program and the processor is configured to execute the computer program stored in the memory to perform the decoding method according to any one of claims 29 to 34.

71. 29. A computer-readable storage medium having instructions stored thereon that, when executed on a computer, enable the computer to perform the steps of the method of any one of claims 1 to 28.

72. A computer-readable storage medium storing instructions which, when executed on a computer, enable the computer to perform steps of the methods recited in any one of claims 29 to 34.

73. A computer program which, when executed, performs the method according to any one of claims 1 to 28.

74. A computer program which, when executed, performs a method according to any one of claims 29 to 34.

Images