Audio data coding method and related apparatus and computer readable storage medium

Abstract

Embodiments of the present application provide an audio data coding method and related apparatus and a computer readable storage medium. A kind of audio data coding method can include: obtaining audio data to be encoded;Using coding neural network to process the audio data to be encoded to generate first latent variable;The first latent variable is quantized to obtain second latent variable;According to the first latent variable and the second latent variable, obtain latent variable compensation parameter;The latent variable compensation parameter is encoded, and the encoding result of the latent variable compensation parameter is written into code stream;Second latent variable is encoded, and the encoding result of the second latent variable is written into code stream. The embodiments of the present application are beneficial to reduce the related input distortion of decoding neural network, to improve the quality of final decoding signal.

Description

Technical Field

[0001] This application relates to the field of audio technology, and in particular to audio data encoding and decoding methods, related communication devices, and related computer-readable storage media. Background Technology

[0002] Currently, with social progress and continuous technological development, users' demands for audio and video services are increasing. How to provide users with higher quality services under limited coding bitrates, or to provide the same quality services using lower coding bitrates, has always been a key focus of audio and video codec research. Some international standards organizations (such as the 3rd Generation Partner Project) are also involved in the development of relevant standards to promote the advancement of audio and video services towards higher quality.

[0003] Audio and video compression is an indispensable part of media applications such as media communication and media broadcasting. With the development of the high-definition audio and video industry and the 3D industry, people's demand for audio and video quality is getting higher and higher, resulting in a rapid increase in the amount of audio and video data in media applications.

[0004] Traditional audio and video compression technologies are based on the fundamental principles of signal processing. They compress the original audio and video data by utilizing the correlation of signals in time and space, thereby reducing the amount of data and making it easier to transmit or store.

[0005] With the continuous development and maturation of artificial intelligence (AI) technology, more and more fields are beginning to use neural network technology to replace or combine with traditional technologies, and have made significant progress. In recent years, researchers have begun to introduce AI technology into the field of audio and video encoding and decoding. Summary of the Invention

[0006] This application provides some audio data encoding and decoding methods, related communication devices, and related computer-readable storage media.

[0007] The first aspect of this application provides an audio data encoding method, which may include: an audio encoder acquiring audio data to be encoded; processing the audio data to be encoded using an encoding neural network to generate a first latent variable; quantizing the first latent variable to obtain a second latent variable; obtaining latent variable compensation parameters based on the first latent variable and the second latent variable; encoding the latent variable compensation parameters and writing the encoding result of the latent variable compensation parameters into a bitstream; encoding the second latent variable and writing the encoding result of the second latent variable into the bitstream.

[0008] In this application, the audio data mentioned in the embodiments may be, for example, an audio signal and / or audio features. The audio signal may be a time-domain audio signal; it may also be a frequency-domain signal obtained by time-frequency transformation of a time-domain signal, such as a frequency-domain signal obtained by modified discrete cosine transform (MDCT) of a time-domain audio signal, or a frequency-domain signal obtained by fast fourier transform (FFT) of a time-domain audio signal; or the audio signal may be a signal filtered by a quadrature mirror filter (QMF); or the audio signal to be encoded may be a residual signal, such as a residual signal from other encoding methods or a residual signal filtered by a linear predictive encoder (LPC). The audio features to be encoded may be features extracted based on the audio signal, such as Mel-frequency cepstral coefficients or latent variables obtained through a neural network.

[0009] In this application, the encoded neural network mentioned in the embodiments can be a fully connected network (i.e., a fully connected neural network), a convolutional neural network (CNN), or other types of neural networks.

[0010] The audio data processing scheme in this application embodiment can be applied to AI codecs with or without a context model.

[0011] The encoding neural network can be pre-trained, and this application does not limit the specific network structure and training method of the encoding neural network. Processing the audio data to be encoded using the encoding neural network can be achieved by directly using the audio data to be encoded as the input of the encoding neural network, or by preprocessing the audio data to be encoded and then using the processed audio data as the input of the encoding neural network. The first latent variable can be the output of the encoding neural network, or it can be obtained by post-processing the output of the encoding neural network.

[0012] As can be seen, in the technical solution provided in this application, the encoder can obtain latent variable compensation parameters based on the quantization error of the latent variables (i.e., the latent variable compensation parameters can indicate the relevant quantization error caused by the encoder quantizing the latent variables), and write the encoding result of the latent variable compensation parameters into the bitstream. This provides a basis for the decoder to perform compensation processing on relevant latent variables based on the latent variable compensation parameters, which helps to reduce the distortion of the input data of the decoding neural network, thereby improving the quality of the final decoded signal. Specifically, for example, it helps to reduce holes in the frequency domain of the decoded audio signal and improve the subjective listening quality of the decoded audio signal.

[0013] In practical applications, there are various ways to obtain the latent variable compensation parameters based on the first latent variable and the second latent variable.

[0014] For example, obtaining the latent variable compensation parameter based on the first latent variable and the second latent variable may include: determining the elements in the third latent variable that satisfy preset conditions, wherein the third latent variable is obtained by dequantizing the second latent variable; obtaining the quantization error of the elements that satisfy the preset conditions based on the first latent variable and the third latent variable; and obtaining the latent variable compensation parameter based on the quantization error.

[0015] For example, obtaining the latent variable compensation parameter based on the first latent variable and the second latent variable may include: determining the elements in the second latent variable that meet preset conditions, obtaining the quantization error of the elements that meet the preset conditions based on the first latent variable and the second latent variable, and obtaining the latent variable compensation parameter based on the quantization error.

[0016] Other implementations of obtaining latent variable compensation parameters based on the first latent variable and the second latent variable will not be elaborated here.

[0017] The specific methods for obtaining latent variable compensation parameters may differ depending on the form of the latent variables. The basic principle is to determine the value of the latent variable compensation parameter based on the quantization error of the elements in the latent variables that satisfy preset conditions. The latent variable compensation parameter indicates the quantization error of the elements in the latent variables that satisfy preset conditions.

[0018] One possible implementation is to use the average quantization error of all elements that meet the conditions as the value of the latent variable compensation parameter. Another possible implementation is to use the weighted average quantization error of all elements that meet the conditions as the value of the latent variable compensation parameter, where the weighting value can be related to the element's index. Elements that meet the preset conditions can be elements in the second latent variable that also meet the preset conditions. Elements in the second latent variable that meet the preset conditions can be elements with the minimum quantized value, elements with the midpoint of the quantization range, or elements with values ​​less than or equal to a preset threshold. When the quantization range includes both positive and negative numbers, whether the quantized value is the midpoint of the quantization range can be chosen as the condition. For example, if the quantized value is an integer between -10 and 10, and the midpoint of the quantization range is 0, then the quantization error of elements with a quantization value of 0 can be used to calculate the average quantization error. Elements that meet the preset conditions can also be elements in the third latent variable that also meet the preset conditions. The third latent variable is obtained by dequantizing the second latent variable. Similarly, the elements in the third latent variable that satisfy the preset conditions can be elements whose element value is equal to the minimum value of the dequantized value, elements whose element value is equal to the preset value, or elements whose element value is less than or equal to a preset threshold.

[0019] The quantization error mentioned in the various embodiments of this application may be the amplitude difference or energy difference or its absolute value between the original value to be quantized and the quantized value, or it may be the amplitude difference or energy difference or its absolute value between the original value to be quantized and the dequantized value.

[0020] For example, encoding the latent variable compensation parameters and writing them into the bitstream may include: performing scalar or vector quantization on the latent variable compensation parameters to obtain quantization indices, and then encoding these quantization indices and writing them into the bitstream. Alternatively, the quantized latent variable compensation parameters may be entropy encoded before being written into the bitstream.

[0021] In some possible implementations, encoding the second latent variable and writing it into the bitstream may include: using an adjustable entropy encoding model to perform arithmetic encoding on the second latent variable and writing the arithmetic encoding result of the second latent variable into the bitstream, or using an entropy encoding model with a preset probability distribution to perform arithmetic encoding on the second latent variable and writing the arithmetic encoding result of the second latent variable into the bitstream.

[0022] It is understandable that the above preset conditions can be varied, and the preset conditions that meet the needs can be set according to the specific scenario.

[0023] For example, a third latent variable or a second latent variable may include a first element, wherein satisfying a preset condition may include: the value of the first element is less than or equal to a preset value. When the first element is an element in the second latent variable, the value of the first element is a quantized value. Or, when the first element is an element in the third latent variable, the value of the first element is a dequantized value.

[0024] In some possible implementations, obtaining the quantization error of the element satisfying the condition based on the first latent variable and the second latent variable may include: determining the quantization error of the first element based on a first value of the first element in the first latent variable and a second value of the first element in the second latent variable. Here, the quantization error of the first element may be, for example, the difference between the first value and the second value or the absolute value of the difference.

[0025] In some possible implementations, obtaining the quantization error of the element satisfying the preset condition based on the first latent variable and the third latent variable includes: determining the quantization error of the first element based on a first value of the first element in the first latent variable and a third value of the first element in the third latent variable. Here, the quantization error of the first element may be, for example, the difference between the first value and the third value, or the absolute value of the difference.

[0026] It is understandable that the latent variables output by different encoded neural networks may have different forms.

[0027] In the embodiments of this application, the encoding neural network can specifically be a fully connected neural network, a convolutional neural network, or other neural networks. When the encoding neural network is a convolutional neural network, the number of channels in this convolutional neural network can be 1, 2, 3, 4, or more.

[0028] For example, when the encoding neural network is a convolutional neural network, and the convolutional neural network includes only one channel, the latent variable compensation parameter can be a scalar, wherein the scalar is used to indicate the quantization error of all elements in the second latent variable that satisfy a preset condition, or the scalar is used to indicate the quantization error of all elements in the third latent variable that satisfy a preset condition.

[0029] For example, when the encoding neural network is a convolutional neural network, the convolutional neural network includes at least two channels, the second latent variable corresponds to the at least two channels, the at least two channels include the first channel, and the second latent variable is an m×n matrix.

[0030] The latent variable compensation parameter can be a scalar, which is used to indicate the quantization error of all elements in the second latent variable that satisfy the preset conditions.

[0031] Alternatively, the latent variable compensation parameter is a vector, wherein the dimension of the vector is equal to the number of channels in the convolutional neural network, the vector elements in the vector correspond one-to-one with the at least two channels, the first vector element in the vector elements corresponds to the first channel, the first vector element is used to indicate the quantization error of all elements in a submatrix of the m×n matrix that satisfy a preset condition, the first channel corresponds to the submatrix, wherein the number of elements in the submatrix is ​​less than m×n. m and n are positive integers.

[0032] For example, when the encoding neural network is a convolutional neural network, the convolutional neural network includes at least three channels, wherein the second latent variable corresponds to the at least three channels, for example, the second latent variable is an m×n matrix.

[0033] The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels in the convolutional neural network. The first vector element in the vector corresponds to at least two of the at least three channels, the at least two channels including a second channel and a first channel. The first vector element is used to indicate the quantization error of all elements in the first submatrix of the m×n matrix that satisfy a preset condition. The first vector element is also used to indicate the quantization error of all elements in the second submatrix of the m×n matrix that satisfy a preset condition. The first submatrix corresponds to the first channel, the second submatrix corresponds to the second channel, the number of elements in the first submatrix is ​​less than m×n, and the number of elements in the second submatrix is ​​less than m×n.

[0034] For example, when the encoding neural network is a convolutional neural network, wherein the convolutional neural network includes at least two channels, the third latent variable corresponds to the at least two channels, the at least two channels include the first channel, and the third latent variable is an m×n matrix. m and n are positive integers.

[0035] The latent variable compensation parameter is a scalar, which is used to indicate the quantization error of all elements in the third latent variable that satisfy the preset conditions.

[0036] Alternatively, the latent variable compensation parameter is a vector, wherein the dimension of the vector is equal to the number of channels in the convolutional neural network, the vector elements in the vector correspond one-to-one with the at least two channels, the first vector element in the vector elements corresponds to the first channel, and the first vector element is used to indicate the quantization error of all elements in a submatrix of the m×n matrix that satisfy a preset condition, wherein the first channel corresponds to the submatrix, and the number of elements in the submatrix is ​​less than m×n. m and n are positive integers.

[0037] For example, when the encoding neural network is a convolutional neural network, the convolutional neural network includes at least three channels, and the third latent variable corresponds to the at least three channels, wherein the third latent variable is an m×n matrix.

[0038] The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels in the convolutional neural network. The first vector element in the vector corresponds to at least two of the at least three channels, the at least two channels including a second channel and a first channel. The first vector element is used to indicate the quantization error of all elements in the first submatrix of the m×n matrix that satisfy a preset condition. The first vector element is also used to indicate the quantization error of all elements in the second submatrix of the m×n matrix that satisfy a preset condition. The first submatrix corresponds to the first channel, the second submatrix corresponds to the second channel, the number of elements in the first submatrix is ​​less than m×n, and the number of elements in the second submatrix is ​​less than m×n.

[0039] For example, when the encoding neural network is a convolutional neural network, wherein the convolutional neural network includes at least two channels, the second latent variable corresponds to the first channel of the at least two channels.

[0040] Wherein, the latent variable compensation parameter is a scalar, which is used to indicate the quantization error of all elements in the at least two latent variables corresponding to the at least two channels that satisfy the preset conditions, and the at least two latent variables include the second latent variable;

[0041] Alternatively, the latent variable compensation parameter is a vector, wherein the dimension of the vector may be equal to the number of channels of the convolutional neural network, the vector elements in the vector correspond one-to-one with the at least two channels, and the vector elements include a first vector element corresponding to the first channel, the first vector element being used to indicate the quantization error of all elements in the second latent variable that satisfy the preset conditions.

[0042] For example, when the encoding neural network is a convolutional neural network, wherein the convolutional neural network includes at least three channels, the second latent variable corresponds to the first channel among the at least three channels.

[0043] The latent variable compensation parameter is a vector, wherein the dimension of the vector is less than the number of channels in the convolutional neural network. A first vector element corresponds to at least two of the at least three channels, including a second channel and the first channel. The first vector element indicates the quantization error of all elements in the second latent variable that satisfy a preset condition. The first vector element also indicates the quantization error of all elements in another latent variable corresponding to the second channel that satisfy a preset condition.

[0044] For example, when the encoding neural network is a convolutional neural network, wherein the convolutional neural network includes at least two channels, the third latent variable corresponds to the first channel of the at least two channels.

[0045] Wherein, the latent variable compensation parameter is a scalar, which is used to indicate the quantization error of all elements in the at least two latent variables corresponding to the at least two channels that satisfy the preset conditions, and the at least two latent variables include the third latent variable;

[0046] Alternatively, the latent variable compensation parameter is a vector, wherein the dimension of the vector is equal to the number of channels of the convolutional neural network, the vector elements in the vector correspond one-to-one with the at least two channels, and the vector element corresponding to the first channel is used to indicate the quantization error of all elements in the third latent variable that satisfy the preset conditions.

[0047] For example, when the encoding neural network is a convolutional neural network, wherein the convolutional neural network includes at least three channels, the third latent variable corresponds to the first channel among the at least three channels.

[0048] Wherein, the latent variable compensation parameter is a vector, wherein the dimension of the vector is less than the number of channels of the convolutional neural network. A first vector element in the vector corresponds to at least two of the at least three channels, the at least two channels including the second channel and the first channel. The first vector element is used to indicate the quantization error of all elements in the third latent variable that satisfy a preset condition. The first vector element is also used to indicate the quantization error of all elements in another latent variable corresponding to the second channel that satisfy a preset condition.

[0049] In some possible implementations, when the first vector element corresponds to both the first and second channels, this correspondence can be written into the bitstream. However, if the encoding and decoding sections use the same element-to-channel correspondence by default, the correspondence between the first vector element and the first and second channels may not need to be written into the bitstream.

[0050] It is understandable that the above examples of obtaining the latent variable compensation parameters corresponding to convolutional neural networks or fully connected networks are conducive to meeting the flexible needs of various application scenarios, thereby improving the scenario adaptability of the solution.

[0051] A second aspect of this application provides an audio data decoding method, which may include: obtaining latent variable compensation parameters and a third latent variable based on the bitstream (the value of the current element in the third latent variable may be a dequantized value); compensating the third latent variable according to the latent variable compensation parameters to obtain a reconstructed first latent variable; and processing the reconstructed first latent variable using a decoding neural network to generate decoded audio data.

[0052] The process of obtaining the latent variable compensation parameter and the third latent variable based on the bitstream may include: decoding the latent variable compensation parameter and the second latent variable from the bitstream, wherein the value of the current element in the second latent variable is a quantized value; and performing dequantization processing on the second latent variable to obtain the third latent variable. Alternatively, the third latent variable may be obtained directly from the bitstream using a specific decoder (in this case, it is unnecessary to perform the steps of decoding the second latent variable from the bitstream and dequantizing the second latent variable to obtain the third latent variable).

[0053] As can be seen, the solution provided in this application's embodiments, because the encoding end writes latent variable compensation parameters into the bitstream, and these latent variable compensation parameters can indicate the relevant quantization errors caused by the encoding end quantizing the latent variables, allows the decoding end to compensate for the latent variables based on the latent variable compensation parameters after obtaining the bitstream. This helps reduce the distortion of the input data of the decoding neural network, thereby improving the quality of the final decoded signal. Specifically, for example, it helps reduce holes in the frequency domain of the decoded audio signal, improving the subjective auditory quality of the decoded audio signal.

[0054] For example, compensating the third latent variable according to the latent variable compensation parameter to obtain the reconstructed first latent variable may include: determining the elements in the third latent variable that meet preset conditions according to the second latent variable or the third latent variable; and compensating the elements in the third latent variable that meet preset conditions according to the latent variable compensation parameter to obtain the reconstructed first latent variable.

[0055] In practical applications, there are many ways to compensate elements in the third latent variable that meet preset conditions according to the latent variable compensation parameters to obtain the reconstructed first latent variable. For example, compensating elements in the third latent variable that meet preset conditions according to the latent variable compensation parameters to obtain the reconstructed first latent variable may include:

[0056] Random noise is generated; the amplitude or energy of the generated random noise is adjusted according to the latent variable compensation parameters to obtain amplitude-adjusted random noise; the elements in the third latent variable that meet the preset conditions are compensated according to the amplitude-adjusted random noise to obtain the reconstructed first latent variable.

[0057] It is understandable that the above preset conditions can be varied, and the preset conditions that meet the needs can be set according to the specific scenario.

[0058] For example, a third latent variable or a second latent variable may include a first element, and satisfying a preset condition may include: the value of the first element is less than or equal to a preset value. When the first element is an element in the second latent variable, the value of the first element is a quantized value. Or, when the first element is an element in the third latent variable, the value of the first element is a dequantized value.

[0059] It is understandable that different structures of decoding neural networks may process latent variables in different forms.

[0060] In the embodiments of this application, the decoding neural network can specifically be a fully connected neural network, a convolutional neural network, or other neural networks. When the decoding neural network is a convolutional neural network, the number of channels of this convolutional neural network can be 1, 2, 3, 4, or more.

[0061] For example, when the decoding neural network is a convolutional neural network, the convolutional neural network includes only one channel; the latent variable compensation parameter is a scalar, wherein the scalar is used to compensate all elements in the third latent variable that satisfy the preset conditions.

[0062] For example, when the decoding neural network is a convolutional neural network, the convolutional neural network includes at least two channels, the third latent variable corresponds to the at least two channels, the at least two channels include the first channel, and the third latent variable is an m×n matrix. m and n are positive integers.

[0063] The latent variable compensation parameter can be a scalar, which is used to compensate all elements in the third latent variable that meet the preset conditions.

[0064] Alternatively, the latent variable compensation parameter can be a vector, wherein the dimension of the vector is equal to the number of channels in the convolutional neural network, the vector elements in the vector correspond one-to-one with the at least two channels, the first vector element corresponds to the first channel, and the first vector element is used to compensate for all elements in a submatrix of the m×n matrix that satisfy a preset condition, the first channel corresponds to the submatrix, wherein the number of elements in the submatrix is ​​less than m×n. m and n are positive integers, the number of elements in the first submatrix is ​​less than m×n, and the number of elements in the second submatrix is ​​less than m×n.

[0065] For example, when the decoding neural network is a convolutional neural network, the convolutional neural network includes at least three channels, and a third latent variable corresponds to the at least three channels. The third latent variable is an m×n matrix. The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels in the convolutional neural network. A first vector element in the vector corresponds to at least two of the at least three channels, including a second channel and a first channel. The first vector element is used to compensate for all elements in the first submatrix of the m×n matrix that satisfy a preset condition. The first vector element is also used to compensate for all elements in the second submatrix of the m×n matrix that satisfy a preset condition. The first submatrix corresponds to the first channel, and the second submatrix corresponds to the second channel. The number of elements in the first submatrix is ​​less than m×n, and the number of elements in the second submatrix is ​​less than m×n. Here, m and n are positive integers.

[0066] For example, when the decoding neural network is a convolutional neural network, wherein the convolutional neural network includes at least two channels, the third latent variable corresponds to the first channel of the at least two channels.

[0067] Wherein, the latent variable compensation parameter is a scalar, and the scalar is used to compensate all elements in the at least two latent variables corresponding to the at least two channels that satisfy the preset conditions, wherein the at least two latent variables include the third latent variable;

[0068] Alternatively, the latent variable compensation parameter is a vector, wherein the dimension of the vector is equal to the number of channels of the convolutional neural network, the vector elements in the vector correspond one-to-one with the at least two channels, and the vector elements in the vector corresponding to the first channel are used to compensate all elements in the third latent variable that meet the preset conditions.

[0069] For example, when the decoding neural network is a convolutional neural network, the convolutional neural network includes at least three channels, and the third latent variable corresponds to the first channel among the at least three channels. The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels in the convolutional neural network. The first vector element in the vector corresponds to at least two channels among the at least three channels, the at least two channels including the second channel and the first channel. The first vector element is used to compensate for all elements in the third latent variable that satisfy a preset condition, and the first vector element is also used to compensate for all elements in another latent variable corresponding to the second channel that satisfy a preset condition.

[0070] The third aspect of this application also provides an audio encoder, which may include several functional units, and the several functional units work together to complete any one of the audio data encoding methods provided in the first aspect.

[0071] For example, an audio encoder may include an acquisition unit, a parameter processing unit, and an encoding unit.

[0072] The acquisition unit is used to acquire the audio data to be encoded.

[0073] The parameter processing unit is used to process the audio data to be encoded using an encoding neural network to generate a first latent variable; to quantize the first latent variable to obtain a second latent variable; and to obtain latent variable compensation parameters based on the first latent variable and the second latent variable.

[0074] The encoding unit is used to encode the latent variable compensation parameters and write the encoding result of the latent variable compensation parameters into the bitstream; and to encode the second latent variable and write the encoding result of the second latent variable into the bitstream.

[0075] The fourth aspect of this application also provides an audio decoder, which may include several functional units, and the several functional units work together to complete any of the audio data decoding methods provided in the second aspect.

[0076] For example, an audio decoder may include a decoding unit and a parameter processing unit.

[0077] The parameter processing unit is used to obtain latent variable compensation parameters and a third latent variable (the value of the current element in the third latent variable can be a dequantized value) based on the bitstream; and to perform compensation processing on the third latent variable according to the latent variable compensation parameters to obtain the reconstructed first latent variable.

[0078] A decoding unit is used to process the reconstructed first latent variable using a decoding neural network to generate decoded audio data.

[0079] The process of obtaining the latent variable compensation parameter and the third latent variable based on the bitstream may include: decoding the latent variable compensation parameter and the second latent variable from the bitstream, wherein the value of the current element in the second latent variable is a quantized value; and performing dequantization processing on the second latent variable to obtain the third latent variable. Alternatively, the third latent variable may be obtained directly from the bitstream using a specific decoder (in this case, it is unnecessary to perform the steps of decoding the second latent variable from the bitstream and dequantizing the second latent variable to obtain the third latent variable).

[0080] A fifth aspect of this application provides an audio encoder, which may include: a processor coupled to a memory, the memory storing a program, wherein when the program instructions stored in the memory are executed by the processor, any one of the methods provided in the first aspect is implemented.

[0081] A sixth aspect of this application provides an audio decoder, which may include: a processor coupled to a memory, the memory storing a program, and when the program instructions stored in the memory are executed by the processor, implementing any of the methods provided in the second aspect.

[0082] A seventh aspect of this application provides a communication system, including: an encoder and an audio decoder; the audio encoder is any type of audio encoder provided in this application. The audio decoder is any type of audio decoder provided in this application.

[0083] An eighth aspect of this application provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform some or all of the steps of any of the methods provided in the first aspect.

[0084] A ninth aspect of this application provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform some or all of the steps of any of the methods provided in the second aspect.

[0085] The tenth aspect of this application provides a computer-readable storage medium storing a bitstream, the bitstream being obtained based on any of the audio data encoding methods provided in the first aspect.

[0086] The eleventh aspect of this application provides a network device including a processor and a memory, wherein the processor is coupled to the memory and is used to read and execute a program stored in the memory to implement some or all of the steps of any of the methods provided in the first or second aspect.

[0087] The network device may be, for example, a chip or a system-on-a-chip.

[0088] The twelfth aspect of this application provides a computer program product, wherein the computer program product includes a computer program that, when run on a computer, causes the computer to perform some or all of the steps of any one of the methods provided in the first or second aspect. Attached Figure Description

[0089] The accompanying drawings, which are used in the description of the embodiments or prior art, will be briefly introduced below.

[0090] Figure 1A and Figure 1B This is a schematic diagram illustrating a scenario where the audio encoding / decoding scheme provided in this application is applied to an audio terminal.

[0091] Figures 1C-1D This is a schematic diagram illustrating the audio encoding and decoding process of a network device in a wired or wireless network, as provided in an embodiment of this application.

[0092] Figure 1E This is a schematic diagram of audio encoding and decoding in audio communication provided in the embodiments of this application.

[0093] Figure 1F This is a schematic diagram of the system architecture for broadcast television applications provided in the embodiments of this application.

[0094] Figure 2 This is a flowchart illustrating an audio data encoding method provided in an embodiment of this application.

[0095] Figure 3 This is a flowchart illustrating another audio data decoding method provided in an embodiment of this application.

[0096] Figure 4A This is a flowchart illustrating another audio data encoding method provided in an embodiment of this application.

[0097] Figure 4B This is a schematic diagram of a potential variable provided in an embodiment of this application.

[0098] Figure 5 This is a flowchart illustrating another audio data decoding method provided in an embodiment of this application.

[0099] Figure 6A This is a flowchart illustrating another audio data encoding method provided in an embodiment of this application.

[0100] Figure 6B This is a schematic diagram of another potential variable provided for an embodiment of this application.

[0101] Figure 7 This is a flowchart illustrating another audio data decoding method provided in an embodiment of this application.

[0102] Figure 8A and Figure 8B This is a flowchart illustrating another audio data encoding method provided in an embodiment of this application.

[0103] Figure 9A and Figure 9B This is a flowchart illustrating another audio data decoding method provided in an embodiment of this application.

[0104] Figure 10A and Figure 10B This is a flowchart illustrating another audio data encoding method provided in an embodiment of this application.

[0105] Figure 11A and Figure 11B This is a flowchart illustrating another audio data decoding method provided in an embodiment of this application.

[0106] Figure 12A and Figure 12B This is a flowchart illustrating another audio data encoding method provided in an embodiment of this application.

[0107] Figure 13A and Figure 13B This is a flowchart illustrating another audio data decoding method provided in an embodiment of this application.

[0108] Figure 14A and Figure 14B This is a schematic diagram showing the comparison of potential variables before and after quantification, as provided in an embodiment of this application.

[0109] Figure 15 This is a schematic diagram of an audio encoder provided in an embodiment of this application.

[0110] Figure 16 This is a schematic diagram of an audio decoder provided in an embodiment of this application.

[0111] Figure 17 This is a schematic diagram of another audio encoder provided in an embodiment of this application.

[0112] Figure 18 This is a schematic diagram of another audio decoder provided in an embodiment of this application.

[0113] Figure 19 This is a schematic diagram of a communication system provided in an embodiment of this application.

[0114] Figure 20 This is a schematic diagram of a network device provided in an embodiment of this application. Detailed Implementation

[0115] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0116] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order.

[0117] See Figures 1A to 1F The following describes the network architecture that the audio codec scheme of this application may be applied to. The audio codec scheme may be applied to audio terminals (such as wired or wireless communication terminals), or to network devices in wired or wireless networks.

[0118] in, Figure 1A and Figure 1B This illustrates a scenario where an audio codec solution is applied to an audio terminal, where the specific product form of the audio terminal can be... Figure 1A The diagram shows terminals 1, 2, and 3, but not limited to these. For example, in audio communication, the audio acquisition unit in the transmitting terminal can acquire audio signals, the audio encoder can encode the audio signals acquired by the audio acquisition unit, and the channel encoder performs channel encoding on the audio encoded signals obtained by the audio encoder to obtain a bitstream. The bitstream is transmitted through a wireless network or wireless network. Correspondingly, the channel decoder in the receiving terminal performs channel decoding on the received bitstream, and then the audio decoder decodes it to obtain the audio signal, which can then be played back by an audio playback unit.

[0119] See Figure 1C , Figure 1D and Figure 1E If a network device in a wired or wireless network needs to perform transcoding, the network device can perform the corresponding audio encoding and decoding processing.

[0120] See Figure 1F The audio encoding and decoding scheme of this application can also be applied to the field of broadcast television. A system architecture for application in the field of broadcast television can be as follows: Figure 1F For example, in a live broadcast scenario, the 3D audio signal generated from the 3D audio production of the live program is encoded using the 3D audio encoding method of this application to obtain a bitstream, which is then transmitted to the user side via the broadcast network. The 3D audio signal is then decoded and reconstructed by the 3D audio decoder in the set-top box, and played back by the speaker array. In a post-production scenario, the 3D audio signal generated from the 3D audio production of the post-production program is encoded using the 3D audio encoding method of this application to obtain a bitstream, which is then transmitted to the user side via the broadcast network or the Internet. The 3D audio signal is then decoded and reconstructed by the 3D audio decoder in the network receiver or mobile terminal, and played back by the speaker array or headphones.

[0121] Furthermore, the audio encoding / decoding scheme of this application can also be applied to the audio encoding / decoding module in virtual reality (VR streaming) services.

[0122] For example, the end-to-end audio signal processing flow can be as follows: Audio signal A passes through the acquisition module and undergoes preprocessing. Preprocessing includes filtering out low-frequency components, typically using 20Hz or 50Hz as a dividing point, extracting location information, and then encoding (Audio encoding) and encapsulating (File / Segment encapsulation) before delivery to the decoding end. Correspondingly, the decoding end first unpacks (File / Segment decapsulation), then decodes (Audio decoding), performs binaural rendering on the decoded signal, and maps the rendered signal onto the listener's headphones. These can be standalone headphones or headphones on glasses devices such as the HTC VIVE.

[0123] Specifically, the audio codec solution of this application can be applied to practical products including wireless access network equipment, core network media gateways, transcoding equipment, media resource servers, mobile terminals, fixed network terminals, etc. It can also be applied to audio codecs in VRstreaming services.

[0124] The following section introduces some audio data encoding and decoding schemes.

[0125] See Figure 2 , Figure 2 This is a flowchart illustrating an audio data encoding method provided in an embodiment of this application. An audio data encoding method (corresponding to an encoding end) may include:

[0126] 201. Obtain the audio data to be encoded.

[0127] The audio data mentioned in the embodiments of this application may be, for example, audio signals and / or audio features.

[0128] The audio signal to be encoded can be a time-domain audio signal; it can also be a frequency-domain signal obtained by time-frequency transformation of a time-domain signal, such as the frequency-domain signal obtained by MDCT transformation of a time-domain audio signal, or the frequency-domain signal obtained by FFT transformation of a time-domain audio signal; the signal to be encoded can also be a signal after QMF filtering; the signal to be encoded can also be a residual signal, such as the residual signal after other encoding or the residual signal after LPC filtering, etc. The audio features to be encoded can be features extracted based on the audio signal, such as Mel-frequency cepstral coefficients or latent variables obtained through neural networks.

[0129] 202. The audio data to be encoded is processed using an encoding neural network to generate a first latent variable.

[0130] The encoding neural network can be pre-trained, and this application does not limit the specific network structure and training method of the encoding neural network. Processing the audio data to be encoded using the encoding neural network can be achieved by directly using the audio data to be encoded as the input of the encoding neural network, or by preprocessing the audio data to be encoded and then using the processed audio data as the input of the encoding neural network. The first latent variable can be the output of the encoding neural network, or it can be obtained by post-processing the output of the encoding neural network.

[0131] 203. Quantify the first latent variable to obtain the second latent variable.

[0132] 204. Obtain the latent variable compensation parameters based on the first latent variable and the second latent variable.

[0133] In practical applications, there are various ways to obtain the latent variable compensation parameters based on the first latent variable and the second latent variable.

[0134] For example, obtaining the latent variable compensation parameter based on the first latent variable and the second latent variable may include: determining the elements in the third latent variable that satisfy preset conditions, wherein the third latent variable is obtained by dequantizing the second latent variable; obtaining the quantization error of the elements that satisfy the preset conditions based on the first latent variable and the third latent variable; and obtaining the latent variable compensation parameter based on the quantization error.

[0135] For example, obtaining the latent variable compensation parameter based on the first latent variable and the second latent variable may include: determining the elements in the second latent variable that satisfy a preset condition; obtaining the quantization error of the elements that satisfy the preset condition based on the first latent variable and the third latent variable, wherein the third latent variable is obtained by dequantizing the second latent variable; and obtaining the latent variable compensation parameter based on the quantization error.

[0136] For example, obtaining the latent variable compensation parameter based on the first latent variable and the second latent variable may include: determining the elements in the second latent variable that meet preset conditions, obtaining the quantization error of the elements that meet the preset conditions based on the first latent variable and the second latent variable, and obtaining the latent variable compensation parameter based on the quantization error.

[0137] Other implementations of obtaining latent variable compensation parameters based on the first latent variable and the second latent variable will not be elaborated here.

[0138] 205. Encode the latent variable compensation parameters and write the encoding results of the latent variable compensation parameters into the bitstream.

[0139] For example, encoding the latent variable compensation parameters and writing them into the bitstream may include: performing scalar or vector quantization on the latent variable compensation parameters to obtain quantization indices, and then encoding these quantization indices and writing them into the bitstream. Alternatively, the latent variable compensation parameters may be quantized, the quantization results may be entropy encoded, and the encoded results may be written into the bitstream.

[0140] 206. Encode the second latent variable and write the encoding result of the second latent variable into the bitstream.

[0141] For example, encoding the second latent variable and writing it into the bitstream can include: using an adjustable entropy encoding model to perform arithmetic encoding on the second latent variable and then writing it into the bitstream, or using an entropy encoding model with a preset probability distribution to perform arithmetic encoding on the second latent variable and then writing it into the bitstream.

[0142] It is understandable that there is no necessary order between steps 206 and steps 204-205. Step 206 may be executed before steps 204-205, or after steps 204-205, or in parallel with steps 204-205.

[0143] The aforementioned audio data processing methods can be applied to AI codecs with or without context models.

[0144] It is understandable that the above preset conditions can be varied, and the preset conditions that meet the needs can be set according to the specific scenario.

[0145] For example, a third latent variable or a second latent variable may include a first element, wherein satisfying a preset condition may include: the value of the first element is less than or equal to a preset value. When the first element is an element in the second latent variable, the value of the first element is a quantized value. Or, when the first element is an element in the third latent variable, the value of the first element is a dequantized value.

[0146] In some possible implementations, obtaining the quantization error of the element satisfying the condition based on the first latent variable and the second latent variable may include: determining the quantization error of the first element based on a first value of the first element in the first latent variable and a second value of the first element in the second latent variable. Here, the quantization error of the first element may be, for example, the difference between the first value and the second value or the absolute value of the difference.

[0147] In some possible implementations, obtaining the quantization error of the element satisfying the preset condition based on the first latent variable and the third latent variable includes: determining the quantization error of the first element based on a first value of the first element in the first latent variable and a third value of the first element in the third latent variable. Here, the quantization error of the first element may be, for example, the difference between the first value and the third value, or the absolute value of the difference.

[0148] It is understandable that the latent variables output by different encoded neural networks may have different forms.

[0149] In the embodiments of this application, the encoding neural network can specifically be a fully connected neural network, a convolutional neural network, or other neural networks. When the encoding neural network is a convolutional neural network, the number of channels in this convolutional neural network can be 1, 2, 3, 4, or more.

[0150] For example, when the encoding neural network is a convolutional neural network, and the convolutional neural network includes only one channel, the latent variable compensation parameter can be a scalar, wherein the scalar is used to indicate the quantization error of all elements in the second latent variable that satisfy a preset condition, or the scalar is used to indicate the quantization error of all elements in the third latent variable that satisfy a preset condition.

[0151] For example, when the encoding neural network is a convolutional neural network, the convolutional neural network includes at least two channels, the second latent variable corresponds to the at least two channels, the at least two channels include the first channel, and the second latent variable is an m×n matrix.

[0152] The latent variable compensation parameter can be a scalar, which is used to indicate the quantization error of all elements in the second latent variable that satisfy the preset conditions.

[0153] Alternatively, the latent variable compensation parameter is a vector, wherein the dimension of the vector is equal to the number of channels of the convolutional neural network, the vector elements in the vector correspond one-to-one with the at least two channels, the first vector element in the vector elements corresponds to the first channel, the first vector element is used to indicate the quantization error of all elements in the submatrix of the m×n matrix that satisfy the preset conditions, the first channel corresponds to the submatrix, wherein the number of elements in the submatrix is ​​less than m×n.

[0154] For example, when the encoding neural network is a convolutional neural network, the convolutional neural network includes at least three channels, wherein the second latent variable corresponds to the at least three channels, for example, the second latent variable is an m×n matrix.

[0155] The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels in the convolutional neural network. The first vector element in the vector corresponds to at least two of the at least three channels, the at least two channels including a second channel and a first channel. The first vector element is used to indicate the quantization error of all elements in the first submatrix of the m×n matrix that satisfy a preset condition. The first vector element is also used to indicate the quantization error of all elements in the second submatrix of the m×n matrix that satisfy a preset condition. The first submatrix corresponds to the first channel, the second submatrix corresponds to the second channel, the number of elements in the first submatrix is ​​less than m×n, and the number of elements in the second submatrix is ​​less than m×n.

[0156] For example, when the encoding neural network is a convolutional neural network, the convolutional neural network includes at least two channels, the third latent variable corresponds to the at least two channels, the at least two channels include the first channel, and the third latent variable is an m×n matrix.

[0157] The latent variable compensation parameter is a scalar, which is used to indicate the quantization error of all elements in the third latent variable that satisfy the preset conditions.

[0158] Alternatively, the latent variable compensation parameter is a vector, wherein the dimension of the vector is equal to the number of channels of the convolutional neural network, the vector elements in the vector correspond one-to-one with the at least two channels, the first vector element in the vector elements corresponds to the first channel, and the first vector element is used to indicate the quantization error of all elements in the submatrix of the m×n matrix that satisfy the preset conditions, wherein the first channel corresponds to the submatrix, and the number of elements in the submatrix is ​​less than m×n.

[0159] For example, when the encoding neural network is a convolutional neural network, the convolutional neural network includes at least three channels, and the third latent variable corresponds to the at least three channels, wherein the third latent variable is an m×n matrix.

[0160] The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels in the convolutional neural network. A first vector element corresponds to at least two of the at least three channels, including a second channel and a first channel. The first vector element indicates the quantization error of all elements in a first submatrix of the m×n matrix that satisfy a preset condition. The first vector element also indicates the quantization error of all elements in a second submatrix of the m×n matrix that satisfy a preset condition. The first submatrix corresponds to the first channel, and the second submatrix corresponds to the second channel. The number of elements in both the first and second submatrixes is less than m×n.

[0161] For example, when the encoding neural network is a convolutional neural network, wherein the convolutional neural network includes at least two channels, the second latent variable corresponds to the first channel of the at least two channels.

[0162] Wherein, the latent variable compensation parameter is a scalar, which is used to indicate the quantization error of all elements in the at least two latent variables corresponding to the at least two channels that satisfy the preset conditions, and the at least two latent variables include the second latent variable;

[0163] Alternatively, the latent variable compensation parameter is a vector, wherein the dimension of the vector can be equal to the number of channels of the convolutional neural network, the vector elements in the vector correspond one-to-one with the at least two channels, and the vector elements in the vector corresponding to the first channel are used to indicate the quantization error of all elements in the second latent variable that satisfy the preset conditions.

[0164] For example, when the encoding neural network is a convolutional neural network, wherein the convolutional neural network includes at least three channels, the second latent variable corresponds to the first channel among the at least three channels.

[0165] Wherein, the latent variable compensation parameter is a vector, wherein the dimension of the vector is less than the number of channels of the convolutional neural network, wherein the first vector element in the vector corresponds to at least two of the at least three channels, the at least two channels including the second channel and the first channel, the first vector element is used to indicate the quantization error of all elements in the second latent variable that satisfy the preset conditions, and the first vector element is also used to indicate the quantization error of all elements in another latent variable corresponding to the second channel that satisfy the preset conditions.

[0166] For example, when the encoding neural network is a convolutional neural network, wherein the convolutional neural network includes at least two channels, the third latent variable corresponds to the first channel of the at least two channels.

[0167] Wherein, the latent variable compensation parameter is a scalar, which is used to indicate the quantization error of all elements in the at least two latent variables corresponding to the at least two channels that satisfy the preset conditions, and the at least two latent variables include the third latent variable;

[0168] Alternatively, the latent variable compensation parameter is a vector, wherein the dimension of the vector is equal to the number of channels of the convolutional neural network, the vector elements in the vector correspond one-to-one with the at least two channels, and the vector element corresponding to the first channel is used to indicate the quantization error of all elements in the third latent variable that satisfy the preset conditions.

[0169] For example, when the encoding neural network is a convolutional neural network, wherein the convolutional neural network includes at least three channels, the third latent variable corresponds to the first channel among the at least three channels.

[0170] Wherein, the latent variable compensation parameter is a vector, wherein the dimension of the vector is less than the number of channels of the convolutional neural network, wherein the first vector element in the vector corresponds to at least two of the at least three channels, the at least two channels including the second channel and the first channel, the first vector element is used to indicate the quantization error of all elements in the third latent variable that satisfy the preset conditions, and the first vector element is also used to indicate the quantization error of all elements in the other latent variable corresponding to the second channel that satisfy the preset conditions.

[0171] In some possible implementations, when the first vector element corresponds to both the first and second channels, this correspondence is also written into the bitstream. Of course, if the encoding and decoding segments use the same default correspondence between elements and channels, the correspondence between the first vector element and the first and second channels may not need to be written into the bitstream.

[0172] As can be seen, in the technical solution provided in this application, the encoder can obtain latent variable compensation parameters based on the quantization error of the latent variables (i.e., the latent variable compensation parameters can indicate the relevant quantization error caused by the encoder quantizing the latent variables), and then encode the latent variable compensation parameters and write them into the bitstream. This provides a basis for the decoder to perform compensation processing on relevant latent variables based on the latent variable compensation parameters, which helps to reduce the distortion of the input data of the decoding neural network, thereby improving the quality of the final decoded signal. Specifically, for example, it helps to reduce holes in the frequency domain of the decoded audio signal and improve the subjective auditory quality of the decoded audio signal.

[0173] The following introduction is related to Figure 2 The example shown corresponds to the relevant decoding scheme.

[0174] See Figure 3 , Figure 3 This is a flowchart illustrating another audio data decoding method according to an embodiment of this application. The other audio data decoding method (corresponding to the decoding end) may include:

[0175] 301. Decode the latent variable compensation parameter and the second latent variable from the bitstream. The value of the current element in the second latent variable is the quantized value.

[0176] 302. The second latent variable is dequantized to obtain the third latent variable.

[0177] Alternatively, a specific decoder can be used to directly decode the third latent variable from the bitstream. In this case, it is not necessary to perform the steps of decoding the second latent variable from the bitstream and dequantizing the second latent variable to obtain the third latent variable.

[0178] 303. The third latent variable is compensated according to the latent variable compensation parameters to obtain the reconstructed first latent variable.

[0179] Among them, the latent variable compensation parameter can be used to indicate the quantization error of the latent variable quantization at the encoding end.

[0180] 304. The first latent variable of the reconstruction is processed using a decoding neural network to generate decoded audio data.

[0181] The decoding neural network is pre-trained, and this application does not limit the specific network structure and training method of the decoding neural network.

[0182] For example, compensating the third latent variable according to the latent variable compensation parameter to obtain the reconstructed first latent variable may include: determining the elements in the third latent variable that meet preset conditions according to the second latent variable or the third latent variable; and compensating the elements in the third latent variable that meet preset conditions according to the latent variable compensation parameter to obtain the reconstructed first latent variable.

[0183] Determining the elements in the third latent variable that satisfy the preset conditions based on the second or third latent variable may include: determining the elements in the second latent variable that satisfy the preset conditions; wherein the elements in the third latent variable that satisfy the preset conditions correspond in position to the elements in the second latent variable that satisfy the preset conditions. Alternatively, the elements in the third latent variable that satisfy the preset conditions may be determined directly based on the third latent variable.

[0184] In practical applications, there are many ways to compensate elements in the third latent variable that meet preset conditions according to the latent variable compensation parameters to obtain the reconstructed first latent variable. For example, compensating elements in the third latent variable that meet preset conditions according to the latent variable compensation parameters to obtain the reconstructed first latent variable may include:

[0185] Random noise is generated; the amplitude or energy of the generated random noise is adjusted according to the latent variable compensation parameters to obtain amplitude-adjusted random noise; the elements in the third latent variable that meet the preset conditions are compensated according to the amplitude-adjusted random noise to obtain the reconstructed first latent variable.

[0186] It is understandable that the above preset conditions can be varied, and the preset conditions that meet the needs can be set according to the specific scenario.

[0187] For example, a third latent variable or a second latent variable may include a first element, and satisfying a preset condition may include: the value of the first element is less than or equal to a preset value. When the first element is an element in the second latent variable, the value of the first element is a quantized value. Or, when the first element is an element in the third latent variable, the value of the first element is a dequantized value.

[0188] It is understandable that different structures of decoding neural networks may process latent variables in different forms.

[0189] In the embodiments of this application, the decoding neural network can specifically be a fully connected neural network, a convolutional neural network, or other neural networks. When the decoding neural network is a convolutional neural network, the number of channels of this convolutional neural network can be 1, 2, 3, 4, or more.

[0190] For example, when the decoding neural network is a convolutional neural network, the convolutional neural network includes only one channel; the latent variable compensation parameter is a scalar, wherein the scalar is used to compensate all elements in the third latent variable that satisfy the preset conditions.

[0191] For example, when the decoding neural network is a convolutional neural network, the convolutional neural network includes at least two channels, the third latent variable corresponds to the at least two channels, the at least two channels include the first channel, and the third latent variable is an m×n matrix.

[0192] The latent variable compensation parameter can be a scalar, which is used to compensate all elements in the third latent variable that meet the preset conditions.

[0193] Alternatively, the latent variable compensation parameter can be a vector, wherein the dimension of the vector is equal to the number of channels of the convolutional neural network, the vector elements in the vector correspond one-to-one with the at least two channels, the first vector element in the vector elements corresponds to the first channel, the first vector element is used to compensate all elements in the submatrix of the m×n matrix that satisfy the preset conditions, the first channel corresponds to the submatrix, wherein the number of elements in the submatrix is ​​less than m×n.

[0194] For example, when the decoding neural network is a convolutional neural network, the convolutional neural network includes at least three channels, and a third latent variable corresponds to the at least three channels. The third latent variable is an m×n matrix. The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels in the convolutional neural network. A first vector element in the vector corresponds to at least two of the at least three channels, including a second channel and a first channel. The first vector element is used to compensate all elements in a first submatrix of the m×n matrix that satisfy a preset condition. The first vector element is also used to compensate all elements in a second submatrix of the m×n matrix that satisfy a preset condition. The first submatrix corresponds to the first channel, and the second submatrix corresponds to the second channel. The number of elements in the first submatrix is ​​less than m×n, and the number of elements in the second submatrix is ​​less than m×n.

[0195] For example, when the decoding neural network is a convolutional neural network, wherein the convolutional neural network includes at least two channels, the third latent variable corresponds to the first channel of the at least two channels.

[0196] Wherein, the latent variable compensation parameter is a scalar, which is used to compensate all elements in the at least two latent variables corresponding to the at least two channels that satisfy preset conditions, wherein the at least two latent variables include the third latent variable. Alternatively, the latent variable compensation parameter is a vector, wherein the dimension of the vector is equal to the number of channels in the convolutional neural network, the vector elements in the vector correspond one-to-one with the at least two channels, and the vector elements in the vector corresponding to the first channel are used to compensate all elements in the third latent variable that satisfy preset conditions.

[0197] For example, when the decoding neural network is a convolutional neural network, the convolutional neural network includes at least three channels, and the third latent variable corresponds to the first channel among the at least three channels. The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels in the convolutional neural network. The first vector element in the vector corresponds to at least two channels among the at least three channels, the at least two channels including the second channel and the first channel. The first vector element is used to compensate for all elements in the third latent variable that satisfy a preset condition, and the first vector element is also used to compensate for all elements in another latent variable corresponding to the second channel that satisfy a preset condition.

[0198] As can be seen, the solution provided in this application's embodiments, because the encoding end writes latent variable compensation parameters into the bitstream, and these latent variable compensation parameters can indicate the relevant quantization errors caused by the encoding end quantizing the latent variables, allows the decoding end to compensate for the quantization errors caused by the encoding end quantizing the latent variables after acquiring the bitstream. This helps reduce the distortion of the input data of the decoding neural network, thereby improving the quality of the final decoded signal. For example, it helps reduce holes in the frequency domain of the decoded audio signal, improving the subjective auditory quality of the decoded audio signal.

[0199] The following examples illustrate this further.

[0200] See Figure 4A , Figure 4A This is a flowchart illustrating another audio encoding processing method provided in an embodiment of this application. This embodiment uses a convolutional neural network as an example for illustration. The other audio data encoding method (corresponding to the encoding end) may include:

[0201] 401. The audio encoder acquires the audio data to be encoded.

[0202] The audio data to be encoded can be either the audio signal to be encoded or the audio features to be encoded.

[0203] The audio signal to be encoded can be a time-domain audio signal; it can also be a frequency-domain signal obtained by time-frequency transformation of a time-domain signal, such as the frequency-domain signal obtained by MDCT transformation of a time-domain audio signal, or the frequency-domain signal obtained by FFT transformation of a time-domain audio signal; the signal to be encoded can also be a signal after QMF filtering; the signal to be encoded can also be a residual signal, such as the residual signal after other encoding or the residual signal after LPC filtering, etc. The audio features to be encoded can be features extracted based on the audio signal, such as Mel-frequency cepstral coefficients or latent variables obtained through neural networks.

[0204] 402. The audio encoder uses a convolutional neural network to process the audio data to be encoded to generate the first latent variable.

[0205] In this system, the first latent variable output by the convolutional neural network (CNN) is an N*M dimensional matrix, where N is the number of channels in the CNN network, and M is the latent size of each channel. (See [link to documentation]). Figure 4B For example, Figure 4B This paper presents a schematic diagram of the latent variables in a CNN network. The channel numbers can be counted starting from 1 or 0, and the element numbers of the latent variables within each channel are also the same.

[0206] 403. The audio encoder quantizes the first latent variable to obtain the second latent variable.

[0207] It is understandable that the values ​​of the elements in the first latent variable are unquantized, while the values ​​of the elements in the second latent variable are quantized.

[0208] For example, each element of the first latent variable can be scalar quantized. The quantization step size can be determined according to different coding rates. Scalar quantization can also have a bias, for example, the latent variable to be quantized is biased before scalar quantization according to a determined quantization step size. Other quantization techniques can also be used to quantize latent variables, and this application does not limit the scope of the quantization method.

[0209] 404. The audio encoder obtains the latent variable compensation parameters based on the first latent variable and the second latent variable.

[0210] The specific methods for obtaining latent variable compensation parameters may differ depending on the form of the latent variables. The basic principle is to determine the value of the latent variable compensation parameter based on the quantization error of the elements in the latent variables that satisfy preset conditions. The latent variable compensation parameter indicates the quantization error of the elements in the latent variables that satisfy preset conditions.

[0211] One possible implementation is to use the average quantization error of all elements that meet the conditions as the value of the latent variable compensation parameter. Another possible implementation is to use the weighted average quantization error of all elements that meet the conditions as the value of the latent variable compensation parameter, where the weighting value can be related to the element's index. Elements that meet the preset conditions can be elements in the second latent variable that also meet the preset conditions. Elements in the second latent variable that meet the preset conditions can be elements with the minimum quantized value, elements with the midpoint of the quantization range, or elements with values ​​less than or equal to a preset threshold. When the quantization range includes both positive and negative numbers, whether the quantized value is the midpoint of the quantization range can be chosen as the condition. For example, if the quantized value is an integer between -10 and 10, and the midpoint of the quantization range is 0, then the quantization error of elements with a quantization value of 0 can be used to calculate the average quantization error. Elements that meet the preset conditions can also be elements in the third latent variable that also meet the preset conditions. The third latent variable is obtained by dequantizing the second latent variable. Similarly, the elements in the third latent variable that satisfy the preset conditions can be elements whose element value is equal to the minimum value of the dequantized value, elements whose element value is equal to the preset value, or elements whose element value is less than or equal to a preset threshold.

[0212] The quantization error mentioned in the various embodiments of this application may be the amplitude difference or energy difference or its absolute value between the original value to be quantized and the quantized value, or it may be the amplitude difference or energy difference or its absolute value between the original value to be quantized and the dequantized value.

[0213] When the encoding neural network is a CNN network, one method for extracting latent variable compensation parameters is as follows: Calculate the quantization error of the elements satisfying the conditions in the latent variables of all channels; determine the average quantization error based on the quantization errors of the elements satisfying the conditions in the latent variables of all channels; and use this average quantization error as the value of the latent variable compensation parameter. When the encoding neural network is a CNN network, the first latent variable output by the CNN network is an N*M dimensional matrix, denoted as L(n,k), where n = 0, 1, ..., N-1, k = 0, 1, ..., M-1. N is the number of channels in the CNN network, and M is the latent size, i.e., the dimension of the latent variable in each channel. The third latent variable is denoted as... The latent variable compensation parameter NF is a scalar. A possible pseudocode for calculating the latent variable compensation parameter NF is shown below. Here, eps1 represents a preset minimum value, and TH1 is a preset value.

[0214]

[0215]

[0216] When the encoding neural network is a CNN network, another method for extracting latent variable compensation parameters is as follows: In each channel, determine the quantization error of the elements in the latent variable that satisfy the conditions, and determine the average quantization error based on the quantization errors of the elements that satisfy the conditions. This average quantization error is then used as the value of the latent variable compensation parameter corresponding to that channel. For example, assuming the encoding neural network is a CNN network, the first latent variable output by the encoding neural network is an N*M dimensional matrix, denoted as L(n,k), where n = 0, 1, ..., N-1, k = 0, 1, ..., M-1. N is the number of channels in the CNN network, and M is the latent size. eps1 is a preset minimum value, and TH1 is a preset value. The third latent variable is denoted as... The latent variable compensation parameter NF(n) is n = 0, 1, ..., N-1. NF(n) is a vector whose dimension is equal to the number of channels in the CNN network. A possible pseudocode for calculating the latent variable compensation parameter NF is shown in the following example.

[0217]

[0218]

[0219] When the encoding neural network is a CNN network, another method for extracting latent variable compensation parameters is as follows: When the latent variable compensation parameters are vectors, and the dimension of the vector is less than the number of channels in the convolutional neural network, the first vector element of the latent variable compensation parameters can correspond to at least two channels of the latent variable. The elements in the channels corresponding to the first vector element of the latent variable that satisfy the conditions are determined, and the average quantization error is determined as the value of the first vector element of the latent variable compensation parameters based on the quantization error of the elements that satisfy the conditions.

[0220] For example, assuming the encoding neural network is a CNN, the first latent variable output by the encoding neural network is an N*M dimensional matrix, denoted as L(n,k), where n = 0, 1, ..., N-1, k = 0, 1, ..., M-1. N is the number of channels in the CNN network, and M is the latent size. The latent variable compensation parameter NF(j), j = 0, 1, ..., P-1, is a vector, and the dimension P of the vector is less than the number of channels N in the CNN network. The j-th vector element NF(j) of the latent variable compensation parameter corresponds to the channels in the CNN network from ADD(j) to ADD(j+1)-1. ADD represents the correspondence between the latent variable compensation parameter vector elements and the CNN network channels, which can be a pre-stored table. ADD(j) is the minimum channel index corresponding to the j-th vector element of the latent variable compensation parameter. A possible pseudocode for calculating the first vector element NF(j) of the latent variable compensation parameter can be illustrated as follows:

[0221]

[0222] It should be noted that when the encoded neural network is a CNN network, the calculation of error, ERR, and NF can be performed in the energy domain or the amplitude domain.

[0223] 405. Encode the latent variable compensation parameters and write the encoding results of the latent variable compensation parameters into the bitstream.

[0224] For example, encoding the latent variable compensation parameters and the encoding result of the latent variable compensation parameters may include: performing scalar quantization or vector quantization on the latent variable compensation parameters to obtain the quantization index of the latent variable compensation parameters, encoding the quantization index of the latent variable compensation parameters, and writing the encoding result of the quantization index of the latent variable compensation parameters into the bitstream.

[0225] Alternatively, the latent variable compensation parameters can be quantized, and then entropy encoded to be written into the bitstream.

[0226] 406. Encode the second latent variable and write the encoding result of the second latent variable into the bitstream.

[0227] For example, encoding the second latent variable may include: using an adjustable entropy encoding model to perform arithmetic encoding on the second latent variable, or using an entropy encoding model with a preset probability distribution to perform arithmetic encoding on the second latent variable.

[0228] It is understandable that there is no necessary order between steps 406 and steps 404-405. The execution of step 405 may precede steps 404-405, or may be later than steps 404-405, or may be concurrent with steps 404-405.

[0229] As can be seen, the solution provided in this application, at the encoding end, obtains latent variable compensation parameters based on the relevant quantization errors of the latent variables, and encodes these parameters before writing them into the bitstream. This provides a basis for the decoding end to compensate for the quantization errors caused by the encoding end's quantization of the latent variables based on the latent variable compensation parameters, thereby helping to reduce the distortion of the input data of the convolutional neural network at the decoding end, and thus improving the quality of the final decoded signal. Specifically, for example, it helps to reduce holes in the frequency domain of the decoded audio signal and improve the subjective auditory quality of the decoded audio signal.

[0230] The following introduction and Figure 4A The corresponding decoding schemes for the encoding scheme described.

[0231] See Figure 5 , Figure 5 This is a flowchart illustrating another audio data decoding method provided in an embodiment of this application. This embodiment uses a convolutional neural network as an example for the decoding neural network. The other audio data decoding method (corresponding to the decoding end) may include:

[0232] 501. The audio decoder decodes the latent variable compensation parameter and the second latent variable (the values ​​of the elements in the second latent variable are quantized values) from the received bitstream.

[0233] The process of decoding the latent variable compensation parameters is the inverse process of quantization encoding of the latent variable compensation parameters. For example, the quantization index of the latent variable compensation parameters can be parsed from the bitstream, and the latent variable compensation parameters can be obtained by using the dequantization method corresponding to the encoder based on the quantization index.

[0234] The process of decoding the second latent variable is the reverse process of encoding the second latent variable at the encoding end. For example, entropy decoding is performed on the bitstream to obtain the second latent variable.

[0235] 502. The audio decoder dequantizes the second latent variable to obtain the third latent variable.

[0236] The process by which the audio decoder obtains the dequantized latent variables is the reverse process of quantizing the latent variables at the encoder. For example, dequantizing the scalar-quantized latent variables yields the dequantized latent variables.

[0237] 503. The audio decoder performs compensation processing on the third latent variable according to the latent variable compensation parameters to obtain the reconstructed first latent variable.

[0238] For example, compensating the third latent variable according to the latent variable compensation parameter to obtain the reconstructed first latent variable may include: determining the elements in the third latent variable that meet preset conditions according to the second latent variable or the third latent variable; and compensating the elements in the third latent variable that meet preset conditions according to the latent variable compensation parameter to obtain the reconstructed first latent variable.

[0239] In practical applications, there are many ways to compensate elements in the third latent variable that meet preset conditions according to the latent variable compensation parameters to obtain the reconstructed first latent variable. For example, compensating elements in the third latent variable that meet preset conditions according to the latent variable compensation parameters to obtain the reconstructed first latent variable may include:

[0240] Random noise is generated; the amplitude or energy of the generated random noise is adjusted according to the latent variable compensation parameters to obtain amplitude-adjusted random noise; the elements in the third latent variable that meet the preset conditions are compensated according to the amplitude-adjusted random noise to obtain the reconstructed first latent variable.

[0241] For example, assuming the decoding neural network is a CNN network, the third latent variable is an N*M dimensional matrix, denoted as, for instance. N represents the number of channels in the CNN network, and M represents the size of the latent variables, i.e., the dimension of the latent variables in each channel. The latent variable compensation parameter is a scalar, denoted as NF. One possible pseudocode for compensating the third latent variable based on the latent variable compensation parameter to obtain the reconstructed first latent variable is shown below. Here, FAC is a parameter controlling the degree of compensation, typically a preset constant, and TH1 is the minimum dequantization value.

[0242]

[0243] In the pseudocode above, the NF parameter is represented in terms of amplitude. If the NF parameter were represented in terms of energy, when adjusting the amplitude of the noise, the NF parameter could be converted to the amplitude domain before adjusting the noise amplitude.

[0244] If the latent variable compensation parameter is a vector of dimension N, where N is the number of channels in the CNN network, then the latent variable compensation parameter at the decoding end is NF(n). Taking NF(n) as a parameter represented by magnitude, the following is a possible pseudocode for compensating the third latent variable according to the latent variable compensation parameter to obtain the reconstructed first latent variable.

[0245]

[0246]

[0247] Among them, FAC is a parameter that controls the degree of compensation, which is usually a preset constant, and TH1 is the minimum dequantization value.

[0248] If the latent variable compensation parameters are vectors of dimension P, where the dimension P is less than the number of channels N in the CNN network, and the latent variable compensation parameters at the decoder are NF(j), j = 0, 1, ..., P-1, a possible pseudocode for compensating the third latent variable based on these parameters to obtain the reconstructed first latent variable is shown below:

[0249]

[0250] Here, NF(j) corresponds to the amplitude domain.

[0251] Of course, other similar compensation methods can be used, but they will not be listed here.

[0252] The compensated latent variable, the third latent variable, can be directly used as the first latent variable for reconstruction, or the compensated latent variable can be post-processed and then used as the first latent variable for reconstruction.

[0253] 504. The audio decoder uses a decoding neural network to process the first latent variable of the reconstruction to generate decoded audio data.

[0254] It is understood that the decoding neural network and the encoding neural network at the encoding end are paired and both are pre-trained. Based on the reconstructed first latent variable, the decoding neural network is used for processing. The output of the decoding neural network can be directly used as the decoded audio signal output. The output of the decoding neural network can be the recovered time-domain audio signal or a frequency-domain signal. If it is a frequency-domain signal, it needs to undergo a frequency-to-time domain transformation to obtain the time-domain audio signal. The output of the decoding neural network can also be an audio residual signal. If it is an audio residual signal, it needs to undergo other corresponding processing after the decoding neural network processing to obtain the decoded audio signal. This application does not limit this.

[0255] As can be seen, the solution provided in this application, by obtaining latent variable compensation parameters based on the quantization error of the latent variables at the encoding end and encoding these parameters before writing them into the bitstream, makes it possible for the decoding end to compensate for the quantization errors caused by the quantization of latent variables at the encoding end based on the latent variable compensation parameters after acquiring the bitstream. This helps reduce the distortion of the input data of the convolutional neural network at the decoding end, thereby improving the quality of the final decoded signal. Specifically, for example, it helps reduce holes in the frequency domain of the decoded audio signal, improving the subjective auditory quality of the decoded audio signal.

[0256] See Figure 6A , Figure 6A This is a flowchart illustrating another audio data encoding method provided in an embodiment of this application. This embodiment uses a fully connected encoding neural network as an example. The other audio data encoding method (corresponding to the encoding end) may include:

[0257] 601. The audio encoder acquires the audio data to be encoded.

[0258] 602. The audio encoder uses a fully connected network to process the audio data to be encoded to generate the first latent variable.

[0259] In this case, the first latent variable output by the fully connected network is an M-dimensional vector, where the dimension M is the size of the first latent variable. (See [link to documentation]). Figure 6B For example, Figure 6B A possible illustration of the latent variables of a fully connected network is given.

[0260] 603. The audio encoder quantizes the first latent variable to obtain the second latent variable (the values ​​of the elements in the second latent variable are quantized values).

[0261] For example, each element of the latent variable can be scalar quantized first. The quantization step size can be determined according to different encoding rates. Scalar quantization can also have a bias, for example, the latent variable to be quantized is biased before scalar quantization according to a determined quantization step size. The quantization method for latent variables can also be implemented using other existing quantization techniques, which are not limited in this application.

[0262] 604. The audio encoder obtains the latent variable compensation parameters based on the first latent variable and the second latent variable.

[0263] When the encoding neural network is a fully connected network, the latent variable output by the fully connected network, i.e., the first latent variable, is a one-dimensional vector, which can be denoted as L(k), k = 0, 1, ..., M-1. M is the magnitude of the latent variable.

[0264] Among them, the dequantized latent variable obtained after dequantizing the second latent variable, namely the third latent variable, is denoted as... Specific methods for calculating the latent variable compensation parameter NF may include: if If the value equals the minimum dequantization value (e.g., 0), then calculate the quantization error at the current element position. Will satisfy The quantization errors of all elements equal to the minimum quantization value are averaged, and this average error is used as a latent variable compensation parameter.

[0265] The following is an example of pseudocode for calculating the latent variable compensation parameter NF. Here, eps1 is a preset minimum value, and TH1 is the minimum dequantization value.

[0266]

[0267]

[0268] For example, when the encoding neural network is a fully connected network, in the method for extracting latent variable compensation parameters, the calculation of the quantization error ERR can be based not only on the absolute value of the difference between the unquantized and dequantized values, but also on the energy of the difference between the unquantized and dequantized values. Another pseudocode for calculating the latent variable compensation parameter NF is shown below. Here, eps1 can be a preset minimum value, and TH1 is the minimum dequantization value.

[0269]

[0270] Additionally, when ERR is calculated in terms of energy dimension, such as based on... The sum or The sum of the latent variable compensation parameters NF can be the average error energy, such as NF = ERR / NUM, or the magnitude of the average error, for example, NF parameters can also be calculated in the logarithmic field.

[0271] 605. The audio encoder encodes the latent variable compensation parameters and writes the encoding result of the latent variable compensation parameters into the bitstream.

[0272] For example, encoding the latent variable compensation parameters and the encoding result of the latent variable compensation parameters may include: performing scalar quantization or vector quantization on the latent variable compensation parameters to obtain the quantization index of the latent variable compensation parameters, encoding the quantization index of the latent variable compensation parameters, and writing the encoding result of the quantization index of the latent variable compensation parameters into the bitstream.

[0273] Alternatively, the latent variable compensation parameters can be quantized, and then entropy encoded to be written into the bitstream.

[0274] 606. The audio encoder encodes the second latent variable and writes the encoding result of the second latent variable into the bitstream.

[0275] For example, encoding the second latent variable may include: using an adjustable entropy encoding model to perform arithmetic encoding on the second latent variable, or using an entropy encoding model with a preset probability distribution to perform arithmetic encoding on the second latent variable.

[0276] It is understandable that there is no necessary order between steps 606 and steps 604-605. Step 606 may be executed before steps 604-605, or after steps 604-605, or in parallel with steps 604-605.

[0277] As can be seen, the solution provided in this application, at the encoding end, obtains latent variable compensation parameters based on the relevant quantization errors of the latent variables, and then encodes and writes these latent variable compensation parameters into the bitstream. This makes it possible for the decoding end to compensate for the relevant quantization errors caused by the encoding end's quantization of the latent variables based on the latent variable compensation parameters. This, in turn, helps reduce the distortion of the input data of the fully connected network at the decoding end, thereby improving the quality of the final decoded signal. Specifically, for example, it helps reduce holes in the frequency domain of the decoded audio signal, improving the subjective auditory quality of the decoded audio signal.

[0278] The following introduction is related to Figure 6A The corresponding decoding schemes for the encoding scheme described.

[0279] See Figure 7 , Figure 7 This is a flowchart illustrating another audio data decoding method provided in an embodiment of this application. This embodiment uses a fully connected network as an example of a decoding neural network. The other audio data decoding method (corresponding to the decoding end) may include:

[0280] 701. The audio decoder decodes the latent variable compensation parameter and the second latent variable (the values ​​of the elements in the second latent variable are quantized values) from the received bitstream.

[0281] The process by which the audio decoder obtains the dequantized latent variables is the reverse process of quantizing the latent variables at the encoder. For example, dequantizing the quantized latent variables yields the dequantized latent variables.

[0282] 702. The audio decoder dequantizes the second latent variable to obtain the third latent variable.

[0283] The process by which the audio decoder obtains the dequantized latent variables is the reverse process of quantization and encoding of the latent variables at the encoding end. For example, arithmetic decoding is performed on the bitstream to obtain the quantized latent variables. Then, the quantized latent variables are dequantized to obtain the dequantized latent variables.

[0284] 703. The audio decoder performs compensation processing on the third latent variable according to the latent variable compensation parameters to obtain the reconstructed first latent variable.

[0285] For example, compensating the third latent variable according to the latent variable compensation parameter to obtain the reconstructed first latent variable may include: determining the elements in the third latent variable that meet preset conditions according to the second latent variable or the third latent variable; and compensating the elements in the third latent variable that meet preset conditions according to the latent variable compensation parameter to obtain the reconstructed first latent variable.

[0286] In practical applications, there are many ways to compensate elements in the third latent variable that meet preset conditions according to the latent variable compensation parameters to obtain the reconstructed first latent variable. For example, compensating elements in the third latent variable that meet preset conditions according to the latent variable compensation parameters to obtain the reconstructed first latent variable may include:

[0287] Random noise is generated; the amplitude or energy of the generated random noise is adjusted according to the latent variable compensation parameters to obtain amplitude-adjusted random noise; the elements in the third latent variable that meet the preset conditions are compensated according to the amplitude-adjusted random noise to obtain the reconstructed first latent variable.

[0288] For example, assuming the decoding neural network is a fully connected network, the third latent variable is an M-dimensional vector, for instance, denoted as... M represents the size of the latent variable. The latent variable compensation parameter is a scalar, denoted as NF. Taking NF as a parameter characterized by amplitude as an example, a possible pseudocode for obtaining the reconstructed first latent variable by compensating the third latent variable according to the latent variable compensation parameter is as follows:

[0289]

[0290] Among them, FAC is a parameter that controls the degree of compensation, which is usually a preset constant, and TH1 is the minimum dequantization value.

[0291] 704. The audio decoder uses a fully connected network to process the first latent variable of the reconstruction to generate decoded audio data.

[0292] As can be seen, the solution provided in this application allows the encoder to obtain latent variable compensation parameters based on the quantization error of the latent variables, and then encode and write these parameters into the bitstream. After the decoder obtains the bitstream, it becomes possible to compensate for the quantization errors caused by the encoder's quantization of the latent variables based on the latent variable compensation parameters. This helps reduce the distortion of the input data of the fully connected network at the decoder, thereby improving the quality of the final decoded signal. Specifically, for example, it helps reduce holes in the frequency domain of the decoded audio signal, improving the subjective auditory quality of the decoded audio signal.

[0293] The following is an example of applying the scheme of this application in an AI encoder that includes a context model.

[0294] See Figure 8A and Figure 8B , Figure 8A and Figure 8BThis is a flowchart illustrating another audio data encoding method provided in an embodiment of this application. An audio data encoding method (corresponding to the encoding end) may include:

[0295] 801. The audio encoder performs windowing processing on the audio signal to obtain the audio signal of the current frame.

[0296] 802. The audio encoder performs MDCT transformation on the audio signal of the current frame to obtain the frequency domain signal of the current frame.

[0297] 803. The audio encoder uses a coding neural network to process the frequency domain signal of the current frame to output the first latent variable.

[0298] One possible implementation is to directly use the frequency domain signal of the current frame as the input to the encoding neural network. Another optional implementation is to preprocess the frequency domain signal of the current frame and use the preprocessed signal as the input to the encoding neural network. Preprocessing may include processes such as FDNS and TNS.

[0299] 804. The audio encoder uses a context model to process the first latent variable to obtain the context bitstream and determines the entropy coding model parameter σ.

[0300] The first latent variable is processed by a context encoding neural network to obtain the latent variables of the context model. The latent scalar of the context model represents the probability distribution of the latent variables of the encoding network.

[0301] Latent variables can be directly used as input to a context encoding neural network, or the absolute value of each element in the latent variables can be taken first and then used as input to the context encoding neural network.

[0302] The latent variables of the context model are quantized, and the quantized latent variables of the context model are arithmetically encoded to obtain the context bitstream, which is then written into the bitstream.

[0303] Arithmetic decoding is performed on the context bitstream, followed by dequantization, to obtain the latent variables of the dequantized context model.

[0304] The dequantized latent variables of the context model are processed by a context decoding neural network to obtain the entropy-encoded model parameters σ.

[0305] 805. Quantify the first latent variable to obtain the second latent variable.

[0306] 806. Obtain the latent variable compensation parameters based on the first latent variable and the second latent variable, and write the latent variable compensation parameters into the bitstream after quantization encoding.

[0307] The latent variable compensation parameters can be obtained based on the first and second latent variables. Alternatively, the second latent variable can be dequantized to obtain the third latent variable, and the latent variable compensation parameters can be obtained based on the third latent variable and the first latent variable.

[0308] Based on the latent variables before and after quantization, the specific method for extracting latent variable compensation parameters can be any of the specific methods for extracting latent variable compensation parameters exemplified in the above embodiments.

[0309] In this embodiment, when the encoding neural network is implemented using a CNN network, the specific method for extracting the latent variable compensation parameters can be as follows: calculate the average quantization error of the elements in the latent variable of each channel that satisfy the preset conditions, and then take the average of the average quantization errors of all channels as the NF parameters. Assume that the latent variable output by the encoding neural network is an N*M dimensional matrix, denoted as L(n,k), where n = 0, 1, ..., N-1, k = 0, 1, ..., M-1. N is the number of CNN network channels, and M is the size of the latent variable. The dequantized latent variable is denoted as... One example of pseudocode for calculating the latent variable compensation parameter NF is shown below. For example, TH1 = 0.

[0310]

[0311]

[0312] For example, the entropy coding model corresponding to the parameter σ can be determined from the entropy coding model with adjustable parameters, and the quantized latent variable can be arithmetic encoded according to the entropy coding model corresponding to the parameter σ, and the arithmetic encoding result can be written into the bitstream.

[0313] As can be seen, the solution provided in this application, in the scenario of an AI encoder including a context model, makes it possible for the decoding end to compensate for the quantization errors caused by the encoding end's quantization of latent variables based on the quantization error of the latent variables at the encoding end, after the latent variable compensation parameters are obtained and encoded into the bitstream. This makes it possible for the decoding end to compensate for the quantization errors caused by the quantization of latent variables at the encoding end based on the latent variable compensation parameters after obtaining the bitstream. This helps to reduce the distortion of the input data of the decoding neural network, thereby improving the quality of the final decoded signal. Specifically, for example, it helps to reduce holes in the frequency domain of the decoded audio signal and improve the subjective auditory quality of the decoded audio signal.

[0314] The following introduction is related to Figure 8A The corresponding decoding schemes for the encoding scheme described.

[0315] See Figure 9A and Figure 9B , Figure 9Aand Figure 9B This is a flowchart illustrating another audio data decoding method provided in an embodiment of this application. The other audio data decoding method (corresponding to the decoding end) may include:

[0316] 901. The audio decoder obtains the entropy coding model parameters σ by decoding the received bitstream from the decoding side of the context model.

[0317] For example, arithmetic decoding and dequantization can be performed on the received bitstream to obtain the latent variables of the dequantized context model. The latent variables of the dequantized context model are then processed by a context decoding neural network to obtain the entropy coding model parameters σ.

[0318] 902. Based on the entropy coding model parameter σ, determine the entropy coding model corresponding to parameter σ from the parameter-adjustable entropy coding models, and perform arithmetic decoding on the latent variable coding information in the bitstream to obtain the quantized latent variables.

[0319] 903. Dequantize the quantized latent variables to obtain the dequantized latent variables.

[0320] 904. Decode the latent variable compensation parameters according to the bitstream, and compensate the dequantized latent variables according to the latent variable compensation parameters to obtain the compensated latent variables.

[0321] The specific method for compensating the dequantized latent variables based on the latent variable compensation parameters can be implemented using any of the specific compensation methods described in the above embodiments at the decoding end. In this embodiment, since the latent variable compensation parameters transmitted at the encoding end are scalar NF, the decoding end compensates for the latent variables of each channel based on the latent variable compensation parameters NF.

[0322] 905. The compensated latent variables are used as input, processed by a decoding neural network, and the decoded frequency domain signal is output.

[0323] 906. Post-process the decoded frequency domain signal to obtain the enhanced frequency domain signal.

[0324] 907 performs IMDCT transform on the enhanced frequency domain signal and then performs dewindowing processing to obtain the final decoded audio signal.

[0325] As can be seen, the solution provided in this application, in the scenario of an AI encoder including a context model, makes it possible for the decoding end to compensate for the quantization errors caused by the encoding end's quantization of latent variables based on the quantization error of the latent variables at the encoding end, after the latent variable compensation parameters are obtained and encoded into the bitstream. This makes it possible for the decoding end to compensate for the quantization errors caused by the quantization of latent variables at the encoding end based on the latent variable compensation parameters after obtaining the bitstream. This helps to reduce the distortion of the input data of the decoding neural network, thereby improving the quality of the final decoded signal. Specifically, for example, it helps to reduce holes in the frequency domain of the decoded audio signal and improve the subjective auditory quality of the decoded audio signal.

[0326] The following is an example of applying the scheme of this application in an AI encoder that does not contain a context model.

[0327] See Figure 10A and Figure 10B , Figure 10A and Figure 10B This is a flowchart illustrating another audio data encoding method provided in an embodiment of this application. The other audio data encoding method (corresponding encoding end) may include:

[0328] 1001. The audio encoder performs windowing processing on the audio signal to obtain the audio signal of the current frame.

[0329] 1002. The audio encoder performs MDCT transformation on the audio signal of the current frame to obtain the frequency domain signal of the current frame.

[0330] 1003. The audio encoder takes the frequency domain signal of the current frame as input, processes it through the encoding neural network, and outputs the first latent variable.

[0331] 1004. The audio encoder quantizes the first latent variable to obtain the second latent variable.

[0332] 1005. The audio encoder extracts latent variable compensation parameters based on the first and second latent variables.

[0333] Extracting latent variable compensation parameters based on the first and second latent variables can also be achieved by dequantizing the second latent variable to obtain a third latent variable, and then extracting the latent variable compensation parameters based on the first and third latent variables. Alternatively, the latent variable compensation parameters can be extracted directly from the first and second latent variables.

[0334] 1006. The audio encoder quantizes and encodes the latent variable compensation parameters and writes them into the bitstream.

[0335] 1007. The audio encoder performs arithmetic encoding on the quantized latent variables according to the entropy encoding model, and writes the arithmetic encoding result into the bitstream.

[0336] As can be seen, the solution provided in this application, in AI encoder scenarios without a context model, allows for the compensation of latent variable errors caused by quantization at the encoding end. This is achieved by obtaining latent variable compensation parameters based on the quantization error of the latent variables and encoding these parameters before writing them into the bitstream. After obtaining the bitstream, the decoding end can then use these latent variable compensation parameters to compensate for the quantization errors caused by the encoding end's quantization of the latent variables. This reduces distortion in the input data of the decoding neural network, thereby improving the quality of the final decoded signal. Specifically, for example, it helps reduce holes in the frequency domain of the decoded audio signal, improving the subjective auditory quality of the decoded audio signal.

[0337] The following introduction is related to Figure 10A The corresponding decoding schemes for the encoding scheme described.

[0338] See Figure 11A and Figure 11B , Figure 11A and Figure 11B This is a flowchart illustrating an audio data decoding method provided in an embodiment of this application. Another audio data decoding method (corresponding to the decoding end) may include:

[0339] 1101. The audio decoder performs arithmetic decoding on the bitstream according to the entropy coding model to obtain the second latent variable.

[0340] 1102. The audio decoder dequantizes the quantized latent variables to obtain the third latent variable.

[0341] 1103. The audio decoder decodes the latent variable compensation parameters according to the bitstream, and compensates the third latent variable according to the latent variable compensation parameters to obtain the compensated latent variable.

[0342] 1104. The audio decoder takes the compensated latent variables as input, processes them through a decoding neural network, and outputs the decoded frequency domain signal.

[0343] 1105. The audio decoder performs post-processing on the decoded frequency domain signal to obtain the enhanced frequency domain signal.

[0344] 1106. The audio decoder performs IMDCT transformation on the enhanced frequency domain signal and performs dewindowing processing to obtain the final decoded audio signal.

[0345] Tests have shown that the method proposed in this embodiment can reduce the distortion of the input data of the decoding neural network, thereby improving the quality of the final decoded signal.

[0346] As can be seen, the solution provided in this application, because the encoder obtains latent variable compensation parameters based on the quantization error of the latent variables, and encodes these parameters before writing them into the bitstream, makes it possible for the decoder to compensate for the quantization errors caused by the encoder's quantization of the latent variables based on the latent variable compensation parameters after obtaining the bitstream. This helps reduce the distortion of the input data of the fully connected network at the decoder, thereby improving the quality of the final decoded signal. Specifically, for example, it helps reduce holes in the frequency domain of the decoded audio signal, improving the subjective auditory quality of the decoded audio signal.

[0347] The following is an example of a method that incorporates a context model, where compensation also occurs within the context model. In other words, it can be used not only to compensate for latent variables in the encoding network but also for latent variables in the context model.

[0348] See Figure 12A and Figure 12B , Figure 12A and Figure 12B This is a flowchart illustrating another audio data encoding method provided in an embodiment of this application. The other audio data encoding method (corresponding encoding end) may include:

[0349] 1201. The audio signal is windowed to obtain the audio signal of the current frame.

[0350] 1202. After performing MDCT transformation on the audio signal of the current frame, the frequency domain signal of the current frame is obtained.

[0351] 1203. The frequency domain signal of the current frame is taken as input, processed by an encoded neural network, and the latent variables are output.

[0352] 1204. Based on the latent variables, the context code stream is obtained through context model processing, and the entropy coding model parameter σ is determined.

[0353] The process of obtaining the context bitstream based on latent variables and through context model processing, and determining the entropy coding model parameters σ, may include the following steps 1204a-1204f:

[0354] 1204a. The latent variables are processed by the context encoding neural network to obtain the latent variables of the context model. The latent variables of the context model represent the probability distribution of the latent variables of the encoding network.

[0355] 1204b. Quantize the latent variables of the context model, and perform arithmetic encoding on the quantized latent variables of the context model to obtain the context bitstream, and write it into the bitstream.

[0356] By dequantizing the latent variables of the quantized context model, we can obtain the latent variables of the dequantized context model. Alternatively, we can extract the latent variable compensation parameters of the context model based on both the unquantized and dequantized latent variables.

[0357] 1204c. Based on the latent variables of the context model before and after quantization, extract the latent variable compensation parameters of the context model, quantize and encode the latent variable compensation parameters of the context model, and write them into the bitstream.

[0358] 1204d. Perform arithmetic decoding based on the context bitstream and dequantize it to obtain the latent variables of the dequantized context model.

[0359] 1204e. Decode the latent variable compensation parameters of the context model. Based on the decoded latent variable compensation parameters of the context model, compensate the latent variables of the dequantized context model to obtain the latent variables of the compensated context model.

[0360] 1204f. The latent variables of the compensated context model are processed by the context decoding neural network to obtain the entropy-encoded model parameters σ.

[0361] 1205. Quantify the latent variables to obtain the quantified latent variables.

[0362] 1206. Based on the latent variables before and after quantization, extract the latent variable compensation parameters, quantize and encode the latent variable compensation parameters, and write them into the bitstream.

[0363] Dequantizing the quantized latent variables yields the dequantized latent variables. Alternatively, latent variable compensation parameters can be extracted based on both the unquantized and dequantized latent variables.

[0364] 1207. Based on the entropy coding model, perform arithmetic coding on the quantized latent variables and write the arithmetic coding result into the bitstream.

[0365] Based on the entropy coding model parameter σ, determine the entropy coding model corresponding to parameter σ from the parameter-adjustable entropy coding models, and perform arithmetic coding on the quantized latent variables according to the entropy coding model corresponding to parameter σ, and write the arithmetic coding result into the bitstream.

[0366] As can be seen, the solution provided in this application, in application scenarios with a context model and where compensation is also within the context model, allows for the compensation of latent variables at the encoding end. This is because the latent variable compensation parameters are obtained based on the quantization error of the latent variables, encoded, and written into the bitstream. After the decoding end obtains the bitstream, it becomes possible to compensate for the quantization errors caused by the quantization of latent variables at the encoding end based on the latent variable compensation parameters. This helps reduce the distortion of the input data to the decoding neural network, thereby improving the quality of the final decoded signal. Specifically, for example, it helps reduce holes in the frequency domain of the decoded audio signal, improving the subjective auditory quality of the decoded audio signal.

[0367] The following introduction is related to Figure 12A The corresponding decoding schemes for the encoding scheme described.

[0368] See Figure 13A and Figure 13B , Figure 13A and Figure 13B This is a flowchart illustrating another audio data decoding method provided in an embodiment of this application. The other audio data decoding method (corresponding to the decoding end) may include:

[0369] 1301. Based on the received bitstream, the entropy coding model parameters σ are obtained by decoding on the decoding side of the context model.

[0370] in,

[0371] Based on the received bitstream, the entropy coding model parameters σ are obtained by decoding on the context model's decoding side. This process includes: performing arithmetic decoding on the received bitstream and dequantizing it to obtain the dequantized latent variables of the context model; decoding the latent variable compensation parameters of the context model based on the bitstream; and compensating the dequantized latent variables of the context model based on these compensation parameters to obtain the compensated latent variables of the context model. The compensated latent variables of the context model are then processed by the context decoding neural network to obtain the entropy coding model parameters σ.

[0372] 1302. Based on the entropy coding model parameter σ, determine the entropy coding model corresponding to parameter σ from the parameter-adjustable entropy coding models, and perform arithmetic decoding on the latent variable coding information in the bitstream to obtain the quantized latent variables.

[0373] 1303. Dequantize the quantized latent variables to obtain the dequantized latent variables.

[0374] 1304. Decode the latent variable compensation parameters according to the bitstream, and compensate the dequantized latent variables according to the latent variable compensation parameters to obtain the compensated latent variables.

[0375] 1305. The compensated latent variables are used as input, processed by a decoding neural network, and the decoded frequency domain signal is output.

[0376] 1306. Post-process the decoded frequency domain signal to obtain the enhanced frequency domain signal.

[0377] 1307. Perform IMDCT transform on the enhanced frequency domain signal and then perform dewindowing processing to obtain the final decoded audio signal.

[0378] As can be seen, the solution provided in this application, in application scenarios with a context model and where compensation is also within the context model, allows for the compensation of latent variables at the encoding end. This is because the latent variable compensation parameters are obtained based on the quantization error of the latent variables, encoded, and written into the bitstream. After the decoding end obtains the bitstream, it becomes possible to compensate for the quantization errors caused by the quantization of latent variables at the encoding end based on the latent variable compensation parameters. This helps reduce the distortion of the input data to the decoding neural network, thereby improving the quality of the final decoded signal. Specifically, for example, it helps reduce holes in the frequency domain of the decoded audio signal, improving the subjective auditory quality of the decoded audio signal.

[0379] In summary, this application proposes a latent variable feature compensation mechanism. The encoder extracts latent variable compensation parameters based on the quantization error of the latent variables, encodes these parameters, and transmits them to the decoder. The decoder then compensates for the latent variables based on these parameters, which helps reduce distortion of the input data to the decoding neural network, thereby improving the quality of the final decoded signal. For example, in audio codecs, this can reduce holes in the frequency domain of the decoded audio signal, improving the subjective auditory quality of the decoded audio signal.

[0380] like Figure 14A For example, after latent variable quantization, the quantized value of the latent variable amplitude at some element positions may be zero. For instance, the latent variable amplitudes at element indices 2, 5, 6, 7, 11, 12, and 16 in the figure have zero quantized values. Figure 14B As shown in the example, after the decoding end uses the method proposed in the embodiment of this application to compensate for the dequantized latent variables, the amplitude of the latent variables at positions such as element number 2, 5, 6, 7, 11, 12, and 16 is restored to a certain extent. This reduces the distortion of the input data of the decoding neural network, thereby improving the quality of the final decoded signal.

[0381] Some device embodiments are described below.

[0382] See Figure 15This application also provides an audio encoder 1500, wherein the audio encoder may include several functional units, and the several functional units cooperate to complete audio encoding-related methods.

[0383] For example, an audio encoder may include: an acquisition unit 1510, a parameter processing unit 1520, and an encoding unit 1530.

[0384] The acquisition unit 1510 is used to acquire the audio data to be encoded.

[0385] The parameter processing unit 1520 is used to process the audio data to be encoded using an encoding neural network to generate a first latent variable; to quantize the first latent variable to obtain a second latent variable; and to obtain latent variable compensation parameters based on the first latent variable and the second latent variable.

[0386] The encoding unit 1530 is used to encode the latent variable compensation parameters and write the encoding result of the latent variable compensation parameters into the bitstream; to encode the second latent variable and write the encoding result of the second latent variable into the bitstream.

[0387] It is understandable that the audio encoder 1500 can be used to implement... Figure 2 , Figure 4A , Figure 6A , Figure 8A , Figure 10A or Figure 12A The relevant methods exemplified above.

[0388] See Figure 16 This application also provides an audio decoder 1600, which may include several functional units, and the several functional units work together to complete the audio decoding related methods.

[0389] Specifically, the audio decoder 1600 may include a decoding unit 1610 and a parameter processing unit 1620.

[0390] The parameter processing unit 1620 is used to obtain latent variable compensation parameters and a third latent variable based on the code stream; and to perform compensation processing on the third latent variable according to the latent variable compensation parameters to obtain the reconstructed first latent variable.

[0391] The decoding unit 1610 is further configured to process the reconstructed first latent variable using a decoding neural network to generate decoded audio data.

[0392] Obtaining latent variable compensation parameters and a third latent variable based on the bitstream may include: decoding the latent variable compensation parameters and the second latent variable from the bitstream, wherein the value of the current element in the second latent variable is a quantized value; and performing dequantization processing on the second latent variable to obtain the third latent variable.

[0393] It is understandable that the audio decoder 1600 can be used to implement... Figure 3 , Figure 5 , Figure 7 , Figure 9A , Figure 11A or Figure 13A The relevant methods exemplified above.

[0394] See Figure 17 This application also provides an audio encoder 1700, including a processor 1710, a memory 1720 coupled together, and a program stored in the memory. When the program instructions stored in the memory are executed by the processor, some or all of the steps of any audio encoding-related method provided in this application are implemented.

[0395] See Figure 18 This application also provides an audio decoder 1800, which may include: a processor 1810, the processor and a memory 1820 coupled together, the memory storing a program, and when the program instructions stored in the memory are executed by the processor, implementing some or all of the steps of any audio decoding-related method provided in this application.

[0396] The processor in this application embodiment is also referred to as a Central Processing Unit (CPU). In specific applications, the components of the audio codec are coupled together, for example, through a bus system. This bus system may include, in addition to a data bus, a power bus, a control bus, and a status signal bus. Any of the methods disclosed in the above-described embodiments of this application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In some implementations, some or all of the steps of the above methods can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, an off-the-shelf programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; for example, a processor can read information from the memory and, in conjunction with its hardware, complete some or all of the steps described above.

[0397] See Figure 19 This application also provides a communication system, including: an audio encoder 1910 and an audio decoder 1920; the audio encoder is any type of audio encoder provided in this application embodiment. The audio decoder is any type of audio decoder provided in this application embodiment.

[0398] See Figure 20 This application also provides a network device 2000, which includes a processor 2010 and a memory 2020. The processor 2010 is coupled to the memory 2020 and is used to read and execute a program stored in the memory to implement some or all of the steps of any of the methods provided in this application.

[0399] The network device may be, for example, a chip or a system-on-a-chip.

[0400] It is understood that the specific details of how the above-mentioned device achieves its related functions can be found in the description of the above method embodiments, and will not be repeated here.

[0401] This application also provides a computer-readable storage medium storing a computer program that, when executed by hardware (such as a processor), can perform some or all of the steps of any method in this application.

[0402] This application also provides a computer-readable storage medium storing a computer program that is executed by hardware (e.g., a processor) to implement some or all of the steps of any method executed by any device in this application.

[0403] This application provides a computer-readable storage medium storing a bitstream, the bitstream being obtained based on any of the audio data encoding methods provided in this application.

[0404] This application also provides a computer program product including instructions that, when run on a computer device, cause the computer device to perform some or all of the steps of any method in this application.

[0405] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., optical disk), or a semiconductor medium (e.g., solid-state drive), etc. In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0406] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0407] In the embodiments provided in this application, it should be understood that the disclosed apparatus can also be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the indirect or direct coupling or communication connections shown or discussed may be through some interfaces; the indirect coupling or communication connections of devices or units may be electrical or other forms.

[0408] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0409] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0410] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium may include, for example, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. An audio data encoding method, characterized in that, include: Obtain the audio data to be encoded; The audio data to be encoded is processed using an encoding neural network to generate a first latent variable; The first latent variable is quantified to obtain the second latent variable; The latent variable compensation parameters are obtained based on the first latent variable and the second latent variable; The latent variable compensation parameters are encoded, and the encoding results of the latent variable compensation parameters are written into the bitstream; The second latent variable is encoded, and the encoding result of the second latent variable is written into the bitstream; The step of obtaining the latent variable compensation parameters based on the first latent variable and the second latent variable includes: Determine the elements in the third latent variable or the second latent variable that meet the preset conditions, wherein the third latent variable is obtained by dequantizing the second latent variable; The quantization error of the element that satisfies the preset condition is obtained based on the first latent variable and the second latent variable or based on the first latent variable and the third latent variable. The latent variable compensation parameters are obtained based on the quantization error.

2. The method according to claim 1, characterized in that, The third latent variable or the second latent variable includes the first element; The conditions for satisfying the preset conditions include: the value of the first element is less than or equal to a preset value; wherein, when the first element is an element in the second latent variable, the value of the first element is a quantized value, or when the first element is an element in the third latent variable, the value of the first element is a dequantized value.

3. The method according to claim 1, characterized in that, The quantization error of the element satisfying the preset condition is obtained based on the first latent variable and the second latent variable, including: determining the quantization error of the first element based on the first value of the first element in the first latent variable and the second value of the first element in the second latent variable; or The step of obtaining the quantization error of the element that satisfies the preset condition based on the first latent variable and the third latent variable includes: determining the quantization error of the first element based on the first value of the first element in the first latent variable and the third value of the first element in the third latent variable.

4. The method according to any one of claims 1 to 3, characterized in that, The encoding neural network is either a fully connected neural network or a convolutional neural network, and the convolutional neural network includes only one channel. The latent variable compensation parameter is a scalar, wherein the scalar is used to indicate the quantization error of all elements in the second latent variable that satisfy the preset conditions, or the scalar is used to indicate the quantization error of all elements in the third latent variable that satisfy the preset conditions.

5. The method according to any one of claims 1 to 3, characterized in that, The encoding neural network is a convolutional neural network, which includes at least two channels. The second latent variable corresponds to the at least two channels, which include the first channel. The second latent variable is an m×n matrix. in, The latent variable compensation parameter is a scalar, which is used to indicate the quantization error of all elements in the second latent variable that satisfy the preset conditions; or, The latent variable compensation parameter is a vector, the dimension of which is equal to the number of channels in the convolutional neural network. The vector elements in the vector correspond one-to-one with the at least two channels. The first vector element in the vector corresponds to the first channel. The first vector element is used to indicate the quantization error of all elements in the submatrix of the m×n matrix that meet the preset conditions. The first channel corresponds to the submatrix, and the number of elements in the submatrix is ​​less than m×n.

6. The method according to any one of claims 1 to 3, characterized in that, The encoding neural network is a convolutional neural network, which includes at least three channels. The second latent variable corresponds to the at least three channels, and the second latent variable is an m×n matrix. The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels in the convolutional neural network. The first vector element in the vector corresponds to at least two of the at least three channels, the at least two channels including a second channel and a first channel. The first vector element is used to indicate the quantization error of all elements in the first submatrix of the m×n matrix that satisfy a preset condition. The first vector element is also used to indicate the quantization error of all elements in the second submatrix of the m×n matrix that satisfy a preset condition. The first submatrix corresponds to the first channel, the second submatrix corresponds to the second channel, the number of elements in the first submatrix is ​​less than m×n, and the number of elements in the second submatrix is ​​less than m×n.

7. The method according to any one of claims 1 to 3, characterized in that, The encoding neural network is a convolutional neural network, which includes at least two channels. The third latent variable corresponds to the at least two channels, which include the first channel. The third latent variable is an m×n matrix. in, The latent variable compensation parameter is a scalar, which is used to indicate the quantization error of all elements in the third latent variable that satisfy the preset conditions; or, The latent variable compensation parameter is a vector, the dimension of which is equal to the number of channels in the convolutional neural network. The vector elements in the vector correspond one-to-one with the at least two channels. The first vector element in the vector elements corresponds to the first channel. The first vector element is used to indicate the quantization error of all elements in the submatrix of the m×n matrix that meet the preset conditions. The first channel corresponds to the submatrix, and the number of elements in the submatrix is ​​less than m×n.

8. The method according to any one of claims 1 to 3, characterized in that, The encoding neural network is a convolutional neural network, which includes at least three channels. The third latent variable corresponds to the at least three channels and is an m×n matrix. The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels in the convolutional neural network. The first vector element in the vector corresponds to at least two of the at least three channels, the at least two channels including a second channel and a first channel. The first vector element is used to indicate the quantization error of all elements in the first submatrix of the m×n matrix that satisfy a preset condition. The first vector element is also used to indicate the quantization error of all elements in the second submatrix of the m×n matrix that satisfy a preset condition. The first submatrix corresponds to the first channel, the second submatrix corresponds to the second channel, the number of elements in the first submatrix is ​​less than m×n, and the number of elements in the second submatrix is ​​less than m×n.

9. The method according to any one of claims 1 to 3, characterized in that, The encoding neural network is a convolutional neural network, which includes at least two channels, and the second latent variable corresponds to the first channel of the at least two channels. in, The latent variable compensation parameter is a scalar, which is used to indicate the quantization error of all elements in the at least two latent variables corresponding to the at least two channels that satisfy the preset conditions. The at least two latent variables include the second latent variable. or, The latent variable compensation parameter is a vector, the dimension of which is equal to the number of channels of the convolutional neural network, and the vector elements in the vector correspond one-to-one with the at least two channels. The vector elements include a first vector element corresponding to the first channel, and the first vector element is used to indicate the quantization error of all elements in the second latent variable that meet the preset conditions.

10. The method according to any one of claims 1 to 3, characterized in that, The encoding neural network is a convolutional neural network, which includes at least three channels, and the second latent variable corresponds to the first channel among the at least three channels; in, The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels of the convolutional neural network. The first vector element in the vector corresponds to at least two of the at least three channels, the at least two channels including the second channel and the first channel. The first vector element is used to indicate the quantization error of all elements in the second latent variable that satisfy the preset conditions. The first vector element is also used to indicate the quantization error of all elements in another latent variable corresponding to the second channel that satisfy the preset conditions.

11. The method according to any one of claims 1 to 3, characterized in that, The encoding neural network is a convolutional neural network, which includes at least two channels, and the third latent variable corresponds to the first channel of the at least two channels; in, The latent variable compensation parameter is a scalar, which is used to indicate the quantization error of all elements in the at least two latent variables corresponding to the at least two channels that satisfy the preset conditions. The at least two latent variables include the third latent variable. or, The latent variable compensation parameter is a vector, the dimension of which is equal to the number of channels of the convolutional neural network, and the vector elements in the vector correspond one-to-one with the at least two channels. The vector element corresponding to the first channel is used to indicate the quantization error of all elements in the third latent variable that meet the preset conditions.

12. The method according to any one of claims 1 to 3, characterized in that, The encoding neural network is a convolutional neural network, which includes at least three channels, and the third latent variable corresponds to the first channel among the at least three channels; in, The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels of the convolutional neural network. The first vector element in the vector corresponds to at least two of the at least three channels, the at least two channels including the second channel and the first channel. The first vector element is used to indicate the quantization error of all elements in the third latent variable that meet the preset conditions. The first vector element is also used to indicate the quantization error of all elements in another latent variable corresponding to the second channel that meet the preset conditions.

13. The method according to claim 6, characterized in that, The method further includes: writing the correspondence between the first vector element and the first channel and the second channel into the bitstream.

14. The method according to claim 8, characterized in that, The method further includes: writing the correspondence between the first vector element and the first channel and the second channel into the bitstream.

15. An audio data decoding method, characterized in that, include: The latent variable compensation parameter and the third latent variable are obtained based on the bitstream; The third latent variable is compensated according to the latent variable compensation parameters to obtain the reconstructed first latent variable; The first latent variable of the reconstruction is processed using a decoding neural network to generate decoded audio data; in, The process of obtaining the latent variable compensation parameter and the third latent variable based on the bitstream includes: decoding the latent variable compensation parameter and the second latent variable from the bitstream, wherein the value of the current element in the second latent variable is a quantized value; and performing dequantization processing on the second latent variable to obtain the third latent variable.

16. The method according to claim 15, characterized in that, The step of compensating the third latent variable according to the latent variable compensation parameter to obtain the reconstructed first latent variable includes: determining the elements in the third latent variable that meet preset conditions according to the second latent variable or the third latent variable; and compensating the elements in the third latent variable that meet preset conditions according to the latent variable compensation parameter to obtain the reconstructed first latent variable.

17. The method according to claim 16, characterized in that, The elements in the third latent variable that meet the preset conditions are compensated according to the latent variable compensation parameters to obtain the reconstructed first latent variable, including: Generate random noise; The amplitude or energy of the generated random noise is adjusted according to the latent variable compensation parameters to obtain amplitude-adjusted random noise. The elements in the third latent variable that meet the preset conditions are compensated based on the random noise adjusted by the amplitude or energy to obtain the reconstructed first latent variable.

18. The method according to claim 16, characterized in that, The step of determining the elements in the third latent variable that satisfy the preset conditions based on the second latent variable or the third latent variable includes: Identify the elements in the second latent variable that satisfy the preset conditions; The elements in the third latent variable that satisfy the preset conditions correspond to the positions of the elements in the second latent variable that satisfy the preset conditions.

19. The method according to claim 17, characterized in that, The step of determining the elements in the third latent variable that satisfy the preset conditions based on the second latent variable or the third latent variable includes: Identify the elements in the second latent variable that satisfy the preset conditions; The elements in the third latent variable that satisfy the preset conditions correspond to the positions of the elements in the second latent variable that satisfy the preset conditions.

20. The method according to any one of claims 16 to 19, characterized in that, The third latent variable or the second latent variable includes the first element; The conditions for satisfying the preset conditions include: when the value of the first element is less than or equal to a preset value, wherein when the first element is an element in the second latent variable, the value of the first element is a quantized value, or when the first element is an element in the third latent variable, the value of the first element is a dequantized value.

21. The method according to any one of claims 16 to 19, characterized in that, The decoding neural network is a fully connected neural network or a convolutional neural network, wherein the convolutional neural network includes only one channel; The latent variable compensation parameter is a scalar, wherein the scalar is used to compensate all elements in the third latent variable that satisfy the preset conditions.

22. The method according to any one of claims 16 to 19, characterized in that, The decoding neural network is a convolutional neural network, which includes at least two channels. The third latent variable corresponds to the at least two channels, which include the first channel. The third latent variable is an m×n matrix. in, The latent variable compensation parameter is a scalar, which is used to compensate all elements in the third latent variable that meet the preset conditions; or, The latent variable compensation parameter is a vector, the dimension of which is equal to the number of channels in the convolutional neural network. The vector elements in the vector correspond one-to-one with the at least two channels. The first vector element in the vector elements corresponds to the first channel. The first vector element is used to compensate all elements in the submatrix of the m×n matrix that meet the preset conditions. The first channel corresponds to the submatrix, and the number of elements in the submatrix is ​​less than m×n.

23. The method according to any one of claims 16 to 19, characterized in that, The decoding neural network is a convolutional neural network, which includes at least three channels. The third latent variable corresponds to the at least three channels and is an m×n matrix. in, The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels in the convolutional neural network. The first vector element in the vector corresponds to at least two of the at least three channels, the at least two channels including the second channel and the first channel. The first vector element is used to compensate all elements in the first submatrix of the m×n matrix that satisfy a preset condition. The first vector element is also used to compensate all elements in the second submatrix of the m×n matrix that satisfy a preset condition. The first submatrix corresponds to the first channel, the second submatrix corresponds to the second channel, the number of elements in the first submatrix is ​​less than m×n, and the number of elements in the second submatrix is ​​less than m×n.

24. The method according to any one of claims 16 to 19, characterized in that, The decoding neural network is a convolutional neural network, which includes at least two channels, and the third latent variable corresponds to the first channel of the at least two channels; in, The latent variable compensation parameter is a scalar, which is used to compensate all elements in the at least two latent variables corresponding to the at least two channels that meet the preset conditions, wherein the at least two latent variables include the third latent variable; or, The latent variable compensation parameter is a vector, the dimension of which is equal to the number of channels in the convolutional neural network, and the vector elements in the vector correspond one-to-one with the at least two channels. The vector elements in the vector corresponding to the first channel are used to compensate all elements in the third latent variable that meet the preset conditions.

25. The method according to any one of claims 16 to 19, characterized in that, The decoding neural network is a convolutional neural network, which includes at least three channels, and the third latent variable corresponds to the first channel among the at least three channels; in, The latent variable compensation parameter is a vector, the dimension of which is less than the number of channels in the convolutional neural network. The first vector element in the vector corresponds to at least two of the at least three channels, the at least two channels including the second channel and the first channel. The first vector element is used to compensate all elements in the third latent variable that meet the preset conditions. The first vector element is also used to compensate all elements in another latent variable corresponding to the second channel that meet the preset conditions.

26. An audio encoder, characterized in that, include: The acquisition unit is used to acquire the audio data to be encoded. The parameter processing unit is used to process the audio data to be encoded using an encoding neural network to generate a first latent variable; to quantize the first latent variable to obtain a second latent variable; and to obtain latent variable compensation parameters based on the first latent variable and the second latent variable. The encoding unit is used to encode the latent variable compensation parameters and write the encoding result of the latent variable compensation parameters into the bitstream; and to encode the second latent variable and write the encoding result of the second latent variable into the bitstream. in, The step of obtaining the latent variable compensation parameters based on the first latent variable and the second latent variable includes: Determine the elements in the third latent variable or the second latent variable that meet the preset conditions, wherein the third latent variable is obtained by dequantizing the second latent variable; The quantization error of the element that satisfies the preset condition is obtained based on the first latent variable and the second latent variable or based on the first latent variable and the third latent variable. The latent variable compensation parameters are obtained based on the quantization error.

27. An audio decoder, characterized in that, include: The parameter processing unit is used to obtain the latent variable compensation parameters and the third latent variable based on the bitstream; The third latent variable is compensated according to the latent variable compensation parameters to obtain the reconstructed first latent variable; The decoding unit is also used to process the reconstructed first latent variable using a decoding neural network to generate decoded audio data; in, The process of obtaining the latent variable compensation parameter and the third latent variable based on the bitstream includes: decoding the latent variable compensation parameter and the second latent variable from the bitstream, wherein the value of the current element in the second latent variable is a quantized value; and performing dequantization processing on the second latent variable to obtain the third latent variable.

28. An audio encoder, characterized in that, include: The method includes a processor coupled to a memory storing a program, wherein the program instructions stored in the memory are executed by the processor to implement the method of any one of claims 1 to 14.

29. An audio decoder, characterized in that, include: The method includes a processor coupled to a memory storing a program, which, when executed by the processor, implements the method of any one of claims 15 to 25.

30. A communication system, characterized in that, include: An audio encoder and an audio decoder; the audio encoder is the audio encoder as described in any one of claims 26 or 28; The audio decoder is the audio decoder as described in any one of claims 27 or 29.

31. A computer-readable storage medium comprising a program that, when run on a computer, causes the computer to perform the method as claimed in any one of claims 1-14.

32. A computer-readable storage medium comprising a program that, when run on a computer, causes the computer to perform the method as claimed in any one of claims 15-24.

33. A network device, comprising a processor and a memory, characterized in that, The processor is coupled to the memory and is used to read and execute instructions stored in the memory to implement the method as claimed in any one of claims 1-24.

34. The network device as described in claim 33, characterized in that, The network device is a chip or a system-on-a-chip.

35. A computer-readable storage medium, characterized in that, The storage contains a bitstream obtained by the method described in any one of claims 1-14.

36. A computer program product, characterized in that, The computer program product includes computer programs. When the computer program is run on a computer, it causes the computer to perform the method as described in any one of claims 1-24.

Images