An audio processing method, apparatus and electronic device

Abstract

The application discloses an audio processing method and device and electronic equipment, and relates to the field of audio processing. An audio waveform is obtained, the audio waveform is input into a configured audio processing model, an MDCT spectrum of the audio waveform is encoded into a continuous code by a causal encoder in the audio processing model, and the continuous code is input into a residual scalar vector quantizer for quantization processing to obtain a quantization result. The quantization result is input into a causal decoder to decode the MDCT spectrum, and the MDCT spectrum is recovered into the audio waveform through inverse MDCT. The MDCT spectrum of the audio waveform is used as a parameter feature of the audio waveform and is used for encoding, quantization and decoding, so that the frequency domain resolution is improved and the calculation complexity in the decoding stage is reduced. In addition, the causal model structure used in the audio processing model does not need to input any future information compared with the traditional non-causal model structure, so that the delay can be greatly reduced, the audio processing efficiency is improved, and the use requirement of the real-time communication scene is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, and more particularly, to an audio processing method, device and electronic equipment. BACKGROUND

[0002] Audio codec is an important research content of audio processing, aiming to discretize audio with the least bit usage while maintaining the highest decoded audio quality, thus playing an important role in various fields, such as real-time communication and speech large model. With the development of deep learning, neural network audio codec is proposed and outperforms traditional methods such as Opus and EVS. Neural network audio codec is generally composed of an encoder, a residual vector quantizer (RVQ) and a decoder. According to the type of the encoding object, neural network audio codec can be divided into two categories: one is the neural network audio codec based on waveform coding, such as SoundStream, Encodec, HiFi-Codec, AudioDec and DAC, which directly discretizes the audio waveform in the time domain. However, this method requires hundreds of up-sampling and down-sampling operations, which will lead to high model complexity and low efficiency, and is not conducive to the application in the speech communication scenario. The other is the neural network audio codec based on spectral coding, for example, APCodec uses a two-way structure to encode, quantize and decode the audio amplitude and phase spectrum, thereby improving the generation efficiency and reducing the model complexity.

[0003] Although the existing research on neural network audio codec has made great progress, there are still some problems. Most of the existing research on neural network audio codec focuses on the design of non-causal structure, which leads to high delay in the audio processing process and does not meet the use requirements of real-time communication scenarios. SUMMARY

[0004] Therefore, the present application discloses an audio processing method, device and electronic equipment to greatly reduce the delay in audio processing, improve the efficiency of audio processing, and thus meet the use requirements of real-time communication scenarios.

[0005] An audio processing method, comprising:

[0006] obtaining an audio waveform;

[0007] The audio waveform is input into an audio processing model configured, a discrete cosine transform (MDCT) spectrum of the audio waveform is encoded into a continuous code by a causal encoder in the audio processing model, the continuous code is input into a residual scalar vector quantizer for quantization processing to obtain a quantization result, the quantization result is input into a causal decoder to decode the MDCT spectrum, and the MDCT spectrum is recovered into the audio waveform by inverse MDCT.

[0008] Optionally, the causal encoder comprises:

[0009] a first causal input convolutional layer, a first causal improved ConvNeXt v2 module set, a first linear layer, a causal down-sampling convolutional layer and a first causal output convolutional layer connected in sequence;

[0010] The first causal input convolutional layer is configured to perform preliminary processing of dimension transformation on the input MDCT spectrum and input the preliminary processing result into the first causal improved ConvNeXt v2 module set.

[0011] The first causal improved ConvNeXt v2 module set is configured to perform deep processing on the preliminary processing result to obtain a deep processing result and input the deep processing result into the first linear layer.

[0012] The first linear layer is configured to perform linear transformation on the deep processing result to obtain a linear transformation result and input the linear transformation result into the causal down-sampling convolutional layer.

[0013] The causal down-sampling convolutional layer is configured to perform feature down-sampling on the linear transformation result to obtain a down-sampling result and input the down-sampling result into the first causal output convolutional layer.

[0014] The first causal output convolutional layer is configured to perform dimension transformation processing on the down-sampling result to obtain the continuous code.

[0015] Optionally, the first causal improved ConvNeXt v2 module set comprises a plurality of identical causal improved ConvNeXt v2 modules.

[0016] Each of the causal improved ConvNeXt v2 modules is obtained by causally processing a ConvNeXt v2 module proposed in an APCodec and an MDCTCodec.

[0017] Optionally, in each of the causal improvement ConvNeXt v2 modules, the output result obtained by sequentially passing the preliminary processing result input by the first causal input convolutional layer through a depth layer causal convolutional layer, a linear layer for mapping features to a higher dimension, a Gaussian error linear unit, and a linear layer for mapping features to the original low dimension, and adding the total result after the preliminary processing result as input, is the final output of the causal improvement ConvNeXt v2 module.

[0018] Optionally, the residual scalar vector quantizer comprises:

[0019] The scalar quantizer set and the improved vector quantizer set are connected in a residual manner, the plurality of scalar quantizers in the scalar quantizer set are connected in a residual manner, and the plurality of improved vector quantizers in the improved vector quantizer set are connected in a residual manner.

[0020] The scalar quantizer is used to process the continuous code input by the causal encoder to construct a rough audio profile by taking rounding as the quantization principle.

[0021] The improved vector quantizer is used to refine acoustic details on the basis of the rough audio profile to obtain the quantization result.

[0022] Optionally, each of the improved vector quantizers is obtained by introducing an online codebook clustering training strategy and a load balancing loss on the basis of an original vector quantizer.

[0023] Optionally, the causal decoder comprises:

[0024] a second causal input convolutional layer, a causal upsampling convolutional layer, a second linear layer, a second causal improvement ConvNeXt v2 module set, and a second causal output convolutional layer are sequentially connected in series;

[0025] The second causal input convolutional layer is used to restore the input quantization result to the original dimension.

[0026] The causal upsampling convolutional layer is used to perform feature upsampling on the original dimension quantization result to obtain an upsampling result, and input the upsampling result to the second linear layer.

[0027] The second linear layer is used to linearly change the upsampling result to obtain a linear transformation result, and input the linear transformation result to the second causal improvement ConvNeXt v2 module set.

[0028] The second causal improvement ConvNeXt v2 module set is used to perform depth processing on the linear transformation result to obtain a depth processing result.

[0029] The second causal output convolution layer is configured to obtain the MDCT spectrum from the deep processing result, to recover the MDCT spectrum into the audio waveform through inverse MDCT.

[0030] Optionally, the audio processing model adopts a generative adversarial training mode, and a generator loss function and a discriminator loss function are used to alternately train a generator and a discriminator in a training process until the audio processing model converges.

[0031] The generator loss function is a linear combination of an adversarial loss function, a feature matching loss function, an MDCT spectrum loss function, a mel spectrum loss function, a codebook loss function, a commitment loss function and a load balancing loss function of the generator according to a preset ratio.

[0032] An audio processing apparatus comprises:

[0033] An acquisition unit configured to acquire an audio waveform;

[0034] A processing unit configured to input the audio waveform into a configured audio processing model, encode a discrete cosine transform (MDCT) spectrum of the audio waveform into a continuous code through a causal encoder in the audio processing model, input the continuous code into a residual scalar vector quantizer for quantization processing to obtain a quantization result, decode the quantization result through a causal decoder to obtain the MDCT spectrum, and recover the MDCT spectrum into the audio waveform through inverse MDCT.

[0035] An electronic device comprises a memory and a processor;

[0036] The memory is configured to store at least one instruction;

[0037] The processor is configured to execute the at least one instruction to implement the audio processing method described above.

[0038] From the above technical solutions can be known, the application discloses an audio processing method, device and electronic equipment, obtain the audio waveform, input the audio waveform to the configured audio processing model, encode the MDCT spectrum of the audio waveform into a continuous code through the causal encoder in the audio processing model, and input to the residual scalar vector quantizer for quantization processing to obtain a quantization result, input the quantization result to the causal decoder to decode the MDCT spectrum, and restore the MDCT spectrum to the audio waveform through the inverse MDCT. The application takes the MDCT spectrum of the audio waveform as the parameter feature of the audio waveform and uses it for encoding, quantization and decoding, realizes the reduction of the calculation complexity in the decoding stage while improving the frequency domain resolution, and the causal model structure used in the audio processing model does not need to input any future information compared with the traditional non-causal model structure, so the delay can be greatly reduced, the audio processing efficiency is improved, and the use demand of the real-time communication scene is met. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by the disclosed drawings without creative labor for those skilled in the art.

[0040] Figure 1 A flow chart of an audio processing method disclosed by the embodiment of the present application;

[0041] Figure 2 A structural schematic diagram of a causal encoder disclosed by the embodiment of the present application;

[0042] Figure 3 A structural schematic diagram of a residual scalar vector quantizer disclosed by the embodiment of the present application;

[0043] Figure 4 A structural schematic diagram of a causal decoder disclosed by the embodiment of the present application;

[0044] Figure 5 A structural schematic diagram of an audio processing model disclosed by the embodiment of the present application;

[0045] Figure 6 A structural schematic diagram of an audio processing device disclosed by the embodiment of the present application;

[0046] Figure 7 A structural schematic diagram of an electronic equipment disclosed by the embodiment of the present application. DETAILED DESCRIPTION

[0047] Clearly, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present application.

[0048] The embodiments of the present application disclose an audio processing method and device and electronic equipment. An audio waveform is obtained, the audio waveform is input into a configured audio processing model, an MDCT spectrum of the audio waveform is coded into a continuous code by a causal encoder in the audio processing model, and the continuous code is input into a residual scalar vector quantizer to obtain a quantization result. The quantization result is input into a causal decoder to decode the MDCT spectrum, and the MDCT spectrum is recovered into the audio waveform by inverse MDCT. The present application takes the MDCT spectrum of the audio waveform as a parameter feature of the audio waveform and uses the MDCT spectrum for coding, quantization and decoding, so that the frequency domain resolution is improved and the calculation complexity in the decoding stage is reduced, and the causal model structure used in the audio processing model does not need to input any future information relative to the traditional non-causal model structure, so that the delay can be greatly reduced and the audio processing efficiency is improved, thereby meeting the use requirement of the real-time communication scene.

[0049] Referring to Figure 1 The embodiments of the present application disclose an audio processing method flowchart, and the method can include the following steps.

[0050] In the actual application, when the audio needs to be processed, the real-time audio waveform can be obtained.

[0051] In the actual application, when the audio needs to be processed, the real-time audio waveform can be obtained.

[0052] In the actual application, when the audio needs to be processed, the real-time audio waveform can be obtained.

[0053] The quantization result obtained by the residual scalar vector quantizer on the input continuous code can be a quantization code.

[0054] The MDCT (Modified Discrete Cosine Transform) is an improved version of the Discrete Cosine Transform (DCT), which is mainly used for transform coding of speech, wideband audio and image signals.

[0055] In audio codec, MDCT can convert time domain signal into frequency domain signal. At the same time, MDCT can effectively remove the periodic noise caused by edge effect, and improve the accuracy of signal processing. And in the case of the same encoding rate, MDCT can realize higher compression ratio and better signal quality compared with DCT.

[0056] Based on this, the MDCT spectrum of the audio waveform is taken as the parameter feature of the audio waveform in the present application, and the MDCT spectrum is obtained by encoding through the causal encoder in the audio processing model, quantization through the residual scalar vector quantizer, and decoding through the causal decoder. Finally, the original audio waveform is obtained by inverse MDCT restoration processing of the MDCT spectrum.

[0057] Among them, the audio processing model in the present application is composed of a causal encoder, a residual scalar vector quantizer and a causal decoder. The causal encoder encodes the MDCT spectrum of the audio waveform into a continuous code. The residual scalar vector quantizer mainly discretizes the continuous code when quantizing the continuous code, generates the corresponding discrete representation and generates the quantization code as the quantization result. The causal decoder takes the quantization code as the input, decodes the MDCT spectrum of the audio waveform from the quantization code, and restores the inverse MDCT to the audio waveform through the inverse MDCT.

[0058] It should be noted that the audio processing model is a streaming neural network audio codec, which follows the basic framework of MDCTCodec and adopts a causal model structure (causal encoder and causal decoder) to realize the low-delay streaming inference of audio. At the same time, in order to make up for the decline in coding quality caused by the causal structure, the audio processing module also introduces a new residual scalar vector quantization strategy, so that the audio processing model can realize low-delay streaming inference of audio while maintaining the coding quality, efficiency and complexity of MDCTCodec, thus meeting the use requirements of real-time communication scenarios.

[0059] The causal model structure adopted in the present application can greatly reduce the delay compared with the traditional non-causal model structure, and the delay refers to the amount of input required for model startup. The reason is that the non-causal model needs to input future information, so it has a high delay. For example, when predicting the result at time t, the model needs to use the input information at time t+t0 at least, and then the delay time is t0. Using the all-causal model of the present application, no future information needs to be input, that is, t0=0, so the delay can be greatly reduced.

[0060] In addition, the decoding audio quality of the audio processing model proposed in the application is comparable to that of the advanced non-streaming audio codec at the present stage, and the audio processing model has the characteristics of low delay, high efficiency and small amount of model parameters, and is suitable for application in real-time communication.

[0061] In conclusion, the application discloses an audio processing method, which acquires an audio waveform, inputs the audio waveform into a configured audio processing model, encodes an MDCT spectrum of the audio waveform into a continuous code through a causal encoder in the audio processing model, inputs the continuous code into a residual scalar vector quantizer for quantization processing to obtain a quantization result, decodes the MDCT spectrum out of the quantization result through a causal decoder, and restores the MDCT spectrum into the audio waveform through inverse MDCT. The application takes the MDCT spectrum of the audio waveform as a parameter feature of the audio waveform and uses it for encoding, quantization and decoding, so as to reduce the computational complexity in the decoding stage while improving the frequency domain resolution, and the causal model structure used in the audio processing model does not need to input any future information relative to the traditional non-causal model structure, so the delay can be greatly reduced and the audio processing efficiency can be improved, thereby meeting the use requirements of real-time communication scenarios.

[0062] Specifically, referring to Figure 2 The application discloses a structure diagram of a causal encoder, and the causal encoder 10 comprises:

[0063] a first causal input convolutional layer, a first causal improved ConvNeXt v2 module set, a first linear layer, a causal down-sampling convolutional layer and a first causal output convolutional layer connected in sequence.

[0064] The first causal improved ConvNeXt v2 module set comprises a plurality of identical causal improved ConvNeXt v2 modules.

[0065] In actual application, the number of the causal improved ConvNeXt v2 modules included in the first causal improved ConvNeXt v2 module set is determined according to actual needs, for example, 8 (see Figure 2 ), which is not limited in the application.

[0066] Each of the causal improved ConvNeXt v2 modules is obtained by causally processing a ConvNeXt v2 module in an APCodec and an MDCTCodec.

[0067] The specific working principles of the components in the causal encoder 10 are as follows:

[0068] The first causal input convolutional layer is configured to perform preliminary processing of dimension transformation on the input MDCT spectrum and input the preliminary processing result to the first causal improved ConvNeXt v2 module set.

[0069] The first causal improvement ConvNeXt v2 module set is used for deep processing of the preliminary processing result to obtain a deep processing result, and the deep processing result is input to the first linear layer.

[0070] The first linear layer is used for linear transformation of the deep processing result to obtain a linear transformation result, and the linear transformation result is input to the causal down-sampling convolution layer.

[0071] The causal down-sampling convolution layer is used for feature down-sampling of the linear transformation result to obtain a down-sampling result, and the down-sampling result is input to the first causal output convolution layer.

[0072] The first causal output convolution layer is used for dimension transformation processing of the down-sampling result to obtain the continuous code.

[0073] Among them, the causal down-sampling convolution layer reduces the time resolution by performing feature down-sampling on the linear transformation result. Finally, the first causal output convolution layer further reduces the feature dimension by performing dimension transformation processing on the down-sampling result to generate the continuous code as the input of the residual scalar vector quantizer.

[0074] In each of the causal improvement ConvNeXt v2 modules, the output result obtained by sequentially passing the preliminary processing result input by the first causal input convolution layer through a depth-wise causal convolution layer, a linear layer that maps features to a higher dimension, a Gaussian error linear unit (GELU), and a linear layer that maps features to the original low dimension, is added to the preliminary processing result as input. The total result after the addition is the final output of the causal improvement ConvNeXt v2 module.

[0075] In order to improve the utilization rate of the audio processing model in the codebook application and compensate for the loss of audio quality caused by structural causality, the application further adds a new residual scalar vector quantization strategy to the audio processing model. The strategy sequentially connects a scalar quantizer and an improved vector quantizer in a residual manner. The scalar quantizer is used to construct a rough audio profile, and the improved vector quantizer is used to refine acoustic details.

[0076] Specifically, referring to Figure 3 The residual scalar vector quantizer disclosed in the embodiment of the application has a structure diagram as shown in FIG. 2. The residual scalar vector quantizer 20 includes a scalar quantizer set and an improved vector quantizer set, and the scalar quantizer set and the improved vector quantizer set are connected in a residual manner.

[0077] The scalar quantizer set in this application includes multiple scalar quantizers (SQ), which are connected using a residual method. For example, Figure 3 The scalar quantizer set includes Ns scalar quantizers, namely scalar quantizer 1 to scalar quantizer Ns.

[0078] The improved vector quantizer set consists of multiple improved vector quantizers (IVQs), which are connected using a residual method. For example, Figure 3 The improved vector quantizer set includes N v There are three improved vector quantizers, namely improved vector quantizer 1 to improved vector quantizer N. v .

[0079] The residual scalar vector quantizer 20 outputs to the causal encoder 10 Discretize and quantize the results. As input to the causal decoder, where, The dimension is D A set of vectors whose elements are real numbers, that is, all D A set of 3D real vectors D It is the dimension of encoding and quantizing features.

[0080] The first input to SQ is The input to other quantizers (scalar quantizers and improved vector quantizers) is the quantization residual of the previous quantizer. The final quantization result... It is the sum of the outputs of all quantizers.

[0081] (1) Scalar quantizer SQ

[0082] SQ is used to process the continuous code input to the causal encoder using rounding as the quantization principle, to construct a rough audio profile.

[0083] The goal of SQ is to roughly construct an audio profile using rounding as the quantization principle. Assume SQ applies a specific input vector... Discretization, output quantization result .

[0084] first After a trainable weight The linear layer is obtained The expression is as follows:

[0085] (1);

[0086] in, Let T denote the transformed input vector, and let T denote the vector transpose. denotes the first element of denotes the Bth element of , B denotes a low dimension suitable for scalar quantization.

[0087] The quantization of is performed by the primary element.

[0088] For b elements , the quantization result is calculated by the quantization result calculation formula of the scalar quantizer shown in the following formula:

[0089] (2);

[0090] wherein, , , denotes the number of elements of the finite scalar codebook subset , that is, the equally spaced real numbers between -1 and 1 contained in .

[0091] Therefore, the untrainable codebook of SQ is , which is composed of B vectors, wherein × denotes the Cartesian product. The quantization result of is calculated by the discrete representation calculation formula of the scalar quantizer shown in formula (3), which is a process of converting multiple non-fixed radix numbers into decimal numbers:

[0092] (3);

[0093] wherein, denotes the discrete representation generated by SQ, denotes the quantization result, denotes the index for multiplication, denotes the number of elements of the finite scalar codebook subset , . It can also be obtained by querying in Finally, goes through a linear layer with trainable weights to obtain the output of SQ, that is, .

[0094] (2) Improved vector quantizer IVQ

[0095] IVQ is used to refine acoustic details on the basis of rough audio profiles to obtain quantization results.

[0096] The goal of IVQ is to refine acoustic details. It is assumed that IVQ is applied to the input vector discretization, the output quantization result .

[0097] First goes through a linear layer with trainable weights to get , where denotes the transformed input vector, denotes the dimension of the code vector.

[0098] Given a trainable codebook , where K denotes the number of code vectors in the codebook, denotes the code vector in the codebook, , denotes a set of vectors with dimension M and real elements.

[0099] The quantization result and the discrete representation are obtained by selecting the code vector closest to the Euclidean distance, that is:

[0100] (4);

[0101] In the formula, denotes the quantization result of IVQ, denotes the discrete representation generated by IVQ.

[0102] Finally goes through a linear layer with trainable weights to get the output of IVQ , that is .

[0103] It should be noted that each improved vector quantizer in the present application is obtained after introducing an online clustering training strategy and a load balancing loss in the original vector quantizer to improve the utilization efficiency of the codebook.

[0104] The structure of the causal encoder and the causal decoder in the present application is basically mirror image, aiming to realize low-delay inference and real-time efficient generation. The main difference between the causal encoder and the causal decoder is that the causal encoder is a causal down-sampling convolutional layer, and the causal decoder is a causal up-sampling convolutional layer.

[0105] Specifically, referring to Figure 4 , the structure of the causal decoder disclosed in the embodiments of the present application is shown in the schematic diagram, and the causal decoder 30 comprises:

[0106] The second causal input convolutional layer, the causal up-sampling convolutional layer, the second linear layer, the second causal improved ConvNeXt v2 module set and the second causal output convolutional layer are connected in sequence.

[0107] Specifically,

[0108] The second causal input convolutional layer is configured to restore the input quantization result to the original dimension;

[0109] The causal up-sampling convolutional layer is configured to perform feature up-sampling on the original dimension quantization result to obtain an up-sampled result, and input the up-sampled result to the second linear layer;

[0110] The second linear layer is configured to perform linear transformation on the up-sampled result to obtain a linear transformation result, and input the linear transformation result to the second causal improved ConvNeXt v2 module set;

[0111] The second causal improved ConvNeXt v2 module set is configured to perform deep processing on the linear transformation result to obtain a deep processing result;

[0112] The second causal output convolutional layer is configured to obtain the MDCT spectrum from the deep processing result, so as to restore the MDCT spectrum to the audio waveform through inverse MDCT.

[0113] In the present application, the causal decoder takes the quantization result of the residual scalar vector quantizer as input, decodes the MDCT spectrum, and finally reconstructs the audio waveform through inverse MDCT. The structure of the causal decoder is roughly mirrored with that of the causal encoder. The quantization result is first restored to the original dimension through the second causal input convolutional layer, and then sequentially passes through the causal up-sampling convolutional layer, the second linear layer, the second causal improved ConvNeXt v2 module set and the second causal output convolutional layer to obtain the decoded MDCT spectrum. Through inverse MDCT processing of the MDCT spectrum, the audio waveform is obtained.

[0114] In practical applications, the number of causal improved ConvNeXt v2 modules included in the second causal improved ConvNeXt v2 module set is determined according to actual needs, for example, 8 (see Figure 4 ), which is not limited in the present application.

[0115] In combination with Figures 2-4 , see Figure 5 , the structure of an audio processing model disclosed in an embodiment of the present application is shown in the figure, which includes: Figure 2 the causal encoder 10 in the embodiment shown, Figure 3 the residual scalar vector quantizer 20 in the embodiment shown, and Figure 4 the causal decoder 30 in the embodiment shown.

[0116] The training criteria of the audio processing model are as follows:

[0117] (1) Loss function

[0118] The audio processing model in the application is a streaming neural network audio codec, and the training of the audio processing model basically follows the training mode of the MDCTCodec, mainly adopting a training strategy based on a generative adversarial network. A multi-resolution MDCT spectrum discriminator and a generator are adopted for adversarial training.

[0119] The multi-resolution MDCT spectrum discriminator is composed of three parallel sub discriminators Each sub discriminator takes the extracted MDCT spectrum in the natural waveform or the decoded waveform as input, but the MDCT configuration used for extracting the MDCT spectrum is different, so that the discriminator can judge the multi-resolution MDCT spectrum. The input of each sub discriminator is first processed by five cascaded blocks, each block consisting of a two-dimensional convolution layer and a LeakyReLU activation function. Finally, a single-channel two-dimensional convolution layer outputs the judgment result.

[0120] The application adopts a hinge form of adversarial loss function, and for the generator and the discriminator, the adversarial loss is defined as follows:

[0121] The expression of the adversarial loss function of the generator is shown in formula (5):

[0122] (5);

[0123] In the formula, represents the mean value, represents the output result of the decoder output speech after passing through the i-th discriminator, represents the decoder output speech.

[0124] The expression of the adversarial loss function of the discriminator is shown in formula (6):

[0125] (6);

[0126] In the formula, represents the mean value, represents the output result of the real speech passing through the i-th discriminator, represents the real speech.

[0127] In addition, the application also uses a feature matching loss function .

[0128] In addition to the loss function based on the generative adversarial network, the application also adopts a spectrum level loss function and a quantization loss function. The spectrum level loss function includes an MDCT spectrum loss function and a mel-spectrogram loss function defined between the decoded waveform and the natural waveform .

[0129] where MDCT-spectrogram loss function is defined as the mean square error between MDCT-spectrograms and mel-spectrogram loss function is defined as the mean square error and mean absolute error between mel-spectrograms .

[0130] Quantization loss includes commitment loss function and codebook loss , both of which are calculated as the mean square error between the input and output of the quantizer and the mean square error between the input and output of the sub-quantizer in the quantizer. But the former is only used to update the parameters of the encoder, while the latter only updates the parameters of the quantizer.

[0131] Unlike MDCTCodec, IVQ used in the present application introduces some improvement strategies in the training layer to improve the utilization rate of the codebook. First, IVQ adopts an online codebook clustering strategy. At each training step, IVQ forces to reactivate the code vectors with low usage frequency and cluster them to the quantized feature space. Further, in order to avoid some code vectors with too high or too low usage frequency, IVQ introduces a load balancing loss function , which is defined as the cross entropy between the posterior code distribution of IVQ and the prior uniform distribution . Wherein, the posterior code is approximated by the frequency of each code being selected in the training process.

[0132] (2) Training process

[0133] The audio processing model disclosed in the present application adopts a generative adversarial training mode, and alternately trains the generator and the discriminator using the generator loss function and the discriminator loss function in the training process;

[0134] wherein the generator loss function is a linear combination of the adversarial loss function, the feature matching loss function, the MDCT-spectrogram loss function, the mel-spectrogram loss function, the codebook loss function, the commitment loss function and the load balancing loss function according to a preset proportion.

[0135] Specifically, the audio processing model disclosed in the present application adopts a standard generative adversarial training mode, and the generator loss function is a linear combination of all the above loss functions according to a preset proportion, that is, the expression of the generator loss function is as follows:

[0136] (7);

[0137] wherein, represents a generator's adversarial loss function, represents a feature matching loss function, represents an MDCT spectrum loss function, represents a mel-spectrum loss function, represents a codebook loss function, represents a commitment loss function, represents a load balancing loss function.

[0138] all represent hyperparameters, and specific values are determined according to actual needs, which are not limited herein.

[0139] In the training process of the audio processing model, the generator loss function and the discriminator loss function are used to alternately train the generator and the discriminator until the audio processing model converges.

[0140] For the process of processing the audio waveform by using the audio processing model, an example is illustrated as follows:

[0141] Taking voice communication as an example, the practical application of the audio processing model (i.e., the streaming neural network audio codec) proposed by the present application is introduced. In voice communication, the sending end encodes the voice signal and sends it to the receiving end, and the receiving end decodes to obtain the voice signal.

[0142] By using the audio processing model proposed by the present application, the sending end first extracts the MDCT spectrum from the to-be-sent voice signal waveform through MDCT, and inputs the MDCT spectrum into the causal encoder to generate continuous codes. Then, the residual scalar vector quantizer discretizes the continuous codes and generates discrete representations , wherein the superscripts i and j represent the i th SQ and the j th IVQ, respectively. The discrete representation is transmitted in binary form to the receiving end. After receiving the discrete representation, the receiving end generates quantized codes according to the codebook and inputs them into the causal decoder to generate the MDCT spectrum, and finally restores the audio waveform through inverse MDCT, thereby realizing the complete voice communication process.

[0143] Corresponding to the method embodiments described above, the present application also discloses an audio processing device.

[0144] Referring to Figure 6 , the audio processing device disclosed by the embodiments of the present application can include:

[0145] The acquisition unit 201 is configured to acquire an audio waveform.

[0146] In actual application, when audio processing is needed, a real-time audio waveform can be acquired.

[0147] The processing unit 202 is configured to input the audio waveform into an audio processing model of the audio waveform input configuration, encode a discrete cosine transform (MDCT) spectrum of the audio waveform into a continuous code by a causal encoder in the audio processing model, input the continuous code into a residual scalar vector quantizer for quantization processing to obtain a quantization result, decode the MDCT spectrum out of the quantization result by a causal decoder, and restore the MDCT spectrum into the audio waveform by inverse MDCT.

[0148] In audio coding, MDCT can convert a time-domain signal into a frequency-domain signal. At the same time, MDCT can effectively remove the periodic noise caused by edge effects, improving the accuracy of signal processing. And in the case of the same encoding rate, MDCT can achieve higher compression ratio and better signal quality than DCT.

[0149] Based on this, the present application takes the MDCT spectrum of the audio waveform as the parameter feature of the audio waveform, encodes the MDCT spectrum by a causal encoder in the audio processing model, quantizes the MDCT spectrum by a residual scalar vector quantizer, decodes the MDCT spectrum by a causal decoder, and finally restores the MDCT spectrum by inverse MDCT to obtain the original audio waveform.

[0150] In the present application, the audio processing model includes a causal encoder, a residual scalar vector quantizer, and a causal decoder. The causal encoder encodes the MDCT spectrum of the audio waveform into a continuous code. The residual scalar vector quantizer mainly discretizes the continuous code when quantizing the continuous code, generates a corresponding discrete representation, and generates a quantization code as a quantization result. The causal decoder takes the quantization code as input, decodes the MDCT spectrum of the audio waveform from the quantization code, and restores the MDCT spectrum into the audio waveform by inverse MDCT.

[0151] It should be noted that the audio processing model is a streaming neural network audio codec that follows the MDCTCodec basic framework and uses a causal model structure (causal encoder and causal decoder) to achieve low-latency streaming inference of audio. At the same time, in order to make up for the decline in coding quality caused by the causal structure, the audio processing module also introduces a new residual scalar vector quantization strategy, so that the audio processing model can maintain the encoding quality, efficiency, and complexity of MDCTCodec, achieve low-latency streaming inference of audio, and meet the use requirements of real-time communication scenarios.

[0152] The causal model structure adopted in the present application can greatly reduce the delay, i.e., the amount of input required for model startup, compared with the traditional non-causal model structure. The reason is that the non-causal model needs to input future information, so it has a higher delay, for example, when predicting the result at time t, the model at least needs to use the input information at time t+t0, and then the delay time is t0. Using the full causal model of the present application, no future information needs to be input, that is, t0=0, so the delay can be greatly reduced.

[0153] In addition, the decoding audio quality of the audio processing model proposed in the present application is comparable to that of the advanced non-streaming audio codec at the present stage, and has the characteristics of low delay, high efficiency and small amount of model parameters, and is suitable for application in real-time communication.

[0154] In summary, the present application discloses an audio processing device, which acquires an audio waveform, inputs the audio waveform into a configured audio processing model, encodes the MDCT spectrum of the audio waveform into a continuous code through a causal encoder in the audio processing model, and inputs the continuous code into a residual scalar vector quantizer for quantization processing to obtain a quantization result. The quantization result is input into a causal decoder to decode the MDCT spectrum, and the MDCT spectrum is restored to the audio waveform through inverse MDCT. The present application uses the MDCT spectrum of the audio waveform as the parameter feature of the audio waveform and uses it for encoding, quantization and decoding, achieving the improvement of frequency domain resolution while reducing the computational complexity in the decoding stage. In addition, the causal model structure used in the audio processing model does not need to input any future information compared with the traditional non-causal model structure, so the delay can be greatly reduced, the audio processing efficiency is improved, and the use demand of real-time communication scene is met.

[0155] In one embodiment, the causal encoder can include:

[0156] a first causal input convolutional layer, a first causal improved ConvNeXt v2 module set, a first linear layer, a causal down-sampling convolutional layer and a first causal output convolutional layer connected in sequence;

[0157] The first causal input convolutional layer is configured to perform preliminary processing of dimension transformation on the input MDCT spectrum and input the preliminary processing result to the first causal improved ConvNeXt v2 module set;

[0158] The first causal improved ConvNeXt v2 module set is configured to perform deep processing on the preliminary processing result to obtain a deep processing result, and input the deep processing result to the first linear layer;

[0159] The first linear layer is configured to perform linear transformation on the deep processing result to obtain a linear transformation result, and input the linear transformation result to the causal down-sampling convolutional layer;

[0160] The causal down-sampling convolutional layer is configured to perform feature down-sampling on the linear variation result to obtain a down-sampled result, and input the down-sampled result into the first causal output convolutional layer.

[0161] The first causal output convolutional layer is configured to perform dimension transformation processing on the down-sampled result to obtain the continuous code.

[0162] In one embodiment, the set of causal improved ConvNeXt v2 modules includes a plurality of identical causal improved ConvNeXt v2 modules.

[0163] Each of the causal improved ConvNeXt v2 modules is obtained by causally improving a ConvNeXt v2 module in an APCodec and an MDCTCodec.

[0164] In one embodiment, in each of the causal improved ConvNeXt v2 modules, the output result obtained by sequentially passing the preliminary processing result input by the first causal input convolutional layer through a deep layer causal convolutional layer, a linear layer for mapping features to a higher dimension, a Gaussian error linear unit, and a linear layer for mapping features to the original low dimension, and adding the total result after adding the preliminary processing result as input, is the final output of the causal improved ConvNeXt v2 module.

[0165] In one embodiment, the residual scalar vector quantizer includes:

[0166] A set of scalar quantizers and a set of improved vector quantizers, the set of scalar quantizers and the set of improved vector quantizers being connected in a residual manner, the plurality of scalar quantizers in the set of scalar quantizers being connected in a residual manner, and the plurality of improved vector quantizers in the set of improved vector quantizers being connected in a residual manner.

[0167] The scalar quantizer is configured to process the continuous code input by the causal encoder by taking rounding as the quantization principle to construct a rough audio profile.

[0168] The improved vector quantizer is configured to refine acoustic details on the basis of the rough audio profile to obtain the quantization result.

[0169] In one embodiment, each of the improved vector quantizers is obtained by introducing an online clustering training strategy and a load balancing loss on the basis of an original vector quantizer.

[0170] In one embodiment, the causal decoder includes:

[0171] The second causal input convolutional layer is configured to restore the input quantization result to an original dimension.

[0172] The second causal input convolutional layer is configured to restore the input quantization result to an original dimension.

[0173] The causal up-sampling convolutional layer is configured to perform feature up-sampling on the original dimension quantization result to obtain an up-sampled result, and input the up-sampled result to the second linear layer.

[0174] The second linear layer is configured to perform linear transformation on the up-sampled result to obtain a linear transformation result, and input the linear transformation result to the second causal improved ConvNeXt v2 module set.

[0175] The second causal improved ConvNeXt v2 module set is configured to perform deep processing on the linear transformation result to obtain a deep processing result.

[0176] The second causal output convolutional layer is configured to obtain the MDCT spectrum from the deep processing result, so as to restore the MDCT spectrum to the audio waveform through inverse MDCT.

[0177] In one embodiment, the audio processing model adopts a generative adversarial training mode, and generator and discriminator are alternately trained using a generator loss function and a discriminator loss function during training until the audio processing model converges.

[0178] The generator loss function is a linear combination of the adversarial loss function, the feature matching loss function, the MDCT spectrum loss function, the mel spectrum loss function, the codebook loss function, the commitment loss function and the load balancing loss function of the generator according to a preset ratio.

[0179] It should be noted that the specific working principles of each component in the device embodiment can be referred to the corresponding part of the method embodiment, which will not be repeated here.

[0180] Corresponding to the above-mentioned embodiments, the present application further discloses a computer storage medium, characterized in that the computer storage medium stores at least one instruction, which is executed by a processor to realize the steps shown in the audio processing method embodiment.

[0181] The computer storage media can be a tangible media that can include or store the programs for use by or in connection with an instruction execution system, apparatus, or device. The computer storage media can be a machine-readable signal media or a machine-readable storage media. The computer storage media can include, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage media will include one or more of the following: an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0182] Corresponding to the above embodiments, as shown in Figure 7 The present application also provides an electronic device, which can include a processor 1 and a memory 2.

[0183] The processor 1 and the memory 2 can complete the communication between each other through a communication bus 3.

[0184] The processor 1 is configured to execute at least one instruction.

[0185] The memory 2 is configured to store at least one instruction.

[0186] The processor 1 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0187] The memory 2 can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.

[0188] The processor executes at least one instruction to implement the steps shown in the audio processing method embodiments.

[0189] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and are not intended to denote the presence of any such actual relationship or order. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0190] The various embodiments in the specification are described with progression in this order of description. Embodiments having the same or similar descriptions are referenced by the same reference numerals, and an overlapping description is not repeated.

[0191] The above description of disclosed embodiments provides enabling disclosure sufficient for one of ordinary skill in the art to practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An audio processing method, characterized in that, include: Obtain the audio waveform; The audio waveform is input into the configured audio processing model. The discrete cosine transform (MDCT) spectrum of the audio waveform is encoded into a continuous code by the causal encoder in the audio processing model. The continuous code is input into the residual scalar vector quantizer for quantization processing to obtain the quantization result. The quantization result is input into the causal decoder to decode the MDCT spectrum. The MDCT spectrum is then used to recover the audio waveform through inverse MDCT. The residual scalar vector quantizer includes: A scalar quantizer set and an improved vector quantizer set are connected in a residual manner. Multiple scalar quantizers in the scalar quantizer set are connected in a residual manner, and multiple improved vector quantizers in the improved vector quantizer set are connected in a residual manner. The scalar quantizer is used to process the continuous code input by the causal encoder using rounding as the quantization principle to construct a rough audio profile. The improved vector quantizer is used to refine the acoustic details based on the coarse audio profile to obtain the quantization result; The causal encoder includes: The first causal input convolutional layer, the first causal improved ConvNeXt v2 module set, the first linear layer, the causal downsampling convolutional layer and the first causal output convolutional layer are sequentially connected in series; The first causal input convolutional layer is used to perform preliminary processing of dimensionality transformation on the input MDCT spectrum, and inputs the preliminary processing result into the first causal improved ConvNeXt v2 module set; The first causal improvement ConvNeXt v2 module set is used to perform deep processing on the preliminary processing result to obtain a deep processing result, and input the deep processing result into the first linear layer; The first linear layer is used to perform a linear transformation on the depth processing result to obtain a linear transformation result, and input the linear transformation result into the causal downsampling convolutional layer; The causal downsampling convolutional layer is used to perform feature downsampling on the linear change result to obtain a downsampling result, and the downsampling result is input into the first causal output convolutional layer; The first causal output convolutional layer is used to perform dimensionality transformation on the downsampling result to obtain the continuous code; The causal decoder includes: The second causal input convolutional layer, the causal upsampling convolutional layer, the second linear layer, the second causal improved ConvNeXt v2 module set, and the second causal output convolutional layer are sequentially connected in series. The second causal input convolutional layer is used to restore the quantized result of the input to its original dimension; The causal upsampling convolutional layer is used to upsample the original dimension quantization result to obtain the upsampling result, and the upsampling result is input into the second linear layer; The second linear layer is used to perform a linear transformation on the upsampling result to obtain a linear transformation result, and inputs the linear transformation result into the second causal improved ConvNeXt v2 module set; The second causal improved ConvNeXt v2 module set is used to perform deep processing on the linear transformation result to obtain a deep processing result; The second causal output convolutional layer is used to obtain the MDCT spectrum from the depth processing result, so as to recover the audio waveform from the MDCT spectrum through inverse MDCT; The difference between the causal encoder and the causal decoder is that the causal encoder is a causal downsampling convolutional layer, while the causal decoder is a causal upsampling convolutional layer.

2. The audio processing method according to claim 1, characterized in that, The causal improvement ConvNeXt v2 module set includes: multiple identical causal improvement ConvNeXt v2 modules; Each of the aforementioned causal improved ConvNeXt v2 modules is obtained by causally transforming the ConvNeXtv2 modules proposed in APCodec and MDCTCodec.

3. The audio processing method according to claim 2, characterized in that, In each of the causal improvement ConvNeXtv2 modules, the preliminary processing result of the first causal input convolutional layer is sequentially passed through a deep-level causal convolutional layer, a linear layer that maps features to a higher dimension, a Gaussian error linear unit, and a linear layer that maps features to the original low dimension. The output result obtained by adding the preliminary processing result as input is the final output of the causal improvement ConvNeXt v2 module.

4. The audio processing method according to claim 1, characterized in that, Each of the improved vector quantizers is obtained by introducing a codebook online clustering training strategy and a load balancing loss on the basis of the original vector quantizer.

5. The audio processing method according to claim 1, characterized in that, The audio processing model adopts a generative adversarial training mode, and during the training process, the generator loss function and the discriminator loss function are used to train the generator and the discriminator alternately until the audio processing model converges. The generator loss function is a linear combination of the generator's adversarial loss function, feature matching loss function, MDCT spectrum loss function, Mel spectrum loss function, codebook loss function, commitment loss function, and load balancing loss function in a preset ratio.

6. An audio processing apparatus, characterized in that, include: The acquisition unit is used to acquire audio waveforms; The processing unit is used to input the audio waveform into a configured audio processing model, encode the discrete cosine transform (MDCT) spectrum of the audio waveform into a continuous code through a causal encoder in the audio processing model, input the continuous code into a residual scalar vector quantizer for quantization processing to obtain a quantization result, input the quantization result into a causal decoder to decode the MDCT spectrum, and recover the audio waveform from the MDCT spectrum through inverse MDCT. The residual scalar vector quantizer includes: A scalar quantizer set and an improved vector quantizer set are connected in a residual manner. Multiple scalar quantizers in the scalar quantizer set are connected in a residual manner, and multiple improved vector quantizers in the improved vector quantizer set are connected in a residual manner. The scalar quantizer is used to process the continuous code input by the causal encoder using rounding as the quantization principle to construct a rough audio profile. The improved vector quantizer is used to refine the acoustic details based on the coarse audio profile to obtain the quantization result; The causal encoder includes: The first causal input convolutional layer, the first causal improved ConvNeXt v2 module set, the first linear layer, the causal downsampling convolutional layer and the first causal output convolutional layer are sequentially connected in series; The first causal input convolutional layer is used to perform preliminary processing of dimensionality transformation on the input MDCT spectrum, and inputs the preliminary processing result into the first causal improved ConvNeXt v2 module set; The first causal improvement ConvNeXt v2 module set is used to perform deep processing on the preliminary processing result to obtain a deep processing result, and input the deep processing result into the first linear layer; The first linear layer is used to perform a linear transformation on the depth processing result to obtain a linear transformation result, and input the linear transformation result into the causal downsampling convolutional layer; The causal downsampling convolutional layer is used to perform feature downsampling on the linear change result to obtain a downsampling result, and the downsampling result is input into the first causal output convolutional layer; The first causal output convolutional layer is used to perform dimensionality transformation on the downsampling result to obtain the continuous code; The causal decoder includes: The second causal input convolutional layer, the causal upsampling convolutional layer, the second linear layer, the second causal improved ConvNeXt v2 module set, and the second causal output convolutional layer are sequentially connected in series. The second causal input convolutional layer is used to restore the quantized result of the input to its original dimension; The causal upsampling convolutional layer is used to upsample the original dimension quantization result to obtain the upsampling result, and the upsampling result is input into the second linear layer; The second linear layer is used to perform a linear transformation on the upsampling result to obtain a linear transformation result, and inputs the linear transformation result into the second causal improved ConvNeXt v2 module set; The second causal improved ConvNeXt v2 module set is used to perform deep processing on the linear transformation result to obtain a deep processing result; The second causal output convolutional layer is used to obtain the MDCT spectrum from the depth processing result, so as to recover the audio waveform from the MDCT spectrum through inverse MDCT; The difference between the causal encoder and the causal decoder is that the causal encoder is a causal downsampling convolutional layer, while the causal decoder is a causal upsampling convolutional layer.

7. An electronic device, characterized in that, The electronic device includes: a memory and a processor; The memory is used to store at least one instruction; The processor is used to execute the at least one instruction to implement the audio processing method as described in any one of claims 1 to 5.

Images