Audio processing method and device and electronic equipment

Abstract

The invention discloses an audio processing method and device and electronic equipment, and relates to the field of audio processing, and the method comprises the steps: obtaining an audio waveform, inputting the audio waveform into a configured audio processing model, coding an MDCT spectrum of the audio waveform into a continuous code through a causal encoder in the audio processing model, and inputting the audio waveform into a residual scalar vector quantizer for quantization processing to obtain a quantization result, inputting the quantization result into a causal decoder to decode an MDCT spectrum, and recovering the MDCT spectrum into the audio waveform through inverse MDCT. According to the invention, the MDCT spectrum of the audio waveform is used as the parameter feature of the audio waveform and is used for coding, quantification and decoding, the frequency domain resolution is improved, the calculation complexity of the decoding stage is reduced, and compared with a traditional non-causal model structure, the causal model structure used in the audio processing model does not need to input any future information. Therefore, the delay can be greatly reduced, the audio processing efficiency is improved, and the use requirement of a real-time communication scene is met.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and more specifically, to an audio processing method, device and electronic equipment. Background Art

[0002] Audio codec is an important research topic in audio processing. It aims to discretize audio with the least bit usage while maintaining the highest decoded audio quality. Therefore, it plays an important role in various fields, such as real-time communication and large speech models. With the development of deep learning, neural network audio codecs have been proposed and have surpassed traditional methods such as Opus and EVS in performance. Neural network audio codecs generally consist of encoders, residual vector quantization (RVQ) and decoders. According to the type of encoding object, neural network audio codecs can be divided into two categories: one is the neural network audio codec based on waveform coding, such as SoundStream, Encodec, HiFi-Codec, AudioDec and DAC, which discretize the audio waveform directly in the time domain. However, this method requires hundreds of upsampling and downsampling operations, which will lead to high model complexity and low efficiency, which is not conducive to the application of voice communication scenarios. The other is the neural network audio codec based on spectral coding. For example, APCodec uses a two-way structure to encode, quantize and decode the audio amplitude and phase spectrum, thereby improving generation efficiency and reducing model complexity.

[0003] Although the research on existing neural network audio codecs has made significant progress, there are still some problems. Most of the existing research on neural network audio codecs focuses on non-causal structure design, which leads to high latency in the audio processing process and does not meet the requirements of real-time communication scenarios. Summary of the invention

[0004] In view of this, the present invention discloses an audio processing method, device and electronic device to significantly reduce the delay during audio processing and improve the audio processing efficiency, thereby meeting the usage requirements of real-time communication scenarios.

[0005] An audio processing method, comprising:

[0006] Get the audio waveform;

[0007] The audio waveform is input into the configured audio processing model, the discrete cosine transform MDCT spectrum of the audio waveform is encoded into a continuous code through the causal encoder in the audio processing model, the continuous code is input into the residual scalar vector quantizer for quantization processing to obtain a quantization result, the quantization result is input into the causal decoder to decode the MDCT spectrum, and the MDCT spectrum is restored to the audio waveform through inverse MDCT.

[0008] Optionally, the causal encoder comprises:

[0009] The first causal input convolutional layer, the first causal improved ConvNeXt v2 module set, the first linear layer, the causal downsampling convolutional layer and the first causal output convolutional layer are connected in series;

[0010] The first causal input convolution layer is used to perform preliminary processing of dimensional transformation on the input MDCT spectrum, and input the preliminary processing result into the first causal improved ConvNeXt v2 module set;

[0011] The first causal improved ConvNeXt v2 module set is used to perform deep processing on the preliminary processing result to obtain a deep processing result, and input the deep processing result into the first linear layer;

[0012] The first linear layer is used to perform a linear transformation on the depth processing result to obtain a linear transformation result, and input the linear transformation result into the causal downsampling convolution layer;

[0013] The causal downsampling convolution layer is used to perform feature downsampling on the linear change result to obtain a downsampling result, and input the downsampling result into the first causal output convolution layer;

[0014] The first causal output convolution layer is used to perform dimension transformation processing on the downsampling result to obtain the continuous code.

[0015] Optionally, the causal improved ConvNeXt v2 module set includes: a plurality of identical causal improved ConvNeXtv2 modules;

[0016] Each of the causal improved ConvNeXt v2 modules is obtained by causally modifying the ConvNeXt v2 modules proposed in APCodec and MDCTCodec.

[0017] Optionally, in each of the causal improved ConvNeXt v2 modules, the output result of the preliminary processing result input by the first causal input convolutional layer is obtained by successively passing through a depth-level causal convolutional layer, a linear layer that maps features to a higher dimension, a Gaussian error linear unit, and a linear layer that maps features to the original low dimension, and the total result after adding the preliminary processing result as input is used as the final output of the causal improved ConvNeXt v2 module.

[0018] Optionally, the residual scalar vector quantizer comprises:

[0019] A scalar quantizer set and an improved vector quantizer set, wherein the scalar quantizer set and the improved vector quantizer set are connected in a residual manner, a plurality of scalar quantizers in the scalar quantizer set are connected in a residual manner, and a plurality of improved vector quantizers in the improved vector quantizer set are connected in a residual manner;

[0020] The scalar quantizer is used to process the continuous code input by the causal encoder by taking rounding as the quantization principle to construct a rough audio profile;

[0021] The improved vector quantizer is used to refine the acoustic details based on the rough audio profile to obtain the quantization result.

[0022] Optionally, each of the improved vector quantizers is obtained by introducing a codebook online clustering training strategy and load balancing loss on the basis of an original vector quantizer.

[0023] Optionally, the causal decoder comprises:

[0024] The second causal input convolution layer, the causal upsampling convolution layer, the second linear layer, the second causal improved ConvNeXt v2 module set and the second causal output convolution layer are sequentially connected in series;

[0025] The second causal input convolution layer is used to restore the input quantization result to the original dimension;

[0026] The causal upsampling convolution layer is used to perform feature upsampling on the original dimension quantization result to obtain an upsampling result, and input the upsampling result into the second linear layer;

[0027] The second linear layer is used to perform a linear transformation on the upsampling result to obtain a linear transformation result, and input the linear transformation result into the second causal improved ConvNeXt v2 module set;

[0028] The second causal improved ConvNeXt v2 module set is used to perform deep processing on the linear transformation result to obtain a deep processing result;

[0029] The second causal output convolution layer is used to obtain the MDCT spectrum from the depth processing result, so as to restore the MDCT spectrum to the audio waveform through inverse MDCT.

[0030] Optionally, the audio processing model adopts a generative adversarial training mode, and during the training process, the generator and the discriminator are alternately trained using a generator loss function and a discriminator loss function until the audio processing model converges;

[0031] Among them, the generator loss function is a linear combination of the generator's adversarial loss function, feature matching loss function, MDCT spectrum loss function, Mel spectrum loss function, codebook loss function, commitment loss function and load balancing loss function in a preset proportion.

[0032] An audio processing device, comprising:

[0033] An acquisition unit, used for acquiring an audio waveform;

[0034] A processing unit is used to input the audio waveform into a configured audio processing model, encode the discrete cosine transform MDCT spectrum of the audio waveform into a continuous code through a causal encoder in the audio processing model, input the continuous code into a residual scalar vector quantizer for quantization processing to obtain a quantization result, input the quantization result into a causal decoder to decode the MDCT spectrum, and restore the MDCT spectrum to the audio waveform through inverse MDCT.

[0035] An electronic device, comprising: a memory and a processor;

[0036] The memory is used to store at least one instruction;

[0037] The processor is used to execute the at least one instruction to implement the above-mentioned audio processing method.

[0038] It can be seen from the above technical solutions that the present invention discloses an audio processing method, device and electronic device, which obtain an audio waveform, input the audio waveform into a configured audio processing model, encode the MDCT spectrum of the audio waveform into a continuous code through the causal encoder in the audio processing model, and input it into the residual scalar vector quantizer for quantization processing to obtain a quantization result, input the quantization result into the causal decoder to decode the MDCT spectrum, and restore the MDCT spectrum to an audio waveform through inverse MDCT. The present application uses the MDCT spectrum of the audio waveform as a parameter feature of the audio waveform and uses it for encoding, quantization and decoding, thereby reducing the computational complexity of the decoding stage while improving the frequency domain resolution, and the causal model structure used in the audio processing model does not require the input of any future information compared to the traditional non-causal model structure, so it can greatly reduce latency and improve audio processing efficiency, thereby meeting the use requirements of real-time communication scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without paying creative work.

[0040] Figure 1 A flow chart of an audio processing method disclosed in an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of the structure of a causal encoder disclosed in an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of the structure of a residual scalar vector quantizer disclosed in an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the structure of a causal decoder disclosed in an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of the structure of an audio processing model disclosed in an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of the structure of an audio processing device disclosed in an embodiment of the present invention;

[0046] Figure 7 The present invention is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The embodiment of the present invention discloses an audio processing method, device and electronic device, which obtain an audio waveform, input the audio waveform into a configured audio processing model, encode the MDCT spectrum of the audio waveform into a continuous code through a causal encoder in the audio processing model, and input it into a residual scalar vector quantizer for quantization processing to obtain a quantization result, input the quantization result into a causal decoder to decode the MDCT spectrum, and restore the MDCT spectrum to an audio waveform through inverse MDCT. The present application uses the MDCT spectrum of the audio waveform as a parameter feature of the audio waveform and uses it for encoding, quantization and decoding, thereby improving the frequency domain resolution while reducing the computational complexity of the decoding stage, and the causal model structure used in the audio processing model does not need to input any future information compared to the traditional non-causal model structure, so it can greatly reduce the delay and improve the audio processing efficiency, thereby meeting the use requirements of real-time communication scenarios.

[0049] See also Figure 1 , a flow chart of an audio processing method disclosed in an embodiment of the present application, the method may include:

[0050] Step S101, obtaining an audio waveform.

[0051] In practical applications, when audio processing is required, real-time audio waveform can be obtained.

[0052] Step S102, input the audio waveform into the configured audio processing model, encode the MDCT spectrum of the audio waveform into a continuous code through the causal encoder in the audio processing model, input the continuous code into the residual scalar vector quantizer for quantization to obtain a quantization result, input the quantization result into the causal decoder to decode the MDCT spectrum, and restore the MDCT spectrum to the audio waveform through inverse MDCT.

[0053] The quantization result obtained by the residual scalar vector quantizer performing quantization processing on the input continuous code may specifically be a quantization code.

[0054] MDCT (Modified Discrete Cosine Transform) is an improved version of Discrete Cosine Transform (DCT), mainly used for transform coding of speech, broadband audio and image signals.

[0055] In audio coding and decoding, MDCT can convert time domain signals into frequency domain signals. At the same time, MDCT can effectively remove periodic noise caused by edge effects and improve the accuracy of signal processing. And at the same coding rate, MDCT can achieve higher compression ratio and better signal quality than DCT.

[0056] Based on this, the present application uses the MDCT spectrum of the audio waveform as the parameter feature of the audio waveform, obtains the MDCT spectrum through encoding by a causal encoder in the audio processing model, quantization by a residual scalar vector quantizer, and decoding by a causal decoder, and finally restores the MDCT spectrum through inverse MDCT to obtain the original audio waveform.

[0057] The audio processing model in the present application is composed of a causal encoder, a residual scalar vector quantizer and a causal decoder. The causal encoder encodes the MDCT spectrum of the audio waveform into a continuous code. When the residual scalar vector quantizer performs quantization processing on the continuous code, it mainly discretizes the continuous code, generates the corresponding discrete representation and generates a quantization code as the quantization result. The causal decoder takes the quantization code as input, decodes the MDCT spectrum of the audio waveform from the quantization code, and restores the inverse MDCT to the audio waveform through the inverse MDCT.

[0058] It should be noted that the audio processing model is a streaming neural network audio codec, which follows the basic framework of MDCTCodec and adopts a causal model structure (causal encoder and causal decoder) to achieve low-latency streaming inference of audio. At the same time, in order to compensate for the degradation of encoding quality caused by the causal structure, the audio processing module also introduces a new residual scalar vector quantization strategy, which enables the audio processing model to achieve low-latency streaming inference of audio while maintaining the encoding quality, efficiency and complexity of MDCTCodec, thereby meeting the use requirements of real-time communication scenarios.

[0059] The causal model structure used in this application can significantly reduce the delay compared to the traditional non-causal model structure. The delay refers to the amount of input required for the model to start. The reason is: the non-causal model needs to input future information, so it has a higher delay. For example, when predicting the result at time t, the model needs to use at least the input information at time t+t0, so the delay time is t0. Using the full causal model of this application, there is no need to input any future information, that is, t0=0, so the delay can be greatly reduced.

[0060] In addition, the decoded audio quality of the audio processing model proposed in this application is comparable to that of the current advanced non-streaming audio codecs, and has the characteristics of low latency, high efficiency and a small number of model parameters, making it suitable for application in real-time communications.

[0061] In summary, the present application discloses an audio processing method, which obtains an audio waveform, inputs the audio waveform into a configured audio processing model, encodes the MDCT spectrum of the audio waveform into a continuous code through a causal encoder in the audio processing model, and inputs it into a residual scalar vector quantizer for quantization processing to obtain a quantization result, inputs the quantization result into a causal decoder to decode the MDCT spectrum, and restores the MDCT spectrum to an audio waveform through inverse MDCT. The present application uses the MDCT spectrum of the audio waveform as a parameter feature of the audio waveform and uses it for encoding, quantization and decoding, thereby improving the frequency domain resolution while reducing the computational complexity of the decoding stage, and the causal model structure used in the audio processing model does not require the input of any future information compared to the traditional non-causal model structure, so it can greatly reduce latency and improve audio processing efficiency, thereby meeting the use requirements of real-time communication scenarios.

[0062] For details, see Figure 2 , a schematic diagram of the structure of a causal encoder disclosed in an embodiment of the present application, the causal encoder 10 comprises:

[0063] The first causal input convolutional layer, the first causal improved ConvNeXt v2 module set, the first linear layer, the causal downsampling convolutional layer and the first causal output convolutional layer are connected in series.

[0064] Among them, the first causal improved ConvNeXt v2 module set includes: multiple identical causal improved ConvNeXt v2 modules.

[0065] In practical applications, the number of causal improved ConvNeXt v2 modules included in the first causal improved ConvNeXt v2 module set is determined according to actual needs, for example, 8 (see Figure 2 ), this application is not limited here.

[0066] Each of the causal improved ConvNeXt v2 modules is obtained by causally modifying the ConvNeXt v2 modules proposed in APCodec and MDCTCodec.

[0067] The specific working principle of each component in the causal encoder 10 is as follows:

[0068] The first causal input convolutional layer is used to perform preliminary processing of dimensional transformation on the input MDCT spectrum, and input the preliminary processing result into the first causal improved ConvNeXt v2 module set.

[0069] The first causal improved ConvNeXt v2 module set is used to perform deep processing on the preliminary processing result to obtain a deep processing result, and input the deep processing result into the first linear layer.

[0070] The first linear layer is used to perform a linear transformation on the depth processing result to obtain a linear transformation result, and input the linear transformation result into the causal downsampling convolution layer.

[0071] The causal downsampling convolution layer is used to perform feature downsampling on the linear change result to obtain a downsampling result, and input the downsampling result to the first causal output convolution layer.

[0072] The first causal output convolution layer is used to perform dimension transformation processing on the downsampling result to obtain the continuous code.

[0073] The causal downsampling convolution layer reduces the temporal resolution by downsampling the linear variation results. Finally, the first causal output convolution layer further reduces the feature dimension by transforming the downsampling results to generate a continuous code as the input of the residual scalar vector quantizer.

[0074] In each of the causal improved ConvNeXt v2 modules, the preliminary processing result input by the first causal input convolution layer is sequentially passed through a depth-wise causal convolution layer, a linear layer that maps features to a higher dimension, a Gaussian error linear unit (GELU), and an output result obtained by mapping features to the original low dimension linear layer. The total result after adding the preliminary processing result as input is used as the final output of the causal improved ConvNeXt v2 module.

[0075] In order to improve the utilization rate of the audio processing model in the codebook application and compensate for the audio quality loss caused by structural causality, this application also adds a new residual scalar vector quantization strategy in the audio processing model. This strategy sequentially connects the scalar quantizer and the improved vector quantizer in a residual manner. The scalar quantizer is used to construct a rough audio contour, and the improved vector quantizer is used to refine the acoustic details.

[0076] Specifically, see Figure 3 , a structural diagram of a residual scalar vector quantizer disclosed in an embodiment of the present application, the residual scalar vector quantizer 20 includes: a scalar quantizer set and an improved vector quantizer set, and the scalar quantizer set and the improved vector quantizer set are connected in a residual manner.

[0077] The scalar quantizer set in the present application includes multiple scalar quantizers (SQ), and the multiple scalar quantizers are connected in a residual manner. For example, Figure 3 The scalar quantizer set includes Ns scalar quantizers, namely scalar quantizer 1 to scalar quantizer Ns.

[0078] The improved vector quantizer set includes multiple improved vector quantizers (IVQ), and multiple improved vector quantizers are connected in a residual manner. For example, Figure 3 The improved vector quantizer set includes N v Improved vector quantizers, namely, improved vector quantizer 1 to improved vector quantizer N v .

[0079] The residual scalar vector quantizer 20 is used to quantize the output of the causal encoder 10 Discretize and quantize the results As the input of the causal decoder, The dimension is D , a vector set whose elements are real numbers, that is, all D The set of dimensional real vectors, D is the dimension for encoding and quantizing features.

[0080] The input to the first SQ is , the input of other quantizers (scalar quantizer and improved vector quantizer) is the quantization residual of the previous quantizer. The final quantization result is the sum of all quantizer outputs.

[0081] (1) Scalar Quantizer SQ

[0082] SQ is used to process the continuous codes input by the causal encoder using rounding as the quantization principle to construct a rough audio profile.

[0083] The goal of SQ is to roughly construct the audio profile using rounding as the quantization principle. Assume that SQ has an input vector Discretization, output quantized results .

[0084] first After a trainable weight is The linear layer gets , the expression is as follows:

[0085] (1);

[0086] in, represents the transformed input vector, T represents the vector transpose, express The first element of express The Bth element of , where B represents a low dimension suitable for scalar quantization.

[0087] right The quantization is performed on the principal elements.

[0088] For b elements , quantitative results The quantization result calculation formula of the scalar quantizer is calculated by the following formula:

[0089] (2);

[0090] in, , , represents a finite scalar codebook subset The number of elements of The real numbers between -1 and 1 are equally spaced.

[0091] Therefore, the untrainable codebook of SQ is ,Depend on B is composed of vectors, where × represents the Cartesian product. The quantitative result is The discrete representation generated by SQ is calculated by the discrete representation calculation formula of the scalar quantizer shown in formula (3), which is a process of converting multiple non-fixed base numbers into decimal numbers:

[0092] (3);

[0093] in, represents the discrete representation produced by SQ, Represents the quantization result, represents the index of the product to be found, represents a finite scalar codebook subset The number of elements of . You can also Finally, After a trainable weight The linear layer gets the output of SQ ,Right now .

[0094] (2) Improved vector quantizer IVQ

[0095] IVQ is used to refine acoustic details based on the rough audio profile and obtain quantitative results.

[0096] The goal of IVQ is to refine acoustic details. Assume that IVQ is used for the input vector Discretization, output quantized results .

[0097] first After a trainable weight is The linear layer gets ,in, represents the transformed input vector, Represents the dimension of the code vector.

[0098] Given a trainable codebook , where K represents the number of code vectors in the codebook, represents the code vector in the codebook, , Represents a vector set with dimension M and elements being real numbers.

[0099] The quantization result and discrete representation are obtained by selecting the code vector with the closest Euclidean distance, that is:

[0100] (4);

[0101] In the formula, represents the quantitative result of IVQ, Represents a discrete representation of IVQ production.

[0102] at last After a trainable weight The linear layer gets the output of IVQ ,Right now .

[0103] It should be noted that each improved vector quantizer in the present application is obtained by introducing a codebook online clustering training strategy and load balancing loss on the basis of the original vector quantizer to improve the codebook utilization efficiency.

[0104] The structures of the causal encoder and the causal decoder in this application are basically mirror images, aiming to achieve low-latency inference and real-time efficient generation. The main difference between the causal encoder and the causal decoder is that the causal encoder is a causal downsampling convolution layer, and the causal decoder is a causal upsampling convolution layer.

[0105] For details, see Figure 4 , a schematic diagram of the structure of a causal decoder disclosed in an embodiment of the present application, the causal decoder 30 comprises:

[0106] The second causal input convolutional layer, the causal upsampling convolutional layer, the second linear layer, the second causal improved ConvNeXt v2 module set and the second causal output convolutional layer are connected in series.

[0107] Specific:

[0108] The second causal input convolution layer is used to restore the input quantization result to the original dimension;

[0109] The causal upsampling convolution layer is used to perform feature upsampling on the original dimension quantization result to obtain an upsampling result, and input the upsampling result into the second linear layer;

[0110] The second linear layer is used to perform a linear transformation on the upsampling result to obtain a linear transformation result, and input the linear transformation result into the second causal improved ConvNeXt v2 module set;

[0111] The second causal improved ConvNeXt v2 module set is used to perform deep processing on the linear transformation result to obtain a deep processing result;

[0112] The second causal output convolution layer is used to obtain the MDCT spectrum from the depth processing result, so as to restore the MDCT spectrum to the audio waveform through inverse MDCT.

[0113] In the present application, the causal decoder uses the quantization result of the residual scalar vector quantizer as input, decodes the MDCT spectrum, and finally reconstructs the audio waveform through the inverse MDCT. The structure of the causal decoder is roughly mirrored with the structure of the causal encoder. The quantization result is first restored to the original dimension through the second causal input convolution layer, and then the decoded MDCT spectrum is obtained by the causal upsampling convolution layer, the second linear layer, the second causal improved ConvNeXt v2 module set and the second causal output convolution layer in sequence. The audio waveform is obtained by inverse MDCT processing of the MDCT spectrum.

[0114] In practical applications, the number of causal improved ConvNeXt v2 modules included in the second causal improved ConvNeXt v2 module set is determined according to actual needs, for example, 8 (see Figure 4 ), this application is not limited here.

[0115] Combination Figure 2~Figure 4 , see Figure 5 , a schematic diagram of the structure of an audio processing model disclosed in an embodiment of the present application, the audio processing model includes: Figure 2 The causal encoder 10 in the illustrated embodiment, Figure 3 The residual scalar vector quantizer 20 in the illustrated embodiment, and Figure 4 The causal decoder 30 in the illustrated embodiment.

[0116] The training criteria for the audio processing model are as follows:

[0117] (1) Loss Function

[0118] The audio processing model in this application is a streaming neural network audio codec. The audio processing model basically follows the training method of MDCTCodec during training, and mainly adopts a training strategy based on generative adversarial networks. Multi-resolution MDCT spectral discriminators and generators are used for adversarial training.

[0119] The multi-resolution MDCT spectrum discriminator consists of three parallel sub-discriminators Each sub-discriminator takes the MDCT spectrum extracted from the natural waveform or decoded waveform as input, but the MDCT configuration used when extracting the MDCT spectrum is different, so that the discriminator can make judgments on multi-resolution MDCT spectra. The input of each sub-discriminator is first processed by 5 cascade blocks, each of which consists of a two-dimensional convolution layer and a LeakyReLU activation function. Finally, a single-channel two-dimensional convolution layer outputs the judgment result.

[0120] The present invention adopts the Hinge form of adversarial loss function. For the generator and the discriminator, the adversarial loss is defined as follows:

[0121] Adversarial loss function for the generator The expression of is shown in formula (5):

[0122] (5);

[0123] In the formula, represents the mean value, It represents the output result of the decoder output speech after passing through the i-th decision device, Represents the speech output by the decoder.

[0124] Adversarial loss function of the discriminator The expression of is shown in formula (6):

[0125] (6);

[0126] In the formula, represents the mean value, represents the output result of the real speech passing through the i-th decision device, Indicates real speech.

[0127] In addition, this application also uses the feature matching loss function .

[0128] In addition to the loss function based on the generative adversarial network, the present invention also adopts a spectrum level loss function and a quantization loss function. The spectrum level loss function includes an MDCT spectrum loss function between the decoded MDCT spectrum and the natural MDCT spectrum. and the Mel-spectrogram loss function defined between the decoded waveform and the natural waveform .

[0129] Among them, the MDCT spectrum loss function Defined as the mean square error between MDCT spectra, Mel spectrum loss function Defined as the mean square error and mean absolute error between mel spectra.

[0130] Quantization loss includes commitment loss function And codebook loss function (codebookloss) , both of which calculate the mean square error between the input and output of the quantizer and the mean square error between the input and output of the sub-quantizer in the quantizer. However, the former is only used to update the parameters of the encoder, while the latter only updates the parameters of the quantizer.

[0131] Unlike MDCTCodec, the IVQ used in this application introduces some improved strategies at the training level compared to VQ to improve codebook utilization. First, IVQ adopts an online codebook clustering strategy. At each training step, IVQ forces the reactivation of less frequently used code vectors and clusters them into the feature space to be quantized. Furthermore, in order to avoid some code vectors being used too frequently or too infrequently, IVQ introduces a load balancing loss function , defined as the posterior code distribution of IVQ and the prior uniform distribution The posterior code is estimated by the frequency of each codeword being selected during the training process.

[0132] (2) Training process

[0133] The audio processing model disclosed in the present application adopts a generative adversarial training mode, and uses a generator loss function and a discriminator loss function to alternately train the generator and the discriminator during the training process;

[0134] Among them, the generator loss function is a linear combination of the generator's adversarial loss function, feature matching loss function, MDCT spectrum loss function, Mel spectrum loss function, codebook loss function, commitment loss function and load balancing loss function in a preset proportion.

[0135] Specifically, the audio processing model disclosed in this application adopts a standard generative adversarial training mode, and the generator loss function is a linear combination of all the above loss functions in a preset ratio, that is, the generator loss function The expression is as follows:

[0136] (7);

[0137] In the formula, represents the adversarial loss function of the generator, represents the feature matching loss function, represents the MDCT spectrum loss function, represents the Mel spectrum loss function, represents the codebook loss function, represents the commitment loss function, represents the load balancing loss function.

[0138] All represent hyperparameters, and their specific values ​​are determined according to actual needs, and this application does not limit them here.

[0139] The audio processing model uses the generator loss function during training. And the discriminator loss function The generator and discriminator are trained alternately until the audio processing model converges.

[0140] The following is an example of how to process an audio waveform using an audio processing model:

[0141] Taking voice communication as an example, the practical application of the audio processing model (i.e., streaming neural network audio codec) proposed in the present invention is introduced. In voice communication, the transmitting end encodes the voice signal and sends it to the receiving end, and the receiving end obtains the voice signal after decoding.

[0142] Using the audio processing model proposed in the present invention, the transmitter first extracts the MDCT spectrum from the waveform of the speech signal to be transmitted through MDCT, and sends it to the causal encoder to generate a continuous code. Then, the residual scalar vector quantizer discretizes the continuous code and generates a discrete representation , where superscripts i and j represent the i-th SQ and j-th IVQ, respectively. The discrete representation is transmitted in binary form and sent to the receiving end. After receiving the discrete representation, the receiving end generates a quantization code according to the codebook and inputs it into the causal decoder to generate the MDCT spectrum. Finally, the audio waveform is restored through the inverse MDCT, thus realizing the complete voice communication process.

[0143] Corresponding to the above method embodiment, the present application also discloses an audio processing device.

[0144] See also Figure 6 , a schematic diagram of the structure of an audio processing device disclosed in an embodiment of the present application, the device may include:

[0145] The acquisition unit 201 is used to acquire an audio waveform.

[0146] In practical applications, when audio processing is required, real-time audio waveform can be obtained.

[0147] The processing unit 202 is used to input the audio waveform into the configured audio processing model, encode the discrete cosine transform MDCT spectrum of the audio waveform into a continuous code through the causal encoder in the audio processing model, input the continuous code into the residual scalar vector quantizer for quantization processing to obtain a quantization result, input the quantization result into the causal decoder to decode the MDCT spectrum, and restore the MDCT spectrum to the audio waveform through inverse MDCT.

[0148] In audio coding and decoding, MDCT can convert time domain signals into frequency domain signals. At the same time, MDCT can effectively remove periodic noise caused by edge effects and improve the accuracy of signal processing. And at the same coding rate, MDCT can achieve higher compression ratio and better signal quality than DCT.

[0149] Based on this, the present application uses the MDCT spectrum of the audio waveform as the parameter feature of the audio waveform, obtains the MDCT spectrum through encoding by a causal encoder in the audio processing model, quantization by a residual scalar vector quantizer, and decoding by a causal decoder, and finally restores the MDCT spectrum through inverse MDCT to obtain the original audio waveform.

[0150] The audio processing model in the present application consists of a causal encoder, a residual scalar vector quantizer and a causal decoder. The causal encoder encodes the MDCT spectrum of the audio waveform into a continuous code. When the residual scalar vector quantizer performs quantization processing on the continuous code, it mainly discretizes the continuous code, generates a corresponding discrete representation and generates a quantization code as a quantization result. The causal decoder takes the quantization code as input, decodes the MDCT spectrum of the audio waveform from the quantization code, and restores the inverse MDCT to the audio waveform through the inverse MDCT.

[0151] It should be noted that the audio processing model is a streaming neural network audio codec, which follows the basic framework of MDCTCodec and adopts a causal model structure (causal encoder and causal decoder) to achieve low-latency streaming inference of audio. At the same time, in order to compensate for the degradation of encoding quality caused by the causal structure, the audio processing module also introduces a new residual scalar vector quantization strategy, which enables the audio processing model to achieve low-latency streaming inference of audio while maintaining the encoding quality, efficiency and complexity of MDCTCodec, thereby meeting the use requirements of real-time communication scenarios.

[0152] The causal model structure used in this application can significantly reduce the delay compared to the traditional non-causal model structure. The delay refers to the amount of input required for the model to start. The reason is: the non-causal model needs to input future information, so it has a higher delay. For example, when predicting the result at time t, the model needs to use at least the input information at time t+t0, so the delay time is t0. Using the full causal model of this application, there is no need to input any future information, that is, t0=0, so the delay can be greatly reduced.

[0153] In addition, the decoded audio quality of the audio processing model proposed in this application is comparable to that of the current advanced non-streaming audio codecs, and has the characteristics of low latency, high efficiency and a small number of model parameters, making it suitable for application in real-time communications.

[0154] In summary, the present application discloses an audio processing device, which obtains an audio waveform, inputs the audio waveform into a configured audio processing model, encodes the MDCT spectrum of the audio waveform into a continuous code through a causal encoder in the audio processing model, and inputs it into a residual scalar vector quantizer for quantization processing to obtain a quantization result, inputs the quantization result into a causal decoder to decode the MDCT spectrum, and restores the MDCT spectrum to an audio waveform through inverse MDCT. The present application uses the MDCT spectrum of the audio waveform as a parameter feature of the audio waveform and uses it for encoding, quantization and decoding, thereby improving the frequency domain resolution while reducing the computational complexity of the decoding stage, and the causal model structure used in the audio processing model does not require the input of any future information compared to the traditional non-causal model structure, so it can greatly reduce latency and improve audio processing efficiency, thereby meeting the use requirements of real-time communication scenarios.

[0155] In one embodiment, the causal encoder may include:

[0156] The first causal input convolutional layer, the first causal improved ConvNeXt v2 module set, the first linear layer, the causal downsampling convolutional layer and the first causal output convolutional layer are connected in series;

[0157] The first causal input convolution layer is used to perform preliminary processing of dimensional transformation on the input MDCT spectrum, and input the preliminary processing result into the first causal improved ConvNeXt v2 module set;

[0158] The first causal improved ConvNeXt v2 module set is used to perform deep processing on the preliminary processing result to obtain a deep processing result, and input the deep processing result into the first linear layer;

[0159] The first linear layer is used to perform a linear transformation on the depth processing result to obtain a linear transformation result, and input the linear transformation result into the causal downsampling convolution layer;

[0160] The causal downsampling convolution layer is used to perform feature downsampling on the linear change result to obtain a downsampling result, and input the downsampling result into the first causal output convolution layer;

[0161] The first causal output convolution layer is used to perform dimension transformation processing on the downsampling result to obtain the continuous code.

[0162] In one embodiment, the causal improved ConvNeXt v2 module set includes: a plurality of identical causal improved ConvNeXt v2 modules;

[0163] Each of the causal improved ConvNeXt v2 modules is obtained by causally modifying the ConvNeXt v2 modules proposed in APCodec and MDCTCodec.

[0164] In one embodiment, in each of the causal improved ConvNeXt v2 modules, the output result of the preliminary processing result input by the first causal input convolutional layer is obtained by sequentially passing through a depth-level causal convolutional layer, a linear layer that maps features to a higher dimension, a Gaussian error linear unit, and a linear layer that maps features to the original low dimension, and the total result after adding the preliminary processing result as input is used as the final output of the causal improved ConvNeXtv2 module.

[0165] In one embodiment, the residual scalar vector quantizer comprises:

[0166] A scalar quantizer set and an improved vector quantizer set, wherein the scalar quantizer set and the improved vector quantizer set are connected in a residual manner, a plurality of scalar quantizers in the scalar quantizer set are connected in a residual manner, and a plurality of improved vector quantizers in the improved vector quantizer set are connected in a residual manner;

[0167] The scalar quantizer is used to process the continuous code input by the causal encoder by using rounding as a quantization principle to construct a rough audio profile;

[0168] The improved vector quantizer is used to refine the acoustic details based on the rough audio profile to obtain the quantization result.

[0169] In one embodiment, each of the improved vector quantizers is obtained by introducing a codebook online clustering training strategy and a load balancing loss on the basis of an original vector quantizer.

[0170] In one embodiment, the causal decoder comprises:

[0171] The second causal input convolution layer, the causal upsampling convolution layer, the second linear layer, the second causal improved ConvNeXt v2 module set and the second causal output convolution layer are sequentially connected in series;

[0172] The second causal input convolution layer is used to restore the input quantization result to the original dimension;

[0173] The causal upsampling convolution layer is used to perform feature upsampling on the original dimension quantization result to obtain an upsampling result, and input the upsampling result into the second linear layer;

[0174] The second linear layer is used to perform a linear transformation on the upsampling result to obtain a linear transformation result, and input the linear transformation result into the second causal improved ConvNeXt v2 module set;

[0175] The second causal improved ConvNeXt v2 module set is used to perform deep processing on the linear transformation result to obtain a deep processing result;

[0176] The second causal output convolution layer is used to obtain the MDCT spectrum from the depth processing result, so as to restore the MDCT spectrum to the audio waveform through inverse MDCT.

[0177] In one embodiment, the audio processing model adopts a generative adversarial training mode, and during the training process, the generator and the discriminator are alternately trained using a generator loss function and a discriminator loss function until the audio processing model converges;

[0178] Among them, the generator loss function is a linear combination of the generator's adversarial loss function, feature matching loss function, MDCT spectrum loss function, Mel spectrum loss function, codebook loss function, commitment loss function and load balancing loss function in a preset proportion.

[0179] It should be noted that the specific working principles of each component in the device embodiment can be found in the corresponding part of the method embodiment, which will not be repeated here.

[0180] Corresponding to the above-mentioned embodiment, the present application also discloses a computer storage medium, characterized in that the computer storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the steps shown in the embodiment of the audio processing method are implemented.

[0181] Computer storage media may be tangible media that may contain or store programs for use by or in conjunction with an instruction execution system, device, or apparatus. Computer storage media may be machine-readable signal media or machine-readable storage media. Computer storage media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0182] Corresponding to the above embodiment, Figure 7 As shown, the present invention also provides an electronic device, which may include: a processor 1 and a memory 2;

[0183] The processor 1 and the memory 2 communicate with each other via a communication bus 3;

[0184] Processor 1, configured to execute at least one instruction;

[0185] Memory 2, used to store at least one instruction;

[0186] The processor 1 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0187] The memory 2 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0188] The processor executes at least one instruction to implement the steps shown in the audio processing method embodiment.

[0189] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0190] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0191] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An audio processing method, characterized in that: include: Get the audio waveform; The audio waveform is input into the configured audio processing model, the discrete cosine transform MDCT spectrum of the audio waveform is encoded into a continuous code through the causal encoder in the audio processing model, the continuous code is input into the residual scalar vector quantizer for quantization processing to obtain a quantization result, the quantization result is input into the causal decoder to decode the MDCT spectrum, and the MDCT spectrum is restored to the audio waveform through inverse MDCT.

2. The audio processing method according to claim 1, characterized in that: The causal encoder comprises: The first causal input convolutional layer, the first causal improved ConvNeXt v2 module set, the first linear layer, the causal downsampling convolutional layer and the first causal output convolutional layer are connected in series; The first causal input convolution layer is used to perform preliminary processing of dimensional transformation on the input MDCT spectrum, and input the preliminary processing result into the first causal improved ConvNeXt v2 module set; The first causal improved ConvNeXt v2 module set is used to perform deep processing on the preliminary processing result to obtain a deep processing result, and input the deep processing result into the first linear layer; The first linear layer is used to perform a linear transformation on the depth processing result to obtain a linear transformation result, and input the linear transformation result into the causal downsampling convolution layer; The causal downsampling convolution layer is used to perform feature downsampling on the linear change result to obtain a downsampling result, and input the downsampling result into the first causal output convolution layer; The first causal output convolution layer is used to perform dimension transformation processing on the downsampling result to obtain the continuous code.

3. The audio processing method according to claim 2, characterized in that: The causal improved ConvNeXt v2 module set includes: a plurality of identical causal improved ConvNeXt v2 modules; Each of the causal improved ConvNeXt v2 modules is obtained by causally modifying the ConvNeXtv2 modules proposed in APCodec and MDCTCodec.

4. The audio processing method according to claim 3, characterized in that: In each of the causal improved ConvNeXtv2 modules, the output result of the preliminary processing result input by the first causal input convolutional layer is obtained by successively passing through a depth-level causal convolutional layer, a linear layer that maps features to a higher dimension, a Gaussian error linear unit, and a linear layer that maps features to the original low dimension. The total result after adding the preliminary processing result as input is used as the final output of the causal improved ConvNeXt v2 module.

5. The audio processing method according to claim 1 or 2, characterized in that: The residual scalar vector quantizer comprises: A scalar quantizer set and an improved vector quantizer set, wherein the scalar quantizer set and the improved vector quantizer set are connected in a residual manner, a plurality of scalar quantizers in the scalar quantizer set are connected in a residual manner, and a plurality of improved vector quantizers in the improved vector quantizer set are connected in a residual manner; The scalar quantizer is used to process the continuous code input by the causal encoder by using rounding as a quantization principle to construct a rough audio profile; The improved vector quantizer is used to refine the acoustic details based on the rough audio profile to obtain the quantization result.

6. The audio processing method according to claim 5, characterized in that: Each of the improved vector quantizers is obtained by introducing a codebook online clustering training strategy and a load balancing loss on the basis of the original vector quantizer.

7. The audio processing method according to claim 1 or 2, characterized in that: The causal decoder comprises: The second causal input convolution layer, the causal upsampling convolution layer, the second linear layer, the second causal improved ConvNeXt v2 module set and the second causal output convolution layer are sequentially connected in series; The second causal input convolution layer is used to restore the input quantization result to the original dimension; The causal upsampling convolution layer is used to perform feature upsampling on the original dimension quantization result to obtain an upsampling result, and input the upsampling result into the second linear layer; The second linear layer is used to perform a linear transformation on the upsampling result to obtain a linear transformation result, and input the linear transformation result into the second causal improved ConvNeXt v2 module set; The second causal improved ConvNeXt v2 module set is used to perform deep processing on the linear transformation result to obtain a deep processing result; The second causal output convolution layer is used to obtain the MDCT spectrum from the depth processing result, so as to restore the MDCT spectrum to the audio waveform through inverse MDCT.

8. The audio processing method according to claim 1, characterized in that: The audio processing model adopts a generative adversarial training mode, and during the training process, the generator loss function and the discriminator loss function are used to alternately train the generator and the discriminator until the audio processing model converges; Among them, the generator loss function is a linear combination of the generator's adversarial loss function, feature matching loss function, MDCT spectrum loss function, Mel spectrum loss function, codebook loss function, commitment loss function and load balancing loss function in a preset proportion.

9. An audio processing device, characterized in that: include: An acquisition unit, used for acquiring an audio waveform; A processing unit is used to input the audio waveform into a configured audio processing model, encode the discrete cosine transform MDCT spectrum of the audio waveform into a continuous code through a causal encoder in the audio processing model, input the continuous code into a residual scalar vector quantizer for quantization processing to obtain a quantization result, input the quantization result into a causal decoder to decode the MDCT spectrum, and restore the MDCT spectrum to the audio waveform through inverse MDCT.

10. An electronic device, characterized in that: The electronic device comprises: a memory and a processor; The memory is used to store at least one instruction; The processor is used to execute the at least one instruction to implement the audio processing method as described in any one of claims 1 to 8.

Images