Audio processing method, device, electronic device and storage medium

The voice processing model of the branch supervised by the vocoder encodes the audio signal into a discrete representation in the audio processing, and is quantized and decoded under the auxiliary supervision of the vocoder, solving the audio quality caused by the quantization of the packet residual vector in HiFiCodec. Problem, achieving higher reconstruction quality and processing speed.

CN118571238BActive Publication Date: 2024-11-12BEIJING YUANJIAN INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411053243.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-12
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

When HiFiCodec uses packet residual vector quantization technology, although it reduces the reconstructed codebook, it affects the audio quality. How to improve the reconstructed audio quality while ensuring model performance becomes a challenge.

Method used

The voice processing model of the vocoder supervised branch is adopted to encode the audio to be processed into a discrete representation, and quantize and decode it under the auxiliary supervision of the vocoder. The speech processing model is obtained through iterative training to improve the quality of the reconstructed audio.

Benefits of technology

While keeping the model performance not degraded, the speed of audio processing and reconstruction quality are improved, the number of codebooks is reduced, and the compression performance is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118571238B_ABST
    Figure CN118571238B_ABST
Patent Text Reader

Abstract

The present application provides an audio processing method, device, electronic device and storage medium, including: inputting the audio to be processed into the encoder network layer of the speech processing model, performing fixed-dimensional encoding processing on the feature vector of the audio to be processed, and outputting the discretized feature vector of the audio to be processed; inputting the discretized feature vector into the quantizer network layer, performing residual vector quantization processing on the discretized feature vector, and outputting the residual vector quantization feature vector of the audio to be processed; inputting the residual vector quantization feature vector into the decoder network layer, decoding the residual vector quantization feature vector, and outputting the reconstructed audio to be processed. The speech processing model combined with the vocoder supervision branch can encode the audio signal into a discrete representation, improve the reconstructed audio quality under the auxiliary supervision of the vocoder, and improve the processing speed without reducing the model performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio processing method, device, electronic device and storage medium. Background Art

[0002] Whether in images, videos or audio, the idea of ​​traditional compression algorithms focuses on reducing redundancy in data sequences and reducing file size, but compression algorithms lose some information from the original information. Neural compression technology is rapidly emerging as a new method that uses neural networks to represent compressed and reconstructed data, achieving high compression rates while achieving low perceptual information loss. Similarly, speech waveforms can be converted into vector format and decoded to regenerate sound. Although SoundStream and Encodec can achieve good compression performance and reconstruction performance, achieving good reconstruction performance requires a lot of codebooks, which increases the pressure and computational overhead of the generated model. In order to maintain the performance of the neural network audio compression model while reducing computational overhead and inference time overhead, HiFiCodec proposes a grouped residual vector quantization technology, which groups the intermediate features, and uses residual vector quantization technology to quantize and compress each group. Finally, the grouped quantized features are combined for decoding, so that the audio can be reconstructed using fewer codebooks while maintaining good reconstruction effects. However, HiFiCodec uses grouped residual vector quantization technology to reduce the reconstructed codebook. Grouped quantization will introduce model learning bias, which will affect the audio quality during reconstruction. Therefore, how to improve the quality of reconstructed audio has become a technical issue that cannot be underestimated. Summary of the invention

[0003] In view of this, the purpose of the present application is to provide an audio processing method, device, electronic device and storage medium. The speech processing model combined with the vocoder supervision branch can encode the audio signal into a discrete representation, improve the reconstructed audio quality under the auxiliary supervision of the vocoder, and improve the processing speed without reducing the model performance.

[0004] The present application provides an audio processing method, which includes:

[0005] Inputting the audio to be processed into the encoder network layer of the pre-trained speech processing model, performing fixed-dimensional encoding processing on the feature vector of the audio to be processed, and outputting the discretized feature vector of the audio to be processed;

[0006] Inputting the discretized feature vector into the quantizer network layer of the speech processing model, performing residual vector quantization processing on the discretized feature vector, and outputting the residual vector quantization feature vector of the audio to be processed;

[0007] The residual vector quantization feature vector is input into the decoder network layer of the speech processing model, the residual vector quantization feature vector is decoded, and the reconstructed audio to be processed is output; wherein the speech processing model is obtained by iteratively training the neural network model based on the target vocoder supervision branch.

[0008] In a possible implementation manner, the speech processing model is determined by the following steps:

[0009] Inputting the Mel spectrum feature of the first sample audio into a pre-trained target vocoder, reconstructing the Mel spectrum feature, and outputting a first reconstructed audio waveform of the first sample audio; wherein the generator of the target vocoder adopts a one-dimensional depthwise separable convolution;

[0010] Inputting the first sample audio into the neural network model, processing based on the initial encoder network layer, the initial quantizer network layer and the initial decoder network layer, and outputting the reconstructed first sample audio;

[0011] Inputting the reconstructed first sample audio into the target vocoder, and determining a first loss value between the reconstructed first sample audio and the first reconstructed audio waveform;

[0012] The neural network model is iteratively trained based on the first loss value, the second loss value of the initial encoder network layer, the third loss value of the initial quantizer network layer, and the fourth loss value of the initial decoder network layer to determine the speech processing model.

[0013] In a possible implementation manner, inputting the Mel-spectrogram feature of the first sample audio into a pre-trained target vocoder, reconstructing the Mel-spectrogram feature, and outputting a first reconstructed audio waveform of the first sample audio includes:

[0014] Inputting the Mel spectrum feature into a generator of a target vocoder, and performing a first convolution process on the Mel spectrum feature in a time dimension based on the generator to generate the Mel spectrum feature after the first convolution process;

[0015] Based on the generator, a second convolution process is performed on the Mel spectrum features after the first convolution process in the spatial dimension to generate a first reconstructed audio waveform of the first sample audio.

[0016] In a possible implementation manner, the target vocoder is determined by the following steps:

[0017] Replacing the one-dimensional convolution of the initial generator in the initial vocoder with a one-dimensional depth-separable convolution to obtain a replaced initial vocoder;

[0018] Inputting the Mel-spectrogram feature of the second sample audio into the initial generator of the replaced initial vocoder for processing to generate a second reconstructed audio waveform of the second sample audio;

[0019] Inputting the reconstructed audio waveform of the second sample audio into the discriminator of the replaced initial vocoder, performing multi-scale and multi-cycle audio identification processing on the second reconstructed audio waveform, and determining the authenticity category of the second reconstructed audio waveform;

[0020] If the category of the second reconstructed audio waveform is a pseudo category, the network parameters of the replaced initial vocoder are changed, and the replaced initial vocoder is iteratively trained until the category of the second reconstructed audio waveform is a true category, then the iterative training is stopped to determine the target vocoder.

[0021] In a possible implementation manner, the iterative training of the neural network model based on the first loss value, the second loss value of the initial encoder network layer, the third loss value of the initial quantizer network layer, and the fourth loss value of the initial decoder network layer to determine the speech processing model includes:

[0022] Performing weighted processing on the first loss value, the second loss value, the third loss value, and the fourth loss value to determine a total loss value;

[0023] If the total loss value is less than or equal to the preset loss threshold, the neural network model is used as the speech processing model. If the total loss value is greater than the preset loss threshold, the network parameters of the neural network model are changed, and the changed neural network model is continued to be iteratively trained.

[0024] In a possible implementation manner, inputting the first sample audio into the neural network model, processing based on an initial encoder network layer, an initial quantizer network layer, and an initial decoder network layer, and outputting a reconstructed first sample audio includes:

[0025] Inputting the first sample audio into the initial encoder network layer for fixed-dimensional encoding processing, and outputting a sample discretized feature vector;

[0026] Inputting the sample discretized feature vector into the initial quantizer network layer for residual vector quantization processing, and outputting a sample residual vector quantization feature vector;

[0027] The sample residual vector quantization feature vector is input into the initial decoder network layer for decoding processing, and the reconstructed first sample audio is output.

[0028] In a possible implementation manner, the second loss value, the third loss value, and the fourth loss value are determined in the following manner:

[0029] Acquire label information of the first sample audio; wherein the label information includes an actual discretized feature vector, an actual residual vector quantization feature vector, and quantized actual audio;

[0030] Calculate the actual discretized feature vector and the sample discretized feature vector based on the loss function to determine the second loss value;

[0031] Calculating the actual residual vector quantization feature vector and the sample residual vector quantization feature vector based on the loss function to determine the third loss value;

[0032] The quantized actual audio and the reconstructed first sample audio are calculated based on the loss function to determine the fourth loss value.

[0033] The present application also provides an audio processing device, the audio processing device comprising:

[0034] An encoding processing module, used for inputting the audio to be processed into the encoder network layer of the pre-trained speech processing model, performing fixed-dimensional encoding processing on the feature vector of the audio to be processed, and outputting a discretized feature vector of the audio to be processed;

[0035] A quantization processing module, used for inputting the discretized feature vector into the quantizer network layer of the speech processing model, performing residual vector quantization processing on the discretized feature vector, and outputting a residual vector quantization feature vector of the audio to be processed;

[0036] A decoding processing module is used to input the residual vector quantization feature vector into the decoder network layer of the speech processing model, decode the residual vector quantization feature vector, and output the reconstructed audio to be processed; wherein the speech processing model is obtained by iteratively training the neural network model based on the target vocoder supervision branch.

[0037] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the audio processing method as described above are performed.

[0038] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the audio processing method as described above are executed.

[0039] The embodiment of the present application provides an audio processing method, device, electronic device and storage medium, the audio processing method comprising: inputting the audio to be processed into the encoder network layer of a pre-trained speech processing model, performing fixed-dimensional encoding processing on the feature vector of the audio to be processed, and outputting the discretized feature vector of the audio to be processed; inputting the discretized feature vector into the quantizer network layer of the speech processing model, performing residual vector quantization processing on the discretized feature vector, and outputting the residual vector quantization feature vector of the audio to be processed; inputting the residual vector quantization feature vector into the decoder network layer of the speech processing model, decoding the residual vector quantization feature vector, and outputting the reconstructed audio to be processed; wherein the speech processing model is obtained by iteratively training the neural network model based on the target vocoder supervision branch. The speech processing model combined with the vocoder supervision branch can encode the audio signal into a discrete representation, improve the quality of the reconstructed audio under the auxiliary supervision of the vocoder, and improve the processing speed without reducing the model performance.

[0040] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A flowchart of an audio processing method provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of the speech processing model training provided in the embodiment of the present application;

[0044] Figure 3 This is one of the structural schematic diagrams of an audio processing device provided in an embodiment of the present application;

[0045] Figure 4 A second structural diagram of an audio processing device provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0048] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied in the field of audio processing technology.

[0049] According to research, whether in images, videos or audio, the idea of ​​traditional compression algorithms focuses on reducing redundancy in data sequences and reducing file size, but compression algorithms will lose some information from the original information. Neural compression technology is rapidly emerging as a new method that uses neural networks to represent compressed and reconstructed data, achieving high compression rates while achieving low perceptual information loss. Similarly, speech waveforms can be converted into vector format and decoded to regenerate sound. Although SoundStream and Encodec can achieve good compression performance and reconstruction performance, achieving good reconstruction performance requires a lot of codebooks, which increases the pressure and computational overhead of the generated model. In order to maintain the performance of the neural network audio compression model while reducing computational overhead and inference time overhead, HiFiCodec proposes a grouped residual vector quantization technology, which groups the intermediate features, and uses residual vector quantization technology to quantize and compress each group. Finally, the grouped quantized features are combined for decoding, so that the audio can be reconstructed using fewer codebooks while maintaining a good reconstruction effect. However, HiFiCodec uses grouped residual vector quantization technology to reduce the reconstructed codebook. Grouped quantization will introduce model learning bias, resulting in poor audio quality during reconstruction.

[0050] Based on this, an embodiment of the present application provides an audio processing method, which can encode the audio signal into a discrete representation in combination with a speech processing model of a vocoder supervision branch, thereby improving the reconstructed audio quality under the auxiliary supervision of the vocoder and improving the processing speed without reducing the model performance.

[0051] See also Figure 1 , Figure 1 This is a flow chart of an audio processing method provided by an embodiment of the present application. Figure 1 As shown in , the audio processing method provided by the embodiment of the present application includes:

[0052] S101: Input the audio to be processed into the encoder network layer of a pre-trained speech processing model, perform fixed-dimensional encoding processing on the feature vector of the audio to be processed, and output a discretized feature vector of the audio to be processed.

[0053] In this step, the audio to be processed is input into the encoder network layer of the speech processing model, the feature vector of the audio to be processed is encoded with fixed dimensions, and the discretized feature vector of the audio to be processed is output.

[0054] The speech processing model is obtained by iteratively training a neural network model according to a target vocoder supervision branch.

[0055] S102: Input the discretized feature vector into the quantizer network layer of the speech processing model, perform residual vector quantization processing on the discretized feature vector, and output the residual vector quantization feature vector of the audio to be processed.

[0056] In this step, the discretized feature vector is input into the quantizer network layer, residual vector quantization is performed on the discretized feature vector, and the residual vector quantization feature vector of the audio to be processed is output.

[0057] Here, in order to reduce the number of codebooks, residual vector quantization quantizes the original vector in stages. The number of quantization stages in this process can be appropriately selected according to the actual situation. VQ quantization is performed once in each stage. After the first quantization, the original input vector is represented by the nearest vector of the first layer of codebook, and then the residual of the original vector and is calculated. After that, the residual is represented by the second codebook, and the residual of and is calculated. After the third VQ quantization, and so on. The quantization process of each layer of residual vector quantization (RVQ, Residual Vector Quantization) only needs to approximate the input vector, and a more accurate representation can be obtained with a smaller number of codebooks. A smaller number of codebooks requires fewer bits to express, and higher quantization accuracy can be obtained at a lower bit rate, providing a higher compression ratio and better compression performance.

[0058] S103: Input the residual vector quantization feature vector into the decoder network layer of the speech processing model, decode the residual vector quantization feature vector, and output the reconstructed audio to be processed.

[0059] In this step, the residual vector quantization feature vector is input into the decoder network layer of the speech processing model, the residual vector quantization feature vector is decoded, and the reconstructed audio to be processed is output.

[0060] Among them, the decoder network layer restores the discrete vector to the original speech vector, and finally obtains the original audio signal.

[0061] The speech processing model is obtained by iteratively training a neural network model based on a target vocoder supervision branch.

[0062] In a possible implementation manner, the speech processing model is determined by the following steps:

[0063] A: Inputting the Mel-spectrogram features of the first sample audio into a pre-trained target vocoder, reconstructing the Mel-spectrogram features, and outputting a first reconstructed audio waveform of the first sample audio; wherein the generator of the target vocoder adopts a one-dimensional depthwise separable convolution.

[0064] Here, the Mel-spectrogram features of the first sample audio are input into a pre-trained target vocoder, the Mel-spectrogram features are reconstructed, and a first reconstructed audio waveform of the first sample audio is output.

[0065] Among them, the target vocoder can be a HiFiGAN vocoder, which can convert the spectrum generated by the acoustic model into high-quality audio. HiFiGAN also introduces multi-scale discriminators and multi-cycle discriminators, with the aim of enhancing the ability of the GAN discriminator to distinguish synthetic or real audio as much as possible. A multi-receptive field fusion module is proposed in the generator. HiFiGAN proposes a residual structure that alternately uses atrous convolution and ordinary convolution to increase the receptive field, ensuring the quality of the synthesized sound while improving the inference speed.

[0066] In a possible implementation manner, inputting the Mel-spectrogram feature of the first sample audio into a pre-trained target vocoder, reconstructing the Mel-spectrogram feature, and outputting a first reconstructed audio waveform of the first sample audio includes:

[0067] (1): Inputting the Mel spectrum feature into the generator of the target vocoder, performing a first convolution process on the Mel spectrum feature in the time dimension based on the generator, and generating the Mel spectrum feature after the first convolution process.

[0068] Here, the Mel spectrum feature is input into the generator of the target vocoder, and the Mel spectrum feature is subjected to the first convolution processing in the time dimension based on the generator to generate the Mel spectrum feature after the first convolution processing.

[0069] (2): Based on the generator, a second convolution processing is performed on the Mel-spectrogram features after the first convolution processing in the spatial dimension to generate a first reconstructed audio waveform of the first sample audio.

[0070] Here, the generator performs a second convolution process on the Mel spectrum features after the first convolution process in the spatial dimension to generate a first reconstructed audio waveform of the first sample audio.

[0071] This application replaces the one-dimensional convolution of the generator with a one-dimensional deep separable convolution. The standard one-dimensional convolution maps the time correlation and channel correlation jointly, while the one-dimensional deep separable convolution maps the time correlation and channel correlation separately. The speech is first segmented, and then the Mel spectrum of one frame is extracted for each segment of speech. Therefore, this application uses a one-dimensional deep separable convolution to separately learn the time correlation and channel correlation in the Mel spectrum. This is more helpful for the network to restore the speech waveform using the Mel spectrum, and it also relatively reduces the learning burden of the network, which helps the network to learn quickly.

[0072] In a possible implementation manner, the target vocoder is determined by the following steps:

[0073] a: The one-dimensional convolution of the initial generator in the initial vocoder is replaced with a one-dimensional depth-wise separable convolution to obtain the replaced initial vocoder.

[0074] Here, the one-dimensional convolution of the initial generator in the initial vocoder is replaced with a one-dimensional depth-separable convolution to obtain a replaced initial vocoder.

[0075] b: Inputting the Mel-spectrogram features of the second sample audio into the initial generator of the replaced initial vocoder for processing to generate a second reconstructed audio waveform of the second sample audio.

[0076] Here, the Mel-spectrogram features of the second sample audio are input into the initial generator of the replaced initial vocoder for processing to generate a second reconstructed audio waveform of the second sample audio.

[0077] The process of determining the second reconstructed audio waveform is consistent with the process of determining the first reconstructed audio waveform, and will not be described in detail in this section.

[0078] c: Input the reconstructed audio waveform of the second sample audio into the discriminator of the replaced initial vocoder, perform multi-scale and multi-cycle audio identification processing on the second reconstructed audio waveform, and determine the authenticity category of the second reconstructed audio waveform.

[0079] Here, the reconstructed audio waveform of the second sample audio is input into the discriminator of the replaced initial vocoder, and multi-scale and multi-period audio identification processing is performed on the second reconstructed audio waveform to determine the authenticity category of the second reconstructed audio waveform.

[0080] The discriminator includes a multi-scale discriminator and a multi-period discriminator. The multi-scale discriminator is derived from the MelGAN vocoder. It continuously averages and pools the speech sequence, gradually halving the length of the speech sequence, and then applies several layers of convolution at different scales of the speech. Finally, it flattens the speech sequence as the output of the multi-scale discriminator. The multi-period discriminator folds the one-dimensional audio sequence into a two-dimensional plane with different sequence lengths and applies two-dimensional convolution on the two-dimensional plane.

[0081] d: If the category of the second reconstructed audio waveform is a pseudo category, the network parameters of the replaced initial vocoder are changed, and the replaced initial vocoder is continuously iteratively trained until the category of the second reconstructed audio waveform is a true category, and the iterative training is stopped to determine the target vocoder.

[0082] Here, if the category of the second reconstructed audio waveform is a pseudo category, the network parameters of the replaced initial vocoder are changed, and the replaced initial vocoder is iteratively trained until the category of the second reconstructed audio waveform is a true category, then the iterative training is stopped to determine the target vocoder.

[0083] Among them, this application replaces the one-dimensional convolution in the generator with a deep one-dimensional separable convolution, which reduces the learning burden of the network and reduces the number of parameters and reasoning consumption.

[0084] B: Input the first sample audio into the neural network model, process it based on the initial encoder network layer, the initial quantizer network layer and the initial decoder network layer, and output the reconstructed first sample audio.

[0085] Here, the first sample audio is input into the neural network model, processed according to the initial encoder network layer, the initial quantizer network layer and the initial decoder network layer, and the quantized first sample audio is output.

[0086] In a possible implementation manner, inputting the first sample audio into the neural network model, processing based on an initial encoder network layer, an initial quantizer network layer, and an initial decoder network layer, and outputting a reconstructed first sample audio includes:

[0087] The first sample audio is input into the initial encoder network layer for fixed-dimensional encoding processing, and a sample discretization feature vector is output; the sample discretization feature vector is input into the initial quantizer network layer for residual vector quantization processing, and a sample residual vector quantization feature vector is output; the sample residual vector quantization feature vector is input into the initial decoder network layer for decoding processing, and the reconstructed first sample audio is output.

[0088] C: Inputting the reconstructed first sample audio into the target vocoder, and determining a first loss value between the reconstructed first sample audio and the first reconstructed audio waveform.

[0089] Here, the reconstructed first sample audio is input into the target vocoder, and the reconstructed first sample audio and the first reconstructed audio waveform are calculated according to the loss function to determine the first loss value.

[0090] D: Iteratively train the neural network model based on the first loss value, the second loss value of the initial encoder network layer, the third loss value of the initial quantizer network layer, and the fourth loss value of the initial decoder network layer to determine the speech processing model.

[0091] Here, the neural network model is iteratively trained according to the first loss value, the second loss value of the initial encoder network layer, the third loss value of the initial quantizer network layer, and the fourth loss value of the initial decoder network layer to determine the speech processing model.

[0092] In a possible implementation manner, the iterative training of the neural network model based on the first loss value, the second loss value of the initial encoder network layer, the third loss value of the initial quantizer network layer, and the fourth loss value of the initial decoder network layer to determine the speech processing model includes:

[0093] i: Perform weighted processing on the first loss value, the second loss value, the third loss value and the fourth loss value to determine a total loss value.

[0094] Here, the first loss value, the second loss value, the third loss value and the fourth loss value are weighted to determine the total loss value.

[0095] ii: If the total loss value is less than or equal to the preset loss threshold, the neural network model is used as the speech processing model; if the total loss value is greater than the preset loss threshold, the network parameters of the neural network model are changed, and the changed neural network model is continued to be iteratively trained.

[0096] Here, if the total loss value is less than or equal to the preset loss threshold, the neural network model is used as the speech processing model. If the total loss value is greater than the preset loss threshold, the network parameters of the neural network model are changed, and the changed neural network model is continued to be iteratively trained.

[0097] In a possible implementation manner, the second loss value, the third loss value, and the fourth loss value are determined in the following manner:

[0098] I: Obtain label information of the first sample audio; wherein the label information includes an actual discretized feature vector, an actual residual vector quantization feature vector, and quantized actual audio.

[0099] II: Calculate the actual discretized feature vector and the sample discretized feature vector based on the loss function to determine the second loss value.

[0100] Here, the actual discretized feature vector and the sample discretized feature vector are calculated according to the loss function to determine the second loss value.

[0101] III: Calculate the actual residual vector quantization feature vector and the sample residual vector quantization feature vector based on the loss function to determine the third loss value.

[0102] Here, the actual residual vector quantization feature vector and the sample residual vector quantization feature vector are calculated according to the loss function to determine the third loss value.

[0103] IV: Calculate the quantized actual audio and the reconstructed first sample audio based on the loss function to determine the fourth loss value.

[0104] Here, the quantized actual audio and the reconstructed first sample audio are calculated according to the loss function to determine the fourth loss value.

[0105] In this application, the target vocoder is combined into the neural network model, a vocoder loss branch is added, the vocoder parameters are frozen during the training process, the branch loss is added to the total loss, and the encoder, quantizer and decoder are continuously iterated and optimized.

[0106] For further information, see Figure 2 , Figure 2 Schematic diagram of speech processing model training provided in the embodiment of the present application. Figure 2As shown, a sample audio is input, and the sample audio is input into a target vocoder and an initial encoder network layer respectively. The target vocoder reconstructs the sample audio to obtain a sample reconstructed audio waveform. The initial encoder network layer processes the sample audio to obtain a sample discretized feature vector. The sample discretized feature vector is input into an initial quantizer network layer to obtain a sample residual vector quantization feature vector. The sample residual vector quantization feature vector is input into an initial decoder network layer to obtain a reconstructed sample audio. The reconstructed sample audio is input into a target vocoder. A first loss value of the vocoder is determined according to the reconstructed sample audio and the sample reconstructed audio waveform. The initial encoder network layer, the initial quantizer network layer and the initial decoder network layer are iteratively trained according to the first loss value, the second loss value, the third loss value and the fourth loss value to generate a final speech processing model.

[0107] This application proposes "speech discretization representation combined with vocoder supervision", using the fine-grained modeling capability of the vocoder to supervise the coding capability of the neural encoder. In addition, the one-dimensional convolution of the generator in the vocoder is replaced by a one-dimensional depth-separable convolution to reduce the number of parameters and training time, and can achieve better reconstruction effects while reducing the codebook. This application makes full use of the fine-grained modeling capability in the vocoder to supervise the neural encoder more finely. The neural encoder proposed in this application has a higher compression rate and better sound quality.

[0108] An audio processing method provided by an embodiment of the present application comprises: inputting the audio to be processed into the encoder network layer of a pre-trained speech processing model, performing fixed-dimensional encoding processing on the feature vector of the audio to be processed, and outputting the discretized feature vector of the audio to be processed; inputting the discretized feature vector into the quantizer network layer of the speech processing model, performing residual vector quantization processing on the discretized feature vector, and outputting the residual vector quantization feature vector of the audio to be processed; inputting the residual vector quantization feature vector into the decoder network layer of the speech processing model, decoding the residual vector quantization feature vector, and outputting the reconstructed audio to be processed; wherein the speech processing model is obtained by iteratively training the neural network model based on the target vocoder supervision branch. The speech processing model combined with the vocoder supervision branch can encode the audio signal into a discrete representation, improve the quality of the reconstructed audio under the auxiliary supervision of the vocoder, and improve the processing speed without reducing the model performance.

[0109] See also Figure 3 , Figure 4 , Figure 3 This is one of the structural schematic diagrams of an audio processing device provided in an embodiment of the present application; Figure 4This is a second structural diagram of an audio processing device provided in an embodiment of the present application. Figure 3 As shown in , the audio processing device 300 includes:

[0110] The encoding processing module 310 is used to input the audio to be processed into the encoder network layer of the pre-trained speech processing model, perform fixed-dimensional encoding processing on the feature vector of the audio to be processed, and output the discretized feature vector of the audio to be processed;

[0111] A quantization processing module 320, configured to input the discretized feature vector into a quantizer network layer of the speech processing model, perform residual vector quantization processing on the discretized feature vector, and output a residual vector quantization feature vector of the audio to be processed;

[0112] The decoding processing module 330 is used to input the residual vector quantization feature vector into the decoder network layer of the speech processing model, decode the residual vector quantization feature vector, and output the reconstructed audio to be processed; wherein the speech processing model is obtained by iteratively training the neural network model based on the target vocoder supervision branch.

[0113] Further, such as Figure 4 As shown, the audio processing device 300 further includes a speech processing model training module 340. The speech processing model training module 340 determines the speech processing model through the following steps:

[0114] Inputting the Mel spectrum feature of the first sample audio into a pre-trained target vocoder, reconstructing the Mel spectrum feature, and outputting a first reconstructed audio waveform of the first sample audio; wherein the generator of the target vocoder adopts a one-dimensional depthwise separable convolution;

[0115] Inputting the first sample audio into the neural network model, processing based on the initial encoder network layer, the initial quantizer network layer and the initial decoder network layer, and outputting the reconstructed first sample audio;

[0116] Inputting the reconstructed first sample audio into the target vocoder, and determining a first loss value between the reconstructed first sample audio and the first reconstructed audio waveform;

[0117] The neural network model is iteratively trained based on the first loss value, the second loss value of the initial encoder network layer, the third loss value of the initial quantizer network layer, and the fourth loss value of the initial decoder network layer to determine the speech processing model.

[0118] Further, when the speech processing model training module 340 is used to input the Mel spectrum feature of the first sample audio into the pre-trained target vocoder, reconstruct the Mel spectrum feature, and output the first reconstructed audio waveform of the first sample audio, the speech processing model training module 340 is specifically used to:

[0119] Inputting the Mel spectrum feature into a generator of a target vocoder, and performing a first convolution process on the Mel spectrum feature in a time dimension based on the generator to generate the Mel spectrum feature after the first convolution process;

[0120] Based on the generator, a second convolution process is performed on the Mel spectrum features after the first convolution process in the spatial dimension to generate a first reconstructed audio waveform of the first sample audio.

[0121] Further, such as Figure 4 As shown, the audio processing device 300 further includes a vocoder training module 350. The vocoder training module 350 determines the target vocoder by the following steps:

[0122] Replacing the one-dimensional convolution of the initial generator in the initial vocoder with a one-dimensional depth-separable convolution to obtain a replaced initial vocoder;

[0123] Inputting the Mel-spectrogram feature of the second sample audio into the initial generator of the replaced initial vocoder for processing to generate a second reconstructed audio waveform of the second sample audio;

[0124] Inputting the reconstructed audio waveform of the second sample audio into the discriminator of the replaced initial vocoder, performing multi-scale and multi-cycle audio identification processing on the second reconstructed audio waveform, and determining the authenticity category of the second reconstructed audio waveform;

[0125] If the category of the second reconstructed audio waveform is a pseudo category, the network parameters of the replaced initial vocoder are changed, and the replaced initial vocoder is iteratively trained until the category of the second reconstructed audio waveform is a true category, then the iterative training is stopped to determine the target vocoder.

[0126] Further, when the speech processing model training module 340 is used to iteratively train the neural network model based on the first loss value, the second loss value of the initial encoder network layer, the third loss value of the initial quantizer network layer, and the fourth loss value of the initial decoder network layer to determine the speech processing model, the speech processing model training module 340 is specifically used to:

[0127] Performing weighted processing on the first loss value, the second loss value, the third loss value, and the fourth loss value to determine a total loss value;

[0128] If the total loss value is less than or equal to the preset loss threshold, the neural network model is used as the speech processing model. If the total loss value is greater than the preset loss threshold, the network parameters of the neural network model are changed, and the changed neural network model is continued to be iteratively trained.

[0129] Further, when the speech processing model training module 340 is used to input the first sample audio into the neural network model, process it based on the initial encoder network layer, the initial quantizer network layer and the initial decoder network layer, and output the reconstructed first sample audio, the speech processing model training module 340 is specifically used to:

[0130] Inputting the first sample audio into the initial encoder network layer for fixed-dimensional encoding processing, and outputting a sample discretized feature vector;

[0131] Inputting the sample discretized feature vector into the initial quantizer network layer for residual vector quantization processing, and outputting a sample residual vector quantization feature vector;

[0132] The sample residual vector quantization feature vector is input into the initial decoder network layer for decoding processing, and the reconstructed first sample audio is output.

[0133] Further, the speech processing model training module 340 determines the second loss value, the third loss value and the fourth loss value in the following manner:

[0134] Acquire label information of the first sample audio; wherein the label information includes an actual discretized feature vector, an actual residual vector quantization feature vector, and quantized actual audio;

[0135] Calculate the actual discretized feature vector and the sample discretized feature vector based on the loss function to determine the second loss value;

[0136] Calculating the actual residual vector quantization feature vector and the sample residual vector quantization feature vector based on the loss function to determine the third loss value;

[0137] The quantized actual audio and the reconstructed first sample audio are calculated based on the loss function to determine the fourth loss value.

[0138] An audio processing device provided by an embodiment of the present application comprises: an encoding processing module for inputting the audio to be processed into the encoder network layer of a pre-trained speech processing model, performing fixed-dimensional encoding processing on the feature vector of the audio to be processed, and outputting the discretized feature vector of the audio to be processed; a quantization processing module for inputting the discretized feature vector into the quantizer network layer of the speech processing model, performing residual vector quantization processing on the discretized feature vector, and outputting the residual vector quantization feature vector of the audio to be processed; a decoding processing module for inputting the residual vector quantization feature vector into the decoder network layer of the speech processing model, performing decoding processing on the residual vector quantization feature vector, and outputting the reconstructed audio to be processed; wherein the speech processing model is obtained by iteratively training the neural network model based on the target vocoder supervision branch. The speech processing model combined with the vocoder supervision branch can encode the audio signal into a discrete representation, improve the reconstructed audio quality under the auxiliary supervision of the vocoder, and improve the processing speed without reducing the model performance.

[0139] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in , the electronic device 500 includes a processor 510 , a memory 520 and a bus 530 .

[0140] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530. When the machine-readable instructions are executed by the processor 510, the above-mentioned Figure 1 as well as Figure 2 The specific implementation of the steps of the audio processing method in the method embodiment shown can be found in the method embodiment, which will not be repeated here.

[0141] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 2 The specific implementation of the steps of the audio processing method in the method embodiment shown can be found in the method embodiment, which will not be repeated here.

[0142] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0144] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0146] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0147] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An audio processing method, characterized in that: The audio processing method comprises: Inputting the audio to be processed into the encoder network layer of the pre-trained speech processing model, performing fixed-dimensional encoding processing on the feature vector of the audio to be processed, and outputting the discretized feature vector of the audio to be processed; Inputting the discretized feature vector into the quantizer network layer of the speech processing model, performing residual vector quantization processing on the discretized feature vector, and outputting the residual vector quantization feature vector of the audio to be processed; Inputting the residual vector quantization feature vector into the decoder network layer of the speech processing model, decoding the residual vector quantization feature vector, and outputting the reconstructed audio to be processed; wherein the speech processing model is obtained by iteratively training the neural network model based on the target vocoder supervision branch; The speech processing model is determined by the following steps: Inputting the Mel-spectrogram feature of the first sample audio into a pre-trained target vocoder, reconstructing the Mel-spectrogram feature, and outputting a first reconstructed audio waveform of the first sample audio; wherein the generator of the target vocoder adopts a one-dimensional depthwise separable convolution; Inputting the first sample audio into the neural network model, processing based on the initial encoder network layer, the initial quantizer network layer and the initial decoder network layer, and outputting the reconstructed first sample audio; Inputting the reconstructed first sample audio into the target vocoder, and determining a first loss value between the reconstructed first sample audio and the first reconstructed audio waveform; The neural network model is iteratively trained based on the first loss value, the second loss value of the initial encoder network layer, the third loss value of the initial quantizer network layer, and the fourth loss value of the initial decoder network layer to determine the speech processing model.

2. The audio processing method according to claim 1, characterized in that: The step of inputting the Mel spectrum feature of the first sample audio into a pre-trained target vocoder, reconstructing the Mel spectrum feature, and outputting a first reconstructed audio waveform of the first sample audio includes: Inputting the Mel spectrum feature into a generator of a target vocoder, and performing a first convolution process on the Mel spectrum feature in a time dimension based on the generator to generate the Mel spectrum feature after the first convolution process; Based on the generator, a second convolution process is performed on the Mel spectrum features after the first convolution process in the spatial dimension to generate a first reconstructed audio waveform of the first sample audio.

3. The audio processing method according to claim 1, characterized in that: The target vocoder is determined by the following steps: Replacing the one-dimensional convolution of the initial generator in the initial vocoder with a one-dimensional depth-separable convolution to obtain a replaced initial vocoder; Inputting the Mel-spectrogram feature of the second sample audio into the initial generator of the replaced initial vocoder for processing to generate a second reconstructed audio waveform of the second sample audio; Inputting the reconstructed audio waveform of the second sample audio into the discriminator of the replaced initial vocoder, performing multi-scale and multi-cycle audio identification processing on the second reconstructed audio waveform, and determining the authenticity category of the second reconstructed audio waveform; If the category of the second reconstructed audio waveform is a pseudo category, the network parameters of the replaced initial vocoder are changed, and the replaced initial vocoder is iteratively trained until the category of the second reconstructed audio waveform is a true category, then the iterative training is stopped to determine the target vocoder.

4. The audio processing method according to claim 1, characterized in that: The iterative training of the neural network model based on the first loss value, the second loss value of the initial encoder network layer, the third loss value of the initial quantizer network layer, and the fourth loss value of the initial decoder network layer to determine the speech processing model includes: Performing weighted processing on the first loss value, the second loss value, the third loss value, and the fourth loss value to determine a total loss value; If the total loss value is less than or equal to the preset loss threshold, the neural network model is used as the speech processing model. If the total loss value is greater than the preset loss threshold, the network parameters of the neural network model are changed, and the changed neural network model is continued to be iteratively trained.

5. The audio processing method according to claim 1, characterized in that: The step of inputting the first sample audio into the neural network model, processing the first sample audio based on an initial encoder network layer, an initial quantizer network layer, and an initial decoder network layer, and outputting the reconstructed first sample audio includes: Inputting the first sample audio into the initial encoder network layer for fixed-dimensional encoding processing, and outputting a sample discretized feature vector; Inputting the sample discretized feature vector into the initial quantizer network layer for residual vector quantization processing, and outputting a sample residual vector quantization feature vector; The sample residual vector quantization feature vector is input into the initial decoder network layer for decoding processing, and the reconstructed first sample audio is output.

6. The audio processing method according to claim 5, characterized in that: The second loss value, the third loss value and the fourth loss value are determined in the following manner: Acquire label information of the first sample audio; wherein the label information includes an actual discretized feature vector, an actual residual vector quantization feature vector, and quantized actual audio; Calculate the actual discretized feature vector and the sample discretized feature vector based on the loss function to determine the second loss value; Calculating the actual residual vector quantization feature vector and the sample residual vector quantization feature vector based on the loss function to determine the third loss value; The quantized actual audio and the reconstructed first sample audio are calculated based on the loss function to determine the fourth loss value.

7. An audio processing device, characterized in that: The audio processing device comprises: An encoding processing module, used for inputting the audio to be processed into the encoder network layer of the pre-trained speech processing model, performing fixed-dimensional encoding processing on the feature vector of the audio to be processed, and outputting a discretized feature vector of the audio to be processed; A quantization processing module, used for inputting the discretized feature vector into the quantizer network layer of the speech processing model, performing residual vector quantization processing on the discretized feature vector, and outputting a residual vector quantization feature vector of the audio to be processed; A decoding processing module, used for inputting the residual vector quantization feature vector into the decoder network layer of the speech processing model, decoding the residual vector quantization feature vector, and outputting the reconstructed audio to be processed; wherein the speech processing model is obtained by iteratively training the neural network model based on the target vocoder supervision branch; The audio processing device also includes a speech processing model training module, which determines the speech processing model through the following steps: Inputting the Mel-spectrogram feature of the first sample audio into a pre-trained target vocoder, reconstructing the Mel-spectrogram feature, and outputting a first reconstructed audio waveform of the first sample audio; wherein the generator of the target vocoder adopts a one-dimensional depthwise separable convolution; Inputting the first sample audio into the neural network model, processing based on the initial encoder network layer, the initial quantizer network layer and the initial decoder network layer, and outputting the reconstructed first sample audio; Inputting the reconstructed first sample audio into the target vocoder, and determining a first loss value between the reconstructed first sample audio and the first reconstructed audio waveform; The neural network model is iteratively trained based on the first loss value, the second loss value of the initial encoder network layer, the third loss value of the initial quantizer network layer, and the fourth loss value of the initial decoder network layer to determine the speech processing model.

8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the audio processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the audio processing method according to any one of claims 1 to 6 are executed.