Decoupled VQ-VAE-based speech synthesis method, apparatus and device, and storage medium

Abstract

The invention relates to the technical field of audio signal processing, and discloses a voice synthesis method, device and equipment based on decoupling VQ-VAE, and a storage medium, and the method comprises the steps: randomly selecting a first frame segment from a Mel spectrogram of a to-be-synthesized voice signal as a reference frame segment, inputting the reference frame segment into a global reference encoder, the time-invariant features are decoupled from the reference frame segments through a global reference encoder, and voice information global representation separated from the time-invariant features is obtained; randomly selecting a second frame segment from the Mel spectrogram of the voice signal to be synthesized, inputting the second frame segment and the voice information global representation into a single-codebook voice codec based on the decoupled VQ-VAE, and decoupling the second frame segment into a discrete sequence with time-invariant features and rich voice information by the single-codebook voice codec through the decoupled VQ-VAE; and decoding the discrete sequence through a decoder to generate a reconstructed Mel spectrogram. According to the invention, the problem of multi-sequence prediction is avoided, and the working efficiency and robustness of the voice codec are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio signal processing, and in particular to a speech synthesis method, apparatus, device and storage medium based on decoupled VQ-VAE. Background Art

[0002] Speech codecs are designed to convert speech signals into compact discrete representations and reconstruct the original audio when needed. Speech codecs have a wide range of applications in speech communication, speech storage, and speech synthesis. In the large language model-based speech synthesis (LLM-TTS) system, the speech codec is responsible for converting speech signals into discrete representations (tokens) so that the large language model can process speech like text, and reconstruct the discrete representations generated by the large language model into high-quality speech waveforms.

[0003] At present, mainstream speech codecs in the industry mostly adopt a multi-codebook structure, and the language model needs to predict multiple discrete sequences, which seriously affects the working efficiency and robustness of the codec. Summary of the invention

[0004] The present invention provides a speech synthesis method, apparatus, device and storage medium based on decoupled VQ-VAE to solve the technical problem that the speech codec in the prior art adopts a multi-codebook structure, and the language model needs to predict multiple discrete sequences, which affects the working efficiency and robustness of the codec.

[0005] In a first aspect, a speech synthesis method based on decoupled VQ-VAE is provided, comprising:

[0006] Randomly selecting a first frame segment from the mel-spectrogram of the speech signal to be synthesized as a reference frame segment, inputting the reference frame segment into a global reference encoder, decoupling the time-invariant features from the reference frame segment through the global reference encoder, and obtaining a global representation of speech information separated from the time-varying features;

[0007] Randomly selecting a second frame segment from the mel-spectrogram of the speech signal to be synthesized, inputting the second frame segment and the global representation of speech information into a single codebook speech codec based on decoupled VQ-VAE, wherein the single codebook speech codec decouples the second frame segment into a discrete sequence rich in speech information and with time-invariant features through the decoupled VQ-VAE;

[0008] Decoding the discrete sequence through a decoder to obtain speech detail information of the speech signal to be synthesized, and adding the speech detail information to the global representation of the speech information to generate a reconstructed Mel-spectrogram;

[0009] The reconstructed Mel-spectrogram is converted into a speech waveform by a vocoder to obtain a speech synthesis result.

[0010] In a second aspect, a speech synthesis device based on a decoupled VQ-VAE is provided, comprising:

[0011] Reference encoding module: used for randomly selecting a first frame segment from the mel-spectrogram of the speech signal to be synthesized as a reference frame segment, inputting the reference frame segment into a global reference encoder, and decoupling the time-invariant features from the reference frame segment through the global reference encoder to obtain a global representation of the speech information separated from the time-varying features;

[0012] An audio coding module is used to randomly select a second frame segment from the Mel-spectrogram of the speech signal to be synthesized, and input the second frame segment and the global representation of speech information into a single codebook speech codec based on decoupled VQ-VAE, wherein the single codebook speech codec decouples the second frame segment into a discrete sequence rich in speech information and with time-invariant features through the decoupled VQ-VAE;

[0013] An audio decoding module is used to decode the discrete sequence through a decoder to obtain speech detail information of the speech signal to be synthesized, and add the speech detail information to the global representation of the speech information to generate a reconstructed Mel-spectrogram;

[0014] Speech synthesis module: used for converting the reconstructed Mel-spectrogram into a speech waveform through a vocoder to obtain a speech synthesis result.

[0015] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned speech synthesis method based on decoupled VQ-VAE when executing the computer program.

[0016] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned speech synthesis method based on decoupled VQ-VAE are implemented.

[0017] In the above-mentioned solution implemented by the speech synthesis method, device, computer equipment and storage medium based on decoupled VQ-VAE, a global reference encoder is introduced before encoding and decoding to decouple the time-invariant features from the speech signal, and the global representation of the speech signal is separated from the time-varying content information, so that the speech codec can embed more speech content information during encoding and decoding. A single codebook speech codec based on decoupled VQ-VAE is used to decouple the speech signal into a discrete sequence with time-invariant features and rich speech information, and only a single codebook is used to quantize the discrete sequence, thereby avoiding the problem of multi-sequence prediction, improving the working efficiency and robustness of the speech codec, and achieving better speech reconstruction quality than a multi-codebook codec at a lower bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0019] Figure 1 is a schematic diagram of an application environment of a speech synthesis method based on decoupled VQ-VAE in one embodiment of the present invention;

[0020] Figure 2 is a flow chart of a speech synthesis method based on decoupled VQ-VAE in the first embodiment of the present invention;

[0021] Figure 3 is a flow chart of a speech synthesis method based on decoupled VQ-VAE in the second embodiment of the present invention;

[0022] Figure 4 is a structural schematic diagram of a speech synthesis device based on decoupled VQ-VAE in one embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0024] Figure 6 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The speech synthesis method based on decoupled VQ-VAE provided in the embodiment of the present invention can be applied to Figure 1In an application environment, the client communicates with the server through a network. The server can randomly select a first frame segment from the Mel-spectrogram of the speech signal to be synthesized as a reference frame segment through the client, input the reference frame segment into a global reference encoder, decouple the time-invariant features from the reference frame segment through the global reference encoder, and obtain a global representation of speech information separated from the time-varying features; randomly select a second frame segment from the Mel-spectrogram of the speech signal to be synthesized, input the second frame segment and the global representation of speech information together into a single codebook speech codec based on decoupled VQ-VAE, and the single codebook speech codec decouples the second frame segment into a discrete sequence rich in time-invariant features and speech information through the decoupled VQ-VAE; decode the discrete sequence through a decoder to obtain speech detail information of the speech signal to be synthesized, and add the speech detail information to the global representation of speech information to generate a reconstructed Mel-spectrogram; convert the reconstructed Mel-spectrogram into a speech waveform through a vocoder to obtain a speech synthesis result, and feed the speech synthesis result back to the client. The present invention is applicable to speech synthesis of customer service in the financial field. For speech synthesis of customer service in the financial field, a global reference encoder can be used to decouple time-invariant features from speech signals, and a single codebook speech codec based on decoupled VQ-VAE can be used to decouple speech signals into discrete sequences with time-invariant features and rich speech information, greatly improving the working efficiency and robustness of the speech codec. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0027] See also Figure 2 As shown, Figure 2 A flowchart of a speech synthesis method based on a decoupled VQ-VAE provided in the first embodiment of the present invention includes the following steps:

[0028] S100: randomly selecting a first frame segment from the mel-spectrogram of the speech signal to be synthesized as a reference frame segment, inputting the reference frame segment into a global reference encoder, decoupling the time-invariant features from the reference frame segment through the global reference encoder, and obtaining a global representation of speech information separated from the time-varying features;

[0029] In this step, the time-invariant feature (Time-Invariant) and the time-variant feature (Time-Variant) are used to describe the change of the response of the speech codec system to the input speech signal. The time-invariant feature indicates that the characteristics and behavior of the speech codec system do not change with time. That is, given an input speech signal of the speech codec system, no matter at what time point the input starts, its output will only depend on the speech signal itself, and has nothing to do with the start time of the input speech signal. The time-variant feature indicates that the characteristics and behavior of the speech codec system will vary according to the start time of the input speech signal. In the embodiment of the present invention, a global reference encoder is introduced before speech coding and decoding. The global reference encoder is composed of a convolutional layer and a GRU (GateRecurrent Unit, gated recurrent unit) layer. After the reference frame fragment is input into the global reference encoder, it is first convoluted through the convolutional layer to obtain the convolutional feature, and then the time-invariant feature is decoupled from the convolutional feature by the gating mechanism through the GRU, and the global representation of speech information including timbre and acoustic environment is obtained after the linear layer transformation, so that the single codebook speech codec can embed more speech content information during speech coding and decoding.

[0030] S110: randomly selecting a second frame segment from the mel-spectrogram of the speech signal to be synthesized, inputting the second frame segment and the global representation of speech information into a single codebook speech codec based on a decoupled VQ-VAE, and the single codebook speech codec decouples the second frame segment into a discrete sequence rich in speech information and with time-invariant features through the decoupled VQ-VAE;

[0031] In this step, VQ-VAE (Vector Quantized Variational Autoencoder) is a deep learning model that combines variational autoencoder (VAE) and quantization technology. The embodiment of the present invention uses a decoupled VQ-VAE to decouple the speech signal into a discrete sequence rich in time-invariant embedding and speech information, and only uses a single codebook to quantize the discrete sequence, thereby avoiding the multi-sequence prediction problem of the speech codec and improving the working efficiency and robustness of the speech codec. Specifically, the single codebook speech codec based on the decoupled VQ-VAE includes an encoder, a decoder and a discrete codebook. After the second frame segment and the global representation of the speech information are input into the single codebook speech codec based on the decoupled VQ-VAE, the phoneme information related to the speech in the second frame segment is first extracted by the encoder, and subtracted from the global representation of the speech information to obtain a potential speech content representation, and then the potential speech content representation is mapped to a discrete codebook space by a vector quantizer to obtain a quantized speech content representation.

[0032] Furthermore, the encoder includes a mixed adoption module, a Conformer module, a resampling module and a BLSTM (Bidirectional Long Short-Term Memory) module, and the encoding algorithm of the encoder specifically includes:

[0033] S111: down-sampling the second frame segment through a hybrid sampling module, and sending the down-sampled features to multiple convolution blocks for convolution feature extraction;

[0034] S112: Send the convolutional features to the BLSTM module for context modeling, obtain context information between adjacent frames, and extract the phoneme information related to speech in the convolutional features through the resampling module;

[0035] It can be understood that the embodiment of the present invention obtains context information by introducing a BLSTM module in the encoder, and uses the context information to help discover the correlation between adjacent frames, thereby improving the clustering efficiency of speech content. The introduction of the resampling module encourages the encoder to extract more phoneme information with lower short-term variance and related to speech from the acoustic sequence.

[0036] S113: after the phoneme information is transformed by the linear layer, it is subtracted from the global representation of the speech information, and then up-sampled by a hybrid sampling module to obtain up-sampled features;

[0037] It should be noted that in S111 and S113, the hybrid sampling module combines convolution and pooling operations to downsample the second frame segment, and combines transposed convolution and replication to upsample the subtracted phoneme information, thereby reducing the distortion of upsampling and downsampling.

[0038] S114: sending the upsampled features to the BLSTM module for context modeling, and transforming the output of the BLSTM module through a linear layer to obtain a potential speech content representation;

[0039] S115: Send the potential speech content representation to a vector quantizer, and map the potential speech content representation to a discrete codebook space through the vector quantizer to obtain a quantized speech content representation.

[0040] S120: decoding the discrete sequence through a decoder to obtain speech detail information of the speech signal to be synthesized, and adding the speech detail information to the global representation of the speech information to generate a reconstructed mel-spectrogram;

[0041] In this step, after the encoder encodes to obtain the quantized speech content representation, the speech content representation is input into the decoder, and the decoder decodes the speech content representation to obtain a reconstructed mel spectrum. Specifically, the decoding algorithm of the decoder for the speech content representation includes:

[0042] S121: downsampling the speech content representation through a hybrid sampling module to obtain downsampling features;

[0043] S122: sending the downsampled features to multiple transposed convolution blocks for feature decoding, sending the decoded features to a residual block combined with a BLSTM module, performing context modeling through the residual block, and further restoring speech detail information of the speech signal through a resampling module;

[0044] S123: The output of the residual block is transformed by a linear layer and then added to the global representation of the speech information, and the added features are sent to the BLSTM module for context modeling;

[0045] S124: The output of the BLSTM module is upsampled by the hybrid sampling module and then transformed by a linear layer to obtain a reconstructed Mel-spectrogram.

[0046] S130: converting the reconstructed Mel-spectrogram into a speech waveform through a vocoder to obtain a speech synthesis result;

[0047] In this step, the reconstructed mel-spectrogram is sent to the pre-trained BigVGAN vocoder, and the reconstructed mel-spectrogram is converted into a speech waveform using the BigVGAN vocoder.

[0048] It can be seen that in the above scheme, the speech synthesis method based on decoupled VQ-VAE provided by the first embodiment of the present invention decouples the time-invariant features from the speech signal by introducing a global reference encoder before encoding and decoding, and separates the global representation of the speech signal from the time-varying content information, so that the speech codec can embed more speech content information during encoding and decoding. And a single codebook speech codec based on decoupled VQ-VAE is used to decouple the speech signal into a discrete sequence with time-invariant features and rich speech information, and only a single codebook is used to quantize the discrete sequence, thereby avoiding the problem of multi-sequence prediction, improving the working efficiency and robustness of the speech codec, and achieving better speech reconstruction quality than a multi-codebook codec at a lower bandwidth.

[0049] See also Figure 3 As shown, Figure 3 A flowchart of a speech synthesis method based on a decoupled VQ-VAE provided for the second embodiment of the present invention includes the following steps:

[0050] S200: converting the speech signal to be synthesized into a Mel-spectrogram by using a Fourier transform method;

[0051] In this step, the Mel Spectrogram is a method for converting the spectrum representation of a speech signal to a Mel frequency scale. The embodiment of the present invention converts the speech signal to be synthesized from the time domain to the frequency domain through the Fourier transform method, which can better simulate the human ear's perception of frequency and facilitate the analysis of the intensity and distribution of each frequency component in the speech signal. It can be understood that the Mel Spectrogram obtained by converting the speech signal to be synthesized using the Fourier transform method is the original Mel Spectrogram.

[0052] S210: randomly selecting a first frame segment from the mel-spectrogram as a reference frame segment, inputting the reference frame segment into a global reference encoder, decoupling the time-invariant features from the reference frame segment through the global reference encoder, and obtaining a global representation of the speech information separated from the time-varying features;

[0053] It can be understood that the processing process of the global reference encoder in this step is the same as S100 in the first embodiment, and will not be described again here to avoid redundancy.

[0054] S220: randomly selecting a second frame segment from the mel-spectrogram, inputting the second frame segment and the global representation of speech information into a single codebook speech codec based on a decoupled VQ-VAE, and the single codebook speech codec decouples the second frame segment into a discrete sequence rich in speech information and with time-invariant features through the decoupled VQ-VAE;

[0055] It can be understood that the encoding process of the single codebook speech codec in this step is the same as S110 in the first embodiment, and will not be described again here to avoid redundancy.

[0056] S230: decoding the discrete sequence through a decoder to obtain speech detail information of the speech signal to be synthesized, and adding the speech detail information to the global representation of the speech information to generate a reconstructed mel-spectrogram;

[0057] It can be understood that the encoding process of the single codebook speech codec in this step is the same as S120 in the first embodiment, and will not be described again here to avoid redundancy.

[0058] S240: sending the reconstructed Mel-spectrogram to a pre-trained BigVGAN vocoder, and converting the reconstructed Mel-spectrogram into a speech waveform through the BigVGAN vocoder;

[0059] S250: sending the reconstructed Mel-spectrogram to the discriminator for true and false discrimination, and performing adversarial training on the discriminator according to the discrimination result, so that the reconstructed Mel-spectrogram is closer to the original Mel-spectrogram;

[0060] In this step, the total loss function for adversarial training of the discriminator is:

[0061] L = L_recon + L_commit + L_adv (1)

[0062] Among them, L_recon represents the reconstruction loss, which is used to measure the difference between the reconstructed Mel-spectrogram and the original Mel-spectrogram. L_commit represents the commitment loss, which is used to measure the difference between the speech content representation before and after quantization, and encourages the encoder output to be closer to the cluster center in the codebook. L_adv represents the adversarial loss, which is used to improve the reconstruction quality of the Mel-spectrogram, making the reconstructed Mel-spectrogram closer to the real distribution.

[0063] It can be seen that in the above scheme, the speech synthesis method based on decoupled VQ-VAE provided by the second embodiment of the present invention uses a discriminator to distinguish the true from the false of the reconstructed Mel-spectrogram, and performs adversarial training on the discriminator, so that the reconstructed Mel-spectrogram is closer to the original Mel-spectrogram, thereby improving the speech reconstruction quality of the speech coding and decoding system.

[0064] It can be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0065] In one embodiment, a speech synthesis device based on decoupled VQ-VAE is provided, and the speech synthesis device based on decoupled VQ-VAE corresponds one-to-one to the speech synthesis method based on decoupled VQ-VAE in the above embodiment. Figure 4 As shown, the speech synthesis device based on decoupled VQ-VAE includes an audio conversion module 101, a reference encoding module 102, an audio encoding module 103, an audio decoding module 104, a speech synthesis module 105 and an adversarial training module 106. The functional modules are described in detail as follows:

[0066] Audio conversion module 101: The voice signal to be synthesized is converted into a Mel-spectrogram by using the Fourier transform method; wherein the Mel-spectrogram is a method for converting the spectrum representation of a voice signal into a Mel-frequency scale. The embodiment of the present invention converts the voice signal to be synthesized from the time domain to the frequency domain by using the Fourier transform method, which can better simulate the human ear's perception of frequency and facilitate the analysis of the intensity and distribution of each frequency component in the voice signal. It can be understood that the Mel-spectrogram obtained by converting the voice signal to be synthesized by using the Fourier transform method is the original Mel-spectrogram.

[0067] Reference encoding module 102: used for randomly selecting a first frame segment from the mel-spectrogram as a reference frame segment, inputting the reference frame segment into a global reference encoder, decoupling the time-invariant features from the reference frame segment through the global reference encoder, and obtaining a global representation of the speech information separated from the time-varying features;

[0068] Audio encoding module 103: used for randomly selecting a second frame segment from the Mel-spectrogram, inputting the second frame segment and the global representation of speech information into a single codebook speech codec based on decoupled VQ-VAE, and the single codebook speech codec decouples the second frame segment into a discrete sequence with time-invariant features and rich speech information through the decoupled VQ-VAE;

[0069] The audio decoding module 104 is used to decode the discrete sequence through a decoder to obtain speech detail information of the speech signal to be synthesized, and add the speech detail information to the global representation of the speech information to generate a reconstructed Mel-spectrogram;

[0070] Speech synthesis module 105: used to convert the reconstructed Mel-spectrogram into a speech waveform through a vocoder to obtain a speech synthesis result; wherein the reconstructed Mel-spectrogram is sent to a pre-trained BigVGAN vocoder, and the reconstructed Mel-spectrogram is converted into a speech waveform through the BigVGAN vocoder;

[0071] Adversarial training module 106: used to send the reconstructed Mel-spectrogram to the discriminator for true and false discrimination, and perform adversarial training on the discriminator according to the discrimination result, so that the reconstructed Mel-spectrogram is closer to the original Mel-spectrogram.

[0072] In one embodiment, the reference encoding module 102 is specifically configured to:

[0073] Down-sample the second frame fragment through the hybrid sampling module, and send the down-sampled features to multiple convolution blocks for convolution feature extraction;

[0074] The convolutional features are fed into the BLSTM module for context modeling to obtain context information between adjacent frames. The context information is then fed into the resampling module to extract speech-related phoneme information from the speech signal.

[0075] After the phoneme information is transformed by the linear layer, it is subtracted from the global representation of the speech information, and then upsampled through the hybrid sampling module to obtain the upsampled features;

[0076] The upsampled features are fed into the BLSTM module for context modeling, and the output of the BLSTM module is transformed through a linear layer to obtain a potential content representation;

[0077] The potential content representation is sent to a vector quantizer, and the potential content representation is mapped to a discrete codebook space by the vector quantizer to obtain a quantized content representation.

[0078] In one embodiment, the audio encoding module 103 is specifically configured to:

[0079] Down-sample the quantized content representation through a hybrid sampling module to obtain down-sampled features;

[0080] The downsampled features are fed into multiple transposed convolution blocks for feature decoding, and the decoded features are fed into a residual block combined with a BLSTM module. The residual block is used for context modeling, and the speech detail information of the speech signal is further restored through the resampling module.

[0081] The output of the residual block is transformed by a linear layer and then added to the global representation of the speech information. The added features are then fed into the BLSTM module for context modeling.

[0082] The output of the BLSTM module is upsampled by the hybrid sampling module and then transformed by the linear layer to obtain the reconstructed Mel-spectrogram.

[0083] The speech synthesis device based on decoupled VQ-VAE provided by the present invention decouples the time-invariant features from the speech signal by introducing a global reference encoder before encoding and decoding, and separates the global representation of the speech signal from the time-varying content information, so that the speech codec can embed more speech content information during encoding and decoding. A single codebook speech codec based on decoupled VQ-VAE is used to decouple the speech signal into a discrete sequence with time-invariant features and rich speech information, and only a single codebook is used to quantize the discrete sequence, thereby avoiding the problem of multi-sequence prediction, improving the working efficiency and robustness of the speech codec, and achieving better speech reconstruction quality than a multi-codebook codec at a lower bandwidth.

[0084] For the specific limitations of the speech synthesis device based on decoupled VQ-VAE, please refer to the limitations of the speech synthesis method based on decoupled VQ-VAE above, which will not be repeated here. Each module in the above-mentioned speech synthesis device based on decoupled VQ-VAE can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0085] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it implements a function or step on the server side of a speech synthesis method based on a decoupled VQ-VAE.

[0086] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage storage medium and an internal memory. The non-volatile storage storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements a function or step on the client side of a speech synthesis method based on a decoupled VQ-VAE.

[0087] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0088] Randomly selecting the first frame segment from the mel-spectrogram of the speech signal to be synthesized as a reference frame segment, inputting the reference frame segment into a global reference encoder, decoupling the time-invariant features from the reference frame segment through the global reference encoder, and obtaining a global representation of speech information separated from the time-varying features;

[0089] A second frame segment is randomly selected from the mel-spectrogram of the speech signal to be synthesized, and the second frame segment and the global representation of speech information are input into a single codebook speech codec based on a decoupled VQ-VAE. The single codebook speech codec decouples the second frame segment into a discrete sequence rich in speech information and with time-invariant features through the decoupled VQ-VAE.

[0090] The discrete sequence is decoded by a decoder to obtain speech detail information of the speech signal to be synthesized, and the speech detail information is added to the global representation of the speech information to generate a reconstructed Mel-spectrogram;

[0091] The reconstructed Mel-spectrogram is converted into a speech waveform through a vocoder to obtain the speech synthesis result.

[0092] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0093] Randomly selecting the first frame segment from the mel-spectrogram of the speech signal to be synthesized as a reference frame segment, inputting the reference frame segment into a global reference encoder, decoupling the time-invariant features from the reference frame segment through the global reference encoder, and obtaining a global representation of speech information separated from the time-varying features;

[0094] A second frame segment is randomly selected from the mel-spectrogram of the speech signal to be synthesized, and the second frame segment and the global representation of speech information are input into a single codebook speech codec based on a decoupled VQ-VAE. The single codebook speech codec decouples the second frame segment into a discrete sequence rich in speech information and with time-invariant features through the decoupled VQ-VAE.

[0095] The discrete sequence is decoded by a decoder to obtain speech detail information of the speech signal to be synthesized, and the speech detail information is added to the global representation of the speech information to generate a reconstructed Mel-spectrogram;

[0096] The reconstructed Mel-spectrogram is converted into a speech waveform through a vocoder to obtain the speech synthesis result.

[0097] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0098] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other storage medium used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0099] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0100] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A speech synthesis method based on decoupled VQ-VAE, characterized in that: include: Randomly selecting a first frame segment from the mel-spectrogram of the speech signal to be synthesized as a reference frame segment, inputting the reference frame segment into a global reference encoder, decoupling the time-invariant features from the reference frame segment through the global reference encoder, and obtaining a global representation of speech information separated from the time-varying features; Randomly selecting a second frame segment from the mel-spectrogram of the speech signal to be synthesized, inputting the second frame segment and the global representation of speech information into a single codebook speech codec based on decoupled VQ-VAE, wherein the single codebook speech codec decouples the second frame segment into a discrete sequence rich in speech information and with time-invariant features through the decoupled VQ-VAE; Decoding the discrete sequence through a decoder to obtain speech detail information of the speech signal to be synthesized, and adding the speech detail information to the global representation of the speech information to generate a reconstructed Mel-spectrogram; The reconstructed Mel-spectrogram is converted into a speech waveform by a vocoder to obtain a speech synthesis result.

2. The speech synthesis method based on decoupled VQ-VAE according to claim 1, characterized in that: Decoupling the time-invariant features from the reference frame segments by the global reference encoder to obtain a global representation of the speech information separated from the time-varying features comprises: The global reference encoder includes a convolution layer and a GRU layer. After the reference frame fragment is input into the global reference encoder, the reference frame fragment is convolved by the convolution layer to obtain convolution features. The GRU layer uses a gating mechanism to decouple the time-invariant features from the convolutional features, and obtains a global representation of the speech information after transformation through a linear layer.

3. The speech synthesis method based on decoupled VQ-VAE according to claim 2, characterized in that: The single codebook speech codec decouples the second frame segment into a discrete sequence rich in speech information and having time-invariant features through a decoupled VQ-VAE, including: Down-sampling the second frame fragment by a hybrid sampling module, and sending the down-sampled features to multiple convolution blocks for convolution feature extraction; The convolutional features are sent to the BLSTM module for context modeling to obtain context information between adjacent frames, and the phoneme information related to speech in the convolutional features is extracted through the resampling module; Subtracting the phoneme information from the global representation of the speech information after the phoneme information is transformed by a linear layer, and upsampling is performed through a hybrid sampling module to obtain upsampling features; The upsampled features are fed into a BLSTM module for context modeling, and the output of the BLSTM module is transformed through a linear layer to obtain a potential speech content representation; The speech content representation is sent to a vector quantizer, and the speech content representation is mapped to a discrete codebook space by the vector quantizer to obtain a quantized speech content representation.

4. The speech synthesis method based on decoupled VQ-VAE according to claim 3, characterized in that: The method of decoding the discrete sequence by a decoder to obtain speech detail information of the speech signal to be synthesized, and adding the speech detail information to the global representation of speech information to generate a reconstructed Mel-spectrogram includes: Downsampling the speech content representation by the hybrid sampling module to obtain downsampling features; The downsampled features are sent to the transposed convolution block for feature decoding, the decoded features are sent to the residual block of the BLSTM module for context modeling, and the speech detail information is restored through the resampling module; The output of the residual block is transformed by a linear layer and then added to the global representation of the speech information, and the added features are sent to a BLSTM module for context modeling; The output of the BLSTM module is upsampled by a hybrid sampling module and then transformed by a linear layer to obtain a reconstructed Mel-spectrogram.

5. The speech synthesis method based on decoupled VQ-VAE according to claim 1, characterized in that: Before randomly selecting the first frame segment from the Mel-spectrogram of the speech signal to be synthesized as the reference frame segment, the method further includes: The speech signal to be synthesized is converted into a Mel-spectrogram by using a Fourier transform method.

6. The speech synthesis method based on decoupled VQ-VAE according to any one of claims 1 to 5, characterized in that: After the discrete sequence is decoded by a decoder to obtain speech detail information of the speech signal to be synthesized, and the speech detail information is added to the global representation of speech information to generate a reconstructed mel-spectrogram, the method further includes: The reconstructed Mel-spectrogram is sent to a discriminator for true and false discrimination, and adversarial training is performed on the discriminator according to the discrimination result, so that the reconstructed Mel-spectrogram is closer to the real distribution.

7. The speech synthesis method based on decoupled VQ-VAE according to claim 6, characterized in that: The total loss function for adversarial training of the discriminator is: L=L_recon+L_commit+L_adv Among them, L_recon represents the reconstruction loss, which is used to measure the difference between the reconstructed Mel-spectrogram and the original Mel-spectrogram; L_commit represents the commitment loss, which is used to measure the difference between the speech content representation before and after quantization; L_adv represents the adversarial loss, which is used to improve the reconstruction quality of the Mel-spectrogram.

8. A speech synthesis device based on decoupled VQ-VAE, characterized in that: include: Reference encoding module: used for randomly selecting a first frame segment from the mel-spectrogram of the speech signal to be synthesized as a reference frame segment, inputting the reference frame segment into a global reference encoder, and decoupling the time-invariant features from the reference frame segment through the global reference encoder to obtain a global representation of the speech information separated from the time-varying features; An audio coding module is used to randomly select a second frame segment from the Mel-spectrogram of the speech signal to be synthesized, and input the second frame segment and the global representation of speech information into a single codebook speech codec based on decoupled VQ-VAE, wherein the single codebook speech codec decouples the second frame segment into a discrete sequence rich in speech information and with time-invariant features through the decoupled VQ-VAE; An audio decoding module is used to decode the discrete sequence through a decoder to obtain speech detail information of the speech signal to be synthesized, and add the speech detail information to the global representation of the speech information to generate a reconstructed Mel-spectrogram; Speech synthesis module: used for converting the reconstructed Mel-spectrogram into a speech waveform through a vocoder to obtain a speech synthesis result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the speech synthesis method based on decoupled VQ-VAE as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the speech synthesis method based on decoupled VQ-VAE as described in any one of claims 1 to 7 are implemented.

Images