Discrete speech coding method, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a discrete voice coding method, electronic equipment and a storage medium, and the method comprises the steps: constructing a voice input and frequency spectrum output VAE, enabling the input voice to have speaker tone disturbance, and enabling the VAE structure to be used for multi-task learning; vector quantization is carried out on the VAE, and a VQ-VAE is initialized, so that a quantization layer is added to continue to reduce an information bottleneck; and additionally training a vocoder, and carrying out waveform reconstruction by starting from the code obtained in the VQ-VAE.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech coding, and in particular relates to a discrete speech coding method, an electronic device and a storage medium. Background Art

[0002] Among the related technologies, there are some technologies in academia and industry that use discrete speech coding generated by reconstruction tasks to generate speech (including text-to-speech synthesis, pure speech dialogue, etc.), so that this synthesis can achieve excellent performance based on the existing language model architecture. Typical representatives of this type of coding include EnCodec proposed by MetaAI and DAC proposed by Descript Inc.

[0003] In addition, coding based on speech discrete self-supervision models also has a place in speech discrete representation, such as coding based on HuBERT and wav2vec2.0. This type of coding has a more compact representation space and often focuses on content information. In addition, in low-bit rate coding, there are also a series of works such as Disen-TF-Codec of Communication University of China, TiCodec of University of Science and Technology of China, SingleCodec of Northwestern Polytechnical University, and WavTokenizer of Zhejiang University.

[0004] Among them, the reconstruction-type discrete speech coding: EnCodec and DAC use a similar framework, that is, the architecture of encoder, quantizer, decoder and discriminator, and are trained through the training criteria of VQVAE (Vector-Quantized VariationalAutoEncoder) and GAN (Generative Adversarial Network). In order to have a sufficiently powerful discrete representation space, multi-layer and multi-group quantization techniques are generally used, so that each time step of the discrete speech coding corresponds to more than one codebook.

[0005] Low-bandwidth discrete coding: Disen-TF-Codec, TiCodec, and SingleCodec all use an additional global representation extractor to extract sentence-level representations and reduce the size of the representation space in the quantization layer, forming an information bottleneck. The extracted global representation is inserted into the decoder, forcing the quantizer to reduce the encoding of global information as much as possible, thereby reducing the bit rate. Summary of the invention

[0006] The embodiments of the present invention provide a discrete speech coding method, an electronic device and a storage medium, which are used to solve at least one of the above technical problems.

[0007] In a first aspect, an embodiment of the present invention provides a discrete speech coding method, comprising: constructing a VAE with speech input and spectrum output, and causing the input speech to have speaker timbre disturbance, wherein the VAE structure is used for multi-task learning; performing vector quantization on the VAE, initializing a VQ-VAE, and adding a quantization layer to further reduce the information bottleneck; and additionally training a vocoder to reconstruct a waveform based on the coding obtained in the VQ-VAE.

[0008] In a second aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the steps of the discrete speech coding method of any embodiment of the present invention.

[0009] In a third aspect, an embodiment of the present invention further provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the method described in the first aspect.

[0010] In a fourth aspect, an embodiment of the present invention further provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0011] In the method of the embodiment of the present application, by constructing a low-bandwidth, speaker-independent speech coding, the transmission overhead is very small, and the difficulty of the speech generation algorithm based on the language model in the future can be greatly reduced. Furthermore, the frame rate of the embodiment of the present application is low (25Hz-50Hz), which is conducive to bridging the huge length gap between speech and text. The vocabulary size of the embodiment of the present application is low (300-1024), which not only makes the generation of each step of the language model easier, but also facilitates the direct addition of the BPE algorithm, so that the bit rate can be further reduced. The embodiment of the present application has only one codebook, which makes it unnecessary for the language model to perform any additional complex design. The embodiment of the present application also performs speaker decoupling, so the language model does not need to model speaker information at each step, but only needs to model prosody information, which greatly reduces the difficulty of modeling, and the speaker's timbre only needs to be rendered in the final stage. And this may also open up a new speech synthesis paradigm that independently controls prosody and timbre. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0013] Figure 1 A flowchart of a discrete speech coding method provided by one embodiment of the present invention; Figure 2 A specific example of the structure and training graph of LSCodec provided by one embodiment of the present invention; Figure 3 A high-level comparison of well-known acoustic speech codecs provided for an embodiment of the present invention; Figure 4 A reconstruction result provided by an embodiment of the present invention; Figure 5 A voice conversion result provided by an embodiment of the present invention; Figure 6 Reconstruction and VC indicators in the training phase provided by an embodiment of the present invention; Figure 7 Ablation research on training technology provided by one embodiment of the present invention; Figure 8 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] Most reconstruction-based speech coding often has too large a bandwidth, resulting in multiple codebooks for each coding step, which requires some additional complex design for subsequent speech generation algorithms based on language models, greatly increasing the difficulty and complexity of modeling. Although some work has explored low-bandwidth coding, the "global representation" proposed in these works is not equivalent to speaker representation, that is, it does not have explicit speaker decoupling, which still results in redundant speaker information at each step of coding, which is not conducive to achieving low bandwidth, and therefore does not completely solve the low-bandwidth task.

[0016] If there is no restriction on the space of reconstruction-class discrete coding, multiple discrete codebooks must be used to ensure reasonable reconstruction quality. In order to pursue a lower coding bandwidth, an obvious idea is to decouple the speaker information, because speaker information is global. However, in the low-bit rate coding schemes mentioned above, there is either no speaker decoupling (such as WavTokenizer), or only implicit decoupling is performed so that the decoupling effect is still poor and the bandwidth is still relatively high.

[0017] The inventors have found that if low-bandwidth discrete speech coding is to be constructed through speaker timbre decoupling, a simple method usually adopted by those skilled in the art is to directly use data with speaker labels for decoupling with gradient reversal training. Or, since speakers can also be "verified" without labels, the existing speaker verification model can be inserted into the construction of discrete coding, and decoupled using explicit decoupling targets such as gradient reversal. In addition, like some of the methods mentioned above, decoupling is performed through the information bottleneck that naturally exists in the quantizer. In addition, explicitly perturbing the speaker information and then encoding it is also a feasible and proven idea.

[0018] However, we believe that constructing discrete codes directly through quantizers and performing decoupling is difficult and not easy to train. Therefore, these methods ignore this difficulty, and techniques such as gradient reversal themselves will complicate the training trajectory and easily lead to training instability. In addition, it is possible that the way of injecting speaker information in previous work was too simple to effectively utilize the information bottleneck, and they did not realize more powerful means of information injection.

[0019] This technology uses a three-stage training framework to build low-bandwidth, speaker-decoupled discrete speech codecs. This framework explicitly introduces speaker information perturbations during training, resulting in outstanding decoupling effects. The following are the training stages: (1) First, we construct a variational autoencoder (VAE) with speech input and spectrum output, and make the input speech perturb the speaker’s timbre. This VAE structure will perform multi-task learning.

[0020] (2) Then use the existing VAE to initialize a VQ-VAE, that is, add a quantization layer to further narrow the information bottleneck.

[0021] (3) Finally, an additional vocoder is trained to directly reconstruct the waveform based on the obtained LSCodec encoding.

[0022] In this way, speaker decoupling is first performed in a continuous representation space, and then a quantization step is added when it reaches a certain level to continue to improve the reconstruction quality.

[0023] Please refer to Figure 1 , which shows a specific example of the present application.

[0024] (1) First, we construct a continuous-space variational autoencoder (VAE). This autoencoder takes a continuous waveform as input, outputs a posterior Gaussian distribution with the encoder, and samples from this posterior distribution and inputs it into the decoder. It should be noted that here we perturb the speech to a certain extent before inputting it into the encoder, so that the speaker information is destroyed. This perturbation actually refers to a perturbation based on stretching: we first use a time stretching algorithm based on the sampling rate to stretch or shorten the length of the original speech to a certain multiple, and then use a stretching method based on the WSOLA algorithm to stretch the stretched speech back to its original length. Before and after this process, the content of the speech is aligned, and only the fundamental frequency and the resonance peak position change, so the timbre is also changed. In this VAE model, the decoder predicts two features: the acoustic Mel spectrum feature and the discrete code from the self-supervised model. For the acoustic Mel spectrum, the model calculates the L1 loss for regression; for the self-supervised code, the model uses the cross entropy loss for classification. In addition, we cut a reference sound from the same sentence during training, extract the timbre representation sequence through the pre-trained WavLM features, and effectively inject this timbre information into the decoder through the position-independent cross-attention technique proposed in CTX-vec2wav. Since VAE itself approaches the posterior distribution to the standard normal distribution, there is a decoupled target and information bottleneck, making it difficult for the timbre to pass through this bottleneck and reach the decoder. The training loss at this stage is the L1 regression loss, the cross entropy classification loss, and the KL divergence loss of VAE.

[0025] (2) Subsequently, the trained VAE is used to extract the hidden layer representation (i.e., the mean vector of the Gaussian distribution) on a portion of the training data, and then manual kmeans clustering is performed. A quantizer is initialized with the clustering result, and this quantizer is inserted into the original VAE to form the VQ-VAE structure. The rest of this VQ-VAE continues the VAE of the previous stage, and the parameters are also copied. The regression and classification loss functions remain the same, and the KL divergence is replaced by the VQ-VAE commitment loss. This step constructs a discrete encoding space, which further reduces the information bottleneck.

[0026] (3) Finally, a CTX-vec2wav-alpha vocoder is trained to directly reconstruct the waveform from the discrete LSCodec representation. This step can greatly improve the audio quality. This vocoder is an improved version of CTX-vec2wav, that is, the acoustic features are improved from Mel-spectrogram to WavLM features, which is consistent with the operations in the first two steps. It can receive discrete codes as input, introduce speaker information from other segments of the same sentence in a position-independent cross-attention manner, and train with GAN criteria to achieve high-quality reconstruction.

[0027] In the process of implementing the embodiments of the present application, the inventors have also adopted the following alternative solutions. For example, after extracting representations of the audio after disturbance and the audio before disturbance with the same network structure, the contrastive loss is calculated to make the representations at the same position close and the representations at other positions of the same sentence farther apart. This goal is similar to the way of making speaker-invariant self-supervised features in ContentVec. Its advantages are simplicity and clarity, and decoupling without a decoder. The disadvantages are that it is difficult to train, inefficient, and the decoupling effect depends largely on the perturbation algorithm itself rather than the combined effect of the perturbation algorithm and the information bottleneck, so it is difficult to obtain better results.

[0028] In the process of implementing the embodiments of the present application, the inventors also used the following beta version. For example, starting directly from the second stage (VQ-VAE), a code with three characteristics of reconstruction, decoupling, and small information bottleneck is constructed from scratch. This is in line with the usual solution ideas for this research problem. This is feasible to a certain extent, because the representation space after discretization is very small, and information decoupling can be achieved naturally. However, we found that this is not as effective as training the first stage first and then training the second stage based on it, both in terms of the intelligibility of reconstruction and the effect of timbre decoupling.

[0029] This solution not only constructs a low-bandwidth, speaker-independent speech codec that reduces transmission overhead, but also greatly reduces the difficulty of future speech generation algorithms based on language models. This is because: 1) The frame rate of this solution is low (25Hz-50Hz), which helps to bridge the huge length gap between speech and text.

[0030] 2) The vocabulary size of this scheme is relatively low (300-1024), which not only makes it easier to generate the language model at each step, but also facilitates the direct addition of the BPE algorithm, so that the bit rate can be further reduced.

[0031] 3) This scheme has only one codebook, which means that the language model does not require any additional complex design.

[0032] 4) This solution decouples speakers, so the language model does not need to model speaker information at every step, but only needs to model prosody information, which greatly reduces the difficulty of modeling, and the speaker's timbre only needs to be rendered in the final stage. This may also open up a new speech synthesis paradigm that independently controls prosody and timbre.

[0033] Please refer to Figure 1 , which shows a flowchart of an embodiment of a discrete speech encoding method of the present application.

[0034] like Figure 1 As shown, in step 101, a VAE with speech input and spectrum output is constructed, and the input speech is perturbed with the speaker's timbre, and the VAE structure is used for multi-task learning; In step 102, the VAE is vector quantized and a VQ-VAE is initialized to add a quantization layer to further reduce the information bottleneck; In step 103, an additional vocoder is trained to reconstruct the waveform based on the encoding obtained in the VQ-VAE; In some optional embodiments, constructing a VAE with speech input and spectrum output, and causing the input speech to have speaker timbre disturbance includes: perturbing the speech input to a certain extent so that speaker information is destroyed; constructing a continuous space VAE, the VAE inputs a continuous waveform, the encoder of the VAE outputs a posterior Gaussian distribution, and sampling from the posterior Gaussian distribution is input into the decoder of the VAE.

[0035] In some optional embodiments, disturbing the speech input to a certain extent so that the speaker information is destroyed includes: First, a time stretching algorithm based on sampling rate is used to stretch or shorten the length of the voice input to a certain multiple; then a stretching method based on WSOLA algorithm is used to stretch the stretched voice input back to the original length of the voice input.

[0036] In some optional embodiments, the decoder of the VAE predicts two features: acoustic Mel-spectrogram features and discrete codes from a self-supervised model. For the acoustic Mel-spectrogram features, the VAE calculates L1 loss for regression, and for the discrete codes, the VAE performs classification through cross entropy loss. The training loss of the VAE includes L1 regression loss, cross entropy classification loss, and KL divergence loss of the VAE.

[0037] In some optional embodiments, for the VAE, the method further includes: cutting a reference sound from the same sentence during training, extracting a timbre representation sequence from the reference sound through pre-trained WavLM features, and effectively injecting these timbre information into the decoder of the VAE through the position-independent cross-attention technique proposed in CTX-vec2wav.

[0038] In some optional embodiments, the VAE is vector quantized, and initializing a VQ-VAE includes: extracting hidden layer representations from the trained VAE on a portion of training data; performing kmeans clustering on the hidden layer representations, and initializing a quantizer with the clustering results, and inserting the quantizer into the VAE to form the VQ-VAE.

[0039] In some optional embodiments, when the VQ-VAE is trained, the L1 regression loss and cross entropy classification loss of the VAE are retained, and the KL divergence loss of the VAE is replaced by the contribution loss of the VQ-VAE.

[0040] In some optional embodiments, the additional training of a vocoder to reconstruct the waveform based on the codes obtained in the VQ-VAE includes: training a CTX-vec2wav-alpha vocoder to directly reconstruct the waveform from the discrete codes of the VQ-VAE, wherein the CTX-vec2wav-alpha vocoder is an improved version of CTX-vec2wav, can receive discrete codes as input, introduce speaker information from other segments of the same sentence in a position-independent cross-attention manner, and is trained using GAN criteria to achieve high-quality reconstruction.

[0041] In the method of the embodiment of the present application, by constructing a low-bandwidth, speaker-independent speech coding, the transmission overhead is very small, and the difficulty of the speech generation algorithm based on the language model in the future can be greatly reduced. Furthermore, the frame rate of the embodiment of the present application is low (25Hz-50Hz), which is conducive to bridging the huge length gap between speech and text. The vocabulary size of the embodiment of the present application is low (300-1024), which not only makes the generation of each step of the language model easier, but also facilitates the direct addition of the BPE algorithm, so that the bit rate can be further reduced. The embodiment of the present application has only one codebook, which makes it unnecessary for the language model to perform any additional complex design. The embodiment of the present application also performs speaker decoupling, so the language model does not need to model speaker information at each step, but only needs to model prosody information, which greatly reduces the difficulty of modeling, and the speaker's timbre only needs to be rendered in the final stage. And this may also open up a new speech synthesis paradigm that independently controls prosody and timbre.

[0042] It should be noted that the above method steps are not used to limit the execution order of each step. In fact, some steps may be executed simultaneously or in the opposite order to the specified step order, and this application has no limitation on this.

[0043] The solution of the present application is described below through a specific embodiment so that those skilled in the art can better understand the solution of the present application.

[0044] In the embodiments of the present application, the inventors found that although discrete speech coding shows great potential in speech generation based on language models, high bandwidth and redundant timbre information limit the development of such models. In the embodiments of the present application, we propose a discrete speech codec LSCodec with both low bandwidth and speaker decoupling capabilities. LSCodec adopts a three-stage training framework and speaker perturbation technology. First, LSCodec establishes an information bottleneck in a continuous space, and then performs vector quantization to produce a discrete speaker decoupling space. A discrete coding vocoder is finally used to improve the acoustic details in LSCodec. Through reconstruction experiments, we prove that LSCodec can show excellent intelligibility and audio quality compared to the baseline with only a single codebook and a smaller vocabulary size. The 25Hz version of LSCodec also achieves the lowest bit rate (0.25kbps) in speech acoustic coding to date, and has good sound quality. The timbre conversion evaluation experiment proves the satisfactory speaker decoupling ability of LSCodec, and the ablation study verifies the effectiveness of the training framework we proposed.

[0045] Currently, discrete speech coding has laid the foundation for speech generation tasks because they can be processed in a similar way to existing language models (LMs). According to the goal, discrete speech coding can be divided into acoustic coding and semantic coding. Acoustic coding, or speech codecs (such as EnCodec and DAC) aim to perfectly reconstruct the audio through a vector quantization (VQ) module. Although these codes (tokens) can achieve excellent audio decompression quality, they usually require multiple VQ layers, which leads to additional complex designs for speech LMs, such as two-level modeling, delay mode, or nested prediction. Coding with lower bandwidth is not only beneficial for transmission, but also simplifies the speech LM architecture to reduce costs.

[0046] Unlike other aspects of speech, the speaker's timbre is a global feature that is almost independent of content and prosody. In most speech coding, timbre is encoded repeatedly at different time steps. We believe that this results in a huge waste of coding efficiency. In contrast, semantic coding originates from discriminative tasks such as self-supervised learning (SSL) and focuses more on content other than acoustics. Therefore, these codes usually have fewer encoding codebooks than acoustic coding, thereby reducing bandwidth. However, semantic coding may discard a lot of paralinguistic information, including time-invariant timbre and time-variant prosody. Since prosody is an important part of emotion and semantics, this makes semantic coding also not an ideal choice for speech generation.

[0047] Therefore, it is very valuable to develop a low-bandwidth, speaker-decoupled discrete speech codec. The bandwidth of a speech codec is determined by three factors: the number of quantizers Q, the frame rate F, and the vocabulary size V for each quantizer. The bandwidth can usually be calculated using these three quantities: Q × F × log2V bps. Using a smaller F can alleviate the length mismatch between speech and text modalities. The smaller Q and V, the simpler the training objective of the speech LM. A smaller V can also make more use of the acoustic BPE (BytePair Encoding). Less speaker variation also makes it easier to model the speech LM, allowing it to focus only on time-varying features, and the speaker's timbre can be rendered after the LM is generated.

[0048] Fortunately, reducing bandwidth and removing speaker information are mutually beneficial. Designs that reduce global information in speech coding are common. In the related art, speaker encoders or global feature extractors are used to provide timbre-like features to the decoder. The natural information bottleneck of the VQ layer helps to reduce time-varying features in the coding. In particular, TiCodec and Single-Codec explored the case of only one VQ codebook. However, since these works did not introduce explicit decoupling techniques, the effectiveness of the bottleneck cannot be guaranteed. Single-Codec also sacrifices the ability of streaming coding due to the use of the Conformer encoder. In SpeechTokenizer, semantic encoding is used to guide the first VQ layer in EnCodec, which is actually distillation rather than decoupling. Speaker decoupling can also be achieved through gradient reversal (with or without supervised data), contrastive loss, or exchange prediction, but the related art does not pursue low-bandwidth coding. Another special low-bandwidth codec is WavTokenizer, which achieves reasonable performance using a single VQ layer, but without speaker decoupling. In the embodiment of this application, we propose LSCodec, which explicitly considers speaker decoupling to form low-bandwidth speech coding. LSCodec adopts a three-stage learning process. First, a speech variational autoencoder (VAE) with speaker perturbations is trained to achieve reasonable speaker decoupling and temporal compression. Then a VQ module is added to form a VQ-VAE, on which LSCodec codes are generated. Finally, a discrete coding vocoder is trained separately on the LSCodec codes to improve audio quality. We use a simple stretch-based perturbation method to explicitly remove timbre. We trained two versions of LSCodec at 50Hz and 25Hz frame rates, and their metadata is compared with other existing works. Figure 3 .in, Figure 3 Meta information comparison of well-known acoustic speech codecs is shown. We will experimentally demonstrate that LSCodec has superior reconstruction performance despite using a single codebook and lower bit rate than the baseline. VC (Voice Conversion) experiments further verify that LSCodec achieves significant speaker decoupling. Therefore, LSCodec paves the way for future speech LMs due to its low bandwidth, high compactness, and speaker invariance.

[0049] LSCodec: LOW-BANDWIDTH SPEAKER-DECOUPLEDCODEC Figure 2The structure and training diagram of LSCodec are shown. LSCodec adopts a three-stage learning process, in which the first two stages use speaker perturbation. This section will introduce the perturbation method and the three training stages in detail.

[0050] A. Speaker Perturbation In order to clearly separate discrete speech coding and speaker information, it is necessary to appropriately modify the speaker's timbre before coding while retaining content and prosodic variations. Previous studies have explored a variety of techniques, such as tone and F0 scaling, random equalizers, and vocal cord length perturbations. In the embodiment of the present application, we used a simple time stretching method that can produce time-aligned perturbed speech with only pitch and timbre changes. Given a coefficient β, our perturbation process starts with a rate-based acceleration effect, scaling the total duration of the sentence by β times. This operation changes the position of the pitch and formant, thereby changing the timbre. Then, through the WSOLA algorithm, a pitch-preserving rhythmic effect is applied to stretch the sentence back to its original duration. This perturbation only changes the global pitch position and timbre features, while retaining the pitch variations of the content. Therefore, to reconstruct speech, timbre must be learned from additional inputs rather than perturbed segments. In the reconstruction process, with the help of a strong information bottleneck, the speaker can be clearly separated from the speech coding. During the training phase, β samples each speech in a range centered around 1 to provide random variation.

[0051] B. Training Phase 1: Speech VAE The first training stage of LSCodec is a speech VAE in continuous space. Given a speech waveform, we randomly chop it into reference prompts and content segments. Then, a Convolutional Neural Network (CNN) encoder compresses the signal in the time domain and outputs an isotropic Gaussian posterior N(µ,σ2). The sampled posterior is fed into a Conformer-based decoder. To form a similar information bottleneck before the decoder, we provide the decoder with sufficient timbre information. We feed the timbre from prompt segments into the decoder using position-independent cross-attention due to its excellent timbre controllability. To provide more discriminative timbre features, we extract hidden embeddings from reference prompts using the WavLM SSL model, inspired by its widely demonstrated advantages in speaker verification. Since the reference prompts and content segments are different and the cross-attention mechanism is position-independent, the reference prompts can almost only provide timbre information. When the decoder provides sufficient timbre and perturbation input, the information bottleneck formed by VAE will naturally separate the post-sound and timbre to a certain extent. Before entering the decoder, the WavLM embedding information will be input into a CNN prompt pre-network.

[0052] The decoder predicts two features: a mel-spectrogram from the original unperturbed content segment and SSL semantic tokens. In other words, the VAE model performs multi-task learning, simultaneously regressing to acoustic mel features and classifying to SSL semantic tokens within a fixed vocabulary size. The SSL original encoding is extracted using HuBERT and k-means clustering techniques. The prediction of the mel-spectrogram ensures the reconstruction ability of the bottleneck feature. Since the semantic encoding retains rich content information in a compact discrete space, the SSL token prediction task is crucial to guide the bottleneck feature to encode enough content. At the same time, the prosodic information in the HuBERT encoding is largely destroyed, so retaining the mel prediction task ensures that the prosody is encoded in the bottleneck. It should be noted that no labels are required for the training of the SSL model, so the entire training pipeline remains unsupervised.

[0053] Therefore, at this stage, the training loss includes KL loss , L1 loss for predicting mel features and cross entropy loss L for predicting SSL index idx The weights of the three losses are γ KL , γ reconand γ idx Due to this VAE bottleneck, speaker timbre can be decoupled to a certain extent.

[0054] C. Training Phase 2: Speech VQVAE Next, we inject a VQ layer with a codebook size of V into the trained VAE to obtain the desired LSCodec encodings. We first extract the Gaussian mean µ from the VAE using part of the training data, and then perform k-means clustering with V centers to initialize the codebook. The structure of the VQ-VAE is almost the same as the first-stage VAE. We restore the encoder, decoder, and hint pre-network parameters before training. Before entering the decoder, the quantized encodings are mapped to codebook vectors. The codebook of the VQ layer is updated by an exponential moving average (EMA). In addition to using the commitment loss L in the VQ layer, cmt In addition to replacing the KL loss, the training criterion also inherits the criterion of stage 1. It should be noted that in this stage, the previous Gaussian variance dimension is disabled. cmt By γ cmt Weighted.

[0055] With the VQ layer and reused parameters, this VQVAE is able to construct a discrete space from the continuous space that is not sufficiently separated from the speaker in the VAE. Therefore, the information bottleneck restricts the timbre that can be encoded to a greater extent.

[0056] D. Training Phase 3: Vocoder CTX-vec2wav α (Vocoder CTX-vec2wav α ) Since synthesizing waveforms from predicted mel spectra usually results in audio quality degradation, we also trained a specialized vocoder from the obtained discrete encodings, similar to Vocos. Since the timbre is properly removed in LSCodec encoding, we adopted CTX-vec2wav, which has strong timbre controllability. In order to be consistent with the previous two stages, we developed an improved version of CTX-vec2wav α (CTX-vec2wav-alpha), which receives the same WavLM output as timbre features instead of mel spectrograms. The rest of the vocoder follows the original CTX-vec2wav. With this vocoder, acoustic details are refined to produce high-fidelity waveforms. The final model also serves as a high-quality VC model, which can be used to change the timbre by providing different reference cues.

[0057] Experimental Results A. Data and Model Setup We trained the model using LibriTTS, an English corpus containing 585 hours of 24kHz speech data. To cover enough timbre variation, we used the entire training set of about 2500 speakers. To ensure appropriate prompt length, we discarded all utterances shorter than 6 seconds. This resulted in a training set of about 360 hours. Prompt segments were randomly cut, with lengths sampled between one-third and one-half of the original duration.

[0058] We trained two versions of LSCodec: 50Hz and 25Hz. The CNN encoders for the 50Hz and 25Hz versions contain 11 and 12 residual blocks, respectively, and operate with 512 hidden dimensions. In stage 1, the encoder output is 128-dimensional, which is then evenly split and projected into µ,σ. In stage 2, only the first half of the encoder output is used for quantization and decoding. The encoded vector is 64-dimensional. The codebook is updated with EMA weights γ=0.99. The random replacement of the codebook fails only after 5000 update steps.

[0059] The Conformer decoder contains 2 Conformer blocks, each with 2 heads and 184 attention dimensions. The pre-prompt network has 4 CNN blocks with scaled residual connections with hidden dimensions of 128, 256, and 512, followed by criss-cross attention.

[0060] We use the sixth layer output of the pre-trained WavLM-Large model to provide timbre features. In the SSL code prediction task, we use the pre-trained HuBERT-Large and perform k-means clustering with 2048 centers on a subset of approximately 83 hours of training data. Since the SSL feature is at 50Hz, in the 25Hz version of LSCodec, we repeat each code twice along the time axis before the Conformer decoder. The frame rate of the 80-dimensional cepstral normalized mel spectrum used for reconstruction is also 50Hz.

[0061] Each stage of LSCodec is trained to 200 epochs with a loss weight of γ recon = γ KL = 60, γ idx =2, γ cmt = 1. Other hyperparameters and CTX-vec2wav α The training method follows the method in the related art.

[0062] B. Speech Reconstruction To evaluate the speech reconstruction ability of LSCodec against other low-bitrate speech codecs, we used testset-B, which contains 500 utterances from speakers that have not been seen in the LibriTTS test-clean set. Since low-bitrate speech codecs cannot fully restore every detail of the original waveform, we focus on comparing the degree of information preservation, especially intelligibility and prosody. We use word error rate (WER) to measure intelligibility and gross pitch error rate (GPE) to measure prosody preservation. WER is calculated using NeMo-ASR and real recording transcriptions. GPE represents the ratio of pitch values ​​whose relative error with the real recording is greater than 20%. We also conducted MOS tests as a subjective evaluation of audio quality. The baseline includes using CTX-vec2wav α Semantic SSL encodings of the vocoders, where the wav2vec 2.0 encodings are obtained from its internal quantizer and the HuBERT encodings are from the 2048 center clusters of the last layer. To fairly compare low-bitrate acoustic speech codecs, we compare codecs with bitrates below 1kbps in Table I, namely EnCodec and TiCodec with only 1 VQ layer and WavTokenizer-small at 40Hz. These baselines all use the official models (Official checkpoints).

[0063] Results Figure 4 . Figure 4 The reconstruction results are shown. The bit rate is in kbps.

[0064] Since the 1VQ version of EnCodec produces almost unintelligible speech, we exclude it from the MOS test. The results show that despite the lower bitrate, both the 50Hz and 25Hz versions of LSCodec significantly outperform other acoustic coding baselines in both WER and audio quality. Although LSCodec has a higher GPE than TiCodec and WavTokenizer, the improvement in WER is still worthwhile because a 3-4% difference in GPE is barely noticeable in human perception, but WER is more important. Compared with semantic coding, LSCodec-50Hz outperforms wav2vec 2.0 encoding in all metrics, and both LSCodec versions have GPE that is much better than HuBERT encoding. Note that wav2vec 2.0 has much worse speaker decoupling than LSCodec, which will be shown in later experiments. The 25Hz version of LSCodec performs worse than the 50Hz version because its bitrate is only 60% of the latter. These findings show that LSCodec has excellent reconstruction performance when using very low bandwidth.

[0065] C. Any-to-Any Voice Conversion While the reconstruction experiments show the degree of preservation of content and prosodic information, the speaker separation should be evaluated by VC. In this section, we randomly assign a different target speaker to each source speech in test set-B to complete the VC task. We measure the speaker embedding cosine similarity (SECS) by Resemblyzer between the target speech and the converted speech. The higher the SECS, the less speaker information is encoded. In addition, we also calculate the correlation coefficient of pitch contours (P.Corr) between the source speech and the converted speech. In this way, we can better understand the VC model in terms of both timbre and prosodic similarity. Both SECS and P.Corr range from -1 to 1. Note that when SECS is low, P.Corr becomes meaningless. We will compare the VC model with Figure 4 We compare the same SSL encodings in CTX-vec2wavα in . Since only a few codec models have VC capabilities, we include TiCodec (1VQ and 4VQ variants) and FACodec in the comparison. We discard the detail encodings in FACodec because we find that these encodings still contain a lot of speaker information, which hurts VC performance.

[0066] The results are as follows Figure 5 As shown. Among them, Figure 5 The speech conversion results are shown. The bit rate is in kbps.

[0067] Specifically, LSCodec outperforms other acoustic codecs in speaker similarity in VC, especially FACodec, which decouples speakers under supervision. We found that TiCodec did not perform satisfactorily in terms of speaker similarity, which resulted in too high P.Corr because the converted speech was very similar to the source speech. This shows that it is difficult to separate speakers only by implicit bottleneck methods. In the VC scenario, LSCodec still exhibits lower WER than TiCodec. Compared with semantic coding, LSCodec has a fairly high SECS when encoded with HuBERT and a fairly high P.Corr when encoded with wav2vec 2.0. These findings reflect that the speaker timbre in LSCodec encoding has been largely removed with little impact on prosody, thus having strong VC capabilities.

[0068] D. Phased evaluation and ablation studies Figure 6 The reconstruction and VC index of the training stage are shown. "S" is the abbreviation of the training stage.

[0069] Since the training of LSCodec consists of three consecutive stages, we monitor the reconstruction and VC performance of these three stages. Figure 6 The WER and GPE evaluation values ​​in reconstruction, as well as SECS and P.Corr values ​​in VC are listed in . In addition, we also include V = 300 and V = 2048 versions with a frame rate of 25Hz. In stages 1 and 2, we use the trained HifiGAN to convert the predicted mels into waveforms for evaluation. We stopped the 25Hz V = 300 version in stage 2 due to the degradation of reconstruction performance. From Figure 6 It can be inferred that by adding a properly sized VQ layer, stage 2 generally outperforms stage 1. In addition to better audio quality, stage 3 also greatly improves the VC performance measured by objective SECS. However, how to better preserve content and precursors when training the vocoder remains a focus of future work. We can also find that the 25Hz version faces more challenges in encoding content information compared to the 50Hz version. A larger V also helps reduce WER and GPE, but at the cost of more speaker leakage.

[0070] Please refer to Figure 7 , which shows an ablation study on training techniques.

[0071] We conducted another ablation study to verify the effectiveness of speaker perturbation, SSL code prediction tasks, and the multi-stage training framework. We only included the first two training stages for perturbation and SSL code prediction. The results are shown in Figure 7 As shown in Figure 2. Removing or limiting the speaker perturbation leads to worse WER and SECS for both stages compared to LSCodec-50Hz with β∈ [0.8,1.2]. This suggests that in order to reduce the loss value, the model sacrifices local content to gain more global information. Similar patterns are found when the range of β is enlarged, as this causes too much damage to the data. In addition, removing the SSL encoding prediction loss has a significant impact on the WER of both stages. Starting training directly from stage 2 also degrades the performance of both metrics. Therefore, we conclude that the proposed training framework can effectively generate the desired LSCodec features.

[0072] in conclusion Our proposed LSCodec is a low-bandwidth and speaker-decoupled speech codec. LSCodec builds a three-stage training framework with speaker perturbations. A VQ layer is applied after the VAE, which separates the speakers in a continuous space, and then the vocoder is trained on this basis. Experiments demonstrate that LSCodec achieves strong reconstruction and VC capabilities with bandwidths as low as 0.25kpbs. Ablation studies validate the proposed training framework. LSCodec paves the way for future LM-based speech generation with its highly compact representation space. Exploring stronger perturbation methods and better prosody preservation will be valuable for future work.

[0073] In some other embodiments, the present invention also provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, the computer executable instructions can execute the discrete speech coding method in any of the above method embodiments, for a discrete speech coding system; As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows: Construct a VAE with speech input and spectrum output, and make the input speech have speaker timbre perturbation, and the VAE structure is used for multi-task learning; Vector quantization is performed on the VAE, and a VQ-VAE is initialized to add a quantization layer to further reduce the information bottleneck; An additional vocoder is trained to reconstruct the waveform from the encodings obtained in the VQ-VAE.

[0074] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the discrete speech coding method and system, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the discrete speech coding method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0075] An embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any of the above-mentioned discrete speech encoding methods.

[0076] Figure 8 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 8 As shown, the device includes: one or more processors 810 and a memory 820, Figure 8 A processor 810 is taken as an example. The device of the discrete speech coding method and system may also include: an input device 830 and an output device 840. The processor 810, the memory 820, the input device 830 and the output device 840 may be connected via a bus or other means. Figure 8 In the example, the bus connection is used. The memory 820 is the above-mentioned non-volatile computer-readable storage medium. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 820, that is, the discrete speech coding method of the above-mentioned method embodiment is implemented. The input device 830 can receive input digital or character information, and generate key signal input related to user settings and function control of the discrete speech coding device. The output device 840 may include a display device such as a display screen.

[0077] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0078] As an implementation mode, the electronic device is applied to a discrete speech coding system, and includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: Construct a VAE with speech input and spectrum output, and make the input speech have speaker timbre perturbation, and the VAE structure is used for multi-task learning; Vector quantization is performed on the VAE, and a VQ-VAE is initialized to add a quantization layer to further reduce the information bottleneck; An additional vocoder is trained to reconstruct the waveform from the encodings obtained in the VQ-VAE.

[0079] The electronic device of the embodiment of the present application exists in various forms, including but not limited to: (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0080] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0081] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0082] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0083] (5) Other electronic devices with data interaction functions.

[0084] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0085] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A discrete speech coding method, comprising: Construct a VAE with speech input and spectrum output, and make the input speech have speaker timbre perturbation, and the VAE structure is used for multi-task learning; Vector quantization is performed on the VAE, and a VQ-VAE is initialized to add a quantization layer to further reduce the information bottleneck; An additional vocoder is trained to reconstruct the waveform from the encodings obtained in the VQ-VAE.

2. The method according to claim 1, wherein: The method of constructing a VAE with speech input and spectrum output, and causing the input speech to have speaker timbre disturbance, comprises: Perturbing the speech input to a certain extent so that speaker information is destroyed; A continuous-space VAE is constructed, wherein the VAE inputs a continuous waveform, the encoder of the VAE outputs a posterior Gaussian distribution, and sampling from the posterior Gaussian distribution is input into the decoder of the VAE.

3. The method according to claim 2, wherein: The voice input is disturbed to a certain extent so that the speaker information is destroyed, including: First, a time stretching algorithm based on sampling rate is used to stretch or shorten the length of the speech input to a certain multiple; Then, the stretched voice input is stretched back to its original length using a stretching method based on the WSOLA algorithm.

4. The method according to claim 1, wherein: The decoder of the VAE predicts two features: the acoustic Mel-spectrogram feature and the discrete code from the self-supervised model. For the acoustic Mel-spectrogram feature, the VAE calculates the L1 loss for regression. For the discrete code, the VAE performs classification through the cross entropy loss. The training loss of the VAE includes the L1 regression loss, the cross entropy classification loss and the KL divergence loss of the VAE.

5. The method according to claim 1, further comprising, for the VAE: A reference sound is cut from the same sentence during training, and the timbre representation sequence is extracted from the reference sound through the pre-trained WavLM features. The timbre information is effectively injected into the decoder of the VAE through the position-independent cross-attention technique proposed in CTX-vec2wav.

6. The method according to claim 1, wherein: The VAE is vector quantized and a VQ-VAE is initialized including: Extracting hidden layer representation from the trained VAE on a portion of training data; Perform kmeans clustering on the hidden layer representation, initialize a quantizer with the clustering result, and insert the quantizer into the VAE to form the VQ-VAE.

7. The method according to claim 6, wherein: When the VQ-VAE is trained, the L1 regression loss and cross entropy classification loss of the VAE are retained, and the KL divergence loss of the VAE is replaced by the contribution loss of the VQ-VAE.

8. The method according to claim 1, wherein: The additional training of a vocoder to reconstruct a waveform based on the encoding obtained from the VQ-VAE includes: Train a CTX-vec2wav-alpha vocoder to directly reconstruct waveforms from the discrete codes of the VQ-VAE, where the CTX-vec2wav-alpha vocoder is an improved version of CTX-vec2wav that can receive discrete codes as input, introduce speaker information from other segments of the same sentence in a position-independent cross-attention manner, and is trained using the GAN criterion to achieve high-quality reconstruction.

9. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 8.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images