Audio coding methods, electronic devices, and storage media for speech generation tasks
By using the acoustic BPE encoding method, discrete audio feature sequences are mapped to Unicode strings and encoded using the BPE model. This solves the problem of excessively long discrete audio feature sequences and achieves better semantic modeling and greater diversity and naturalness in synthesized speech.
Patent Information
- Application Number
- CN202311338371.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-16
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-10-16
AI Technical Summary
The existing discrete audio feature sequences are too long, making it difficult for models to learn effectively, and the existing encoding methods cannot recover the original encoding, which limits their application in speech generation tasks.
The acoustic BPE encoding method is adopted to map discrete audio feature sequences to Unicode strings and encode them using a trained BPE model to generate acoustic BPE features, thereby reducing sequence length and utilizing the relationship between feature sequences.
It improves the semantic modeling ability in speech generation tasks, simplifies the complexity of model modeling, and improves the diversity and naturalness of generated speech by selecting the best synthesis result through speech language model.
Smart Images

Figure CN117253493B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech generation tasks, and in particular to an audio encoding method, electronic device, and storage medium for speech generation tasks. Background Technology
[0002] In related technologies, discrete audio features derived from self-supervised learning models have been widely used in speech generation tasks. However, existing methods using discrete audio features do not process them, and this direct use of audio features poses a challenge for sequence modeling because long feature sequences make it difficult for the model to learn. Furthermore, this method relies on the model to construct correlations between discrete features, further complicating the modeling process.
[0003] The existing methods for solving the modeling difficulties caused by excessively long discrete audio feature sequences mainly involve deduplicating continuously repeating features. For example, the feature sequence [1,1,2,2,2,3,4] can be deduplicated into [1,2,3,4] for use in language modeling and speech recognition tasks.
[0004] The inventors believe that the encoding method of this scheme is unidirectional, meaning that the original encoding cannot be recovered from the encoded result. This makes it only applicable to specific tasks such as speech recognition, and not to tasks such as speech generation. Summary of the Invention
[0005] This invention provides an audio encoding method, electronic device, and storage medium for speech generation tasks, which at least solves one of the above-mentioned technical problems.
[0006] In a first aspect, embodiments of the present invention provide an audio encoding method for a speech generation task, comprising: extracting semantic features of an audio waveform using a pre-trained model, discretizing the semantic features into a discrete feature sequence; mapping the discrete feature sequence into a Unicode string, and then training a BPE model on the Unicode string; and encoding the Unicode string using a pre-trained byte pair encoding model to obtain acoustic BPE features.
[0007] In a second aspect, embodiments of the present invention provide an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above-described audio encoding methods for speech generation tasks of the present invention.
[0008] Thirdly, embodiments of the present invention provide a storage medium storing one or more programs including execution instructions, the execution instructions being readable and executable by electronic devices (including but not limited to computers, servers, or network devices, etc.) to perform any of the above-described audio encoding methods for speech generation tasks of the present invention.
[0009] Fourthly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-mentioned audio encoding methods for speech generation tasks.
[0010] The proposed scheme uses BPE (Broadcast Precision Encoding) to encode frequently occurring models in discrete audio sequences into a single unit, reducing sequence length and making it easier to model the sequence. Simultaneously, this encoding method incorporates the relationships between audio feature sequences, alleviating the complexity of model building. These two points make acoustic BPE more suitable for speech generation tasks, and our experiments demonstrate that this acoustic BPE encoding method improves semantic modeling capabilities. Attached Figure Description
[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 A flowchart illustrating an audio encoding method for a speech generation task according to an embodiment of the present invention;
[0013] Figure 2 A flowchart illustrating a specific implementation of the acoustic BPE encoding algorithm provided in an embodiment of the present invention;
[0014] Figure 3 A method for rescoring multiple synthesis results of rhythmic TTS provided in an embodiment of the present invention:
[0015] Figure 4 A comparison of sequence length and inference speed of four acoustic BPE (aBPE) variants provided in an embodiment of the present invention;
[0016] Figure 5 The semantic accuracy of four acoustic BPE variants provided in an embodiment of the present invention;
[0017] Figure 6A comparison of cross-entropy of four acoustic BPE variants provided in an embodiment of the present invention;
[0018] Figure 7 A comparison of the accuracy of the first 1 to the first 3 of four acoustic BPE variants provided in an embodiment of the present invention;
[0019] Figure 8 This is a schematic diagram illustrating the results of two preference tests provided in an embodiment of the present invention;
[0020] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] Please refer to Figure 1 The diagram illustrates a flowchart of an audio encoding method for speech generation tasks provided by an embodiment of the present invention.
[0023] like Figure 1 As shown, in step 101, the semantic features of the audio waveform are extracted using the trained pre-trained model, and the semantic features are discretized into a discrete feature sequence.
[0024] In step 102, the discrete feature sequence is mapped to a Unicode string, and then the BPE model is trained on the Unicode string;
[0025] In step 103, the trained BPE model is used to encode the Unicode string to obtain acoustic BPE features.
[0026] Among them, BPE (byte-pair encoding) is an encoding method that splits words into subwords, while Unicode is a text encoding method.
[0027] The solution in this application effectively reduces sequence length by encoding frequently occurring feature patterns into individual units and utilizes the morphological information present in the feature sequence, thereby alleviating the challenge of modeling the correlation between features.
[0028] In some optional embodiments, the pre-trained model is HuBERT, and discretizing the semantic features into a discrete feature sequence includes using the k-means clustering method to discretize the semantic features into a discrete feature sequence. HuBERT is a self-supervised speech model that primarily extracts semantic information from speech. k-means is a traditional machine learning algorithm that encodes continuous features into discrete features.
[0029] In some alternative embodiments, the method further includes training a speech language model using the acoustic byte pair encoded features.
[0030] In a further optional embodiment, the speech language model is a Decoder-only Transformer, and training the speech language model using the acoustic BPE features includes: encoding the audio waveform into an acoustic BPE sequence; and training the speech language model using the acoustic BPE sequence. Here, Transformer is a self-attention-based model architecture.
[0031] In some optional embodiments, the method further includes: using the speech language model to calculate the likelihood of multiple TTS synthesis results of the prosodic TTS system, and selecting the synthesis result with the highest likelihood among the multiple TTS synthesis results.
[0032] The solution in this application presents a detailed study of the properties of acoustic BPE in speech generation, and is the first solution to use acoustic BPE combined with a speech language model to select from multiple TTS synthesis results.
[0033] By using the BPE encoding method, frequently occurring models in discrete audio sequences are encoded into a single unit, reducing sequence length and making it easier for the model to model the sequence. Simultaneously, this encoding method incorporates the relationships between audio feature sequences, alleviating the complexity of model building. These two points make acoustic BPE more suitable for speech generation tasks, and our experiments also demonstrate that this acoustic BPE encoding method improves semantic modeling capabilities.
[0034] Leveraging acoustic BPE, our proposed rescore method uses a speech language model to evaluate multiple synthesized results from prosodic TTS generation, balancing the prosodic diversity of the synthesized speech with its naturalness. Rescore is a method that evaluates multiple results generated by a prosodic TTS model and selects the synthesized result with the best naturalness.
[0035] Please refer to Figure 2 It illustrates the encoding algorithm for acoustic BPE.
[0036] like Figure 2As shown, the coded acoustic BPE mainly consists of three steps:
[0037] a) First, the semantic features of the audio are extracted using a pre-trained HuBERT model, and then the k-means clustering method is used to discretize these semantic features into discrete feature sequences.
[0038] b) Map discrete feature sequences from numbers to Unicode strings, and then train the BPE model on these strings.
[0039] c) Use the trained BPE model to encode the Unicode string from step 2 to obtain the acoustic BPE features.
[0040] Rescore methods for various synthesis results of prosodic TTS:
[0041] Prosodic TTS systems can synthesize speech with great diversity. However, these systems typically involve a sampling process during the inference phase. This sampling process results in diverse prosody in the synthesized audio, but also leads to inconsistent naturalness. To mitigate this problem, we can synthesize the same sentence multiple times to obtain multiple results, and then use human subjective evaluation to obtain the best result. However, this method is time-consuming and labor-intensive, making it difficult to apply.
[0042] Leveraging the excellent modeling capabilities of acoustic BPE in speech generation tasks, we trained a speech-language model (SLM) using acoustic BPE. This SLM is a decoder-only Transformer. During training, we first encode audio into an acoustic BPE sequence, and then use this sequence to train the SLM. After obtaining the model, we use it to calculate the likelihood of multiple TTS synthesis results, and then select the synthesis result with the highest likelihood. This process is as follows: Figure 3 As shown.
[0043] In the process of developing this application, the inventors also tried the following approach: in addition to existing methods, other supervisory information can be added to the training process of acoustic BPE. For example, we can obtain text-speech alignment information (i.e., which segment in the audio corresponds to which word in the text) using some tools, and then insert delimiters at the boundaries between words when converting to Unicode strings to obtain a better BPE model. However, this approach yielded minimal results in experiments. The poor performance may be due to errors in the alignment information itself, which are amplified during BPE training, making the improvement effect insignificant.
[0044] The following detailed description of the inventor's specific implementation process and experimental data for the embodiments of this application is provided to enable those skilled in the art to better understand the technical solution of this application.
[0045] To address one or more issues in related technologies, we propose Acoustic BPE, which utilizes the BPE algorithm to encode discrete audio features. By encoding frequently occurring feature patterns into single units, it effectively reduces sequence length and leverages morphological information present in the feature sequences, thus alleviating the challenge of modeling correlations between features. Using Acoustic BPE, we trained a speech language model. By exploring various properties of this model, we confirmed the significant advantages offered by Acoustic BPE, including faster autoregressive inference speed and better semantic capture capabilities. Furthermore, we propose a novel rescore method for selecting the best synthesized speech from multiple candidate speech generated by a prosodic TTS model. Experiments demonstrate that the rescore selection is closely related to human preferences, highlighting the potential of Acoustic BPE in other speech generation tasks.
[0046] The emergence of self-supervised learning (SSL) models in the audio domain has brought new options for audio feature selection. For example, wav2vec utilizes a contrastive loss objective to leverage high-level representations of audio signals. HuBERT employs a mask prediction objective, where the discrete objective is derived from k-means clustering. W2v-BERT combines contrastive learning and mask prediction in an end-to-end manner. In addition to modeling the speech content in the audio signal, WavLM also encodes speaker-related information through a denoising objective, requiring the model to predict pseudo-labels for the original audio that is overlapped with noisy or other speech segments. vqwav2vec and wav2vec 2.0 use quantizers to discretize continuous features.
[0047] Discretization can further compress redundant information, allowing algorithms in the Natural Language Processing (NLP) field to be applied to discrete audio features. Through k-means clustering or vector quantizers, the aforementioned SSL features can be used as pseudotext for speech generation. GSLM extracts SSL features from pre-trained HubERT and then discretizes them using k-means clustering. These discrete word chunks are used as input to Tacotron2 for synthesizing Mel spectrograms. VQTTS replaces the Mel spectrogram with vq-wav2vec discretized audio features and 3D prosodic features, serving as a bridge between the acoustic model and the vocoder, improving the quality of synthesized speech compared to the Mel spectrogram. AudioLM trains a language model on discretized audio features using w2v-BERT and k-means clustering, demonstrating the effectiveness of discrete audio features as pseudotext units in language modeling. SPEAR-TTS further confirms this, utilizing discrete audio features as pseudotext units in low-resource scenarios.
[0048] However, existing methods directly utilize features, which poses challenges for sequence modeling. This is mainly because feature sequences are typically long, and the model needs to capture the correlations between features. While some solutions have been proposed to alleviate this problem, such as deduplication of consecutive audio features, these methods suffer from encoding corruption and are therefore unsuitable for speech generation tasks. In text language modeling, this problem can be addressed by combining a continuous sequence of features according to certain rules. For example, Byte-Pair Encoding (BPE) dynamically creates word units based on frequency, encoding morphological information.
[0049] Inspired by text-based methods, we propose an acoustic BPE for speech generation tasks, extending BPE to discrete audio feature sequences to reduce sequence length and fully utilize morphological information within the feature sequences. Previous research has investigated the effectiveness of acoustic BPE in SSL model pre-training and Automatic Speech Recognition (ASR). For example, Hubert-AP uses BPE to encode pseudo-target labels used in Hubert pre-training to bridge the gap between audio signals and natural language. However, it does not extend BPE to the inference process. Related techniques explore the improvements that acoustic BPE brings to ASR tasks. Text-to-speech (TTS) is considered as a machine translation task, where discrete VQVAE sequences are encoded by BPE. However, less attention has been paid to exploring the specific advantages of acoustic BPE in speech generation tasks.
[0050] In this application, we employ our proposed Acoustic BPE to train a language model, which we call a Speech Language Model (SLM). This model is trained using methods from natural language processing after encoding audio into discrete sequences. SLM is a generative model with various applications, including speech continuation and speech evaluation. Through comprehensive research, we have identified several significant advantages of using Acoustic BPE, including faster inference speed, stronger semantic capture capabilities, and improved diversity and richness of generated speech. These advantages can enhance various speech generation tasks, including text-to-speech, speech cloning, and speech enhancement. As an example of the application of Acoustic BPE in speech generation tasks, we introduce a method for re-scoring synthesized speech using Acoustic BPE. This method utilizes the Speech Language Model to evaluate the quality of different candidate speech generated by a prosodic TTS model. By selecting the best synthesized speech, our rescore method achieves a balance between diversity and naturalness. Experimental results show that the rescore selection is closely related to human preferences, further highlighting the effectiveness of Acoustic BPE in other speech generation tasks.
[0051] Next, we introduce the acoustic BPE and the speech language model used in the experiment, detail the research on SLM, and introduce a novel rescoring method as an application of acoustic BPE in speech generation tasks.
[0052] Acoustic BPE speech language model
[0053] In this section, we first provide a comprehensive explanation of acoustic BPE (aBPE). Then, we introduce the unconditional speech language model (SLM) used in our experiments. Finally, we present a novel rescore method that efficiently selects the best synthesized speech from a rich variety of candidate speech generated by TTS.
[0054] Acoustic BPE
[0055] In text corpora, Byte-Pair Encoding (BPE) works by initializing a vocabulary containing all unique characters in the training text. It then iteratively merges the most frequent character pairs into a single unit and adds it to the vocabulary. This continues until the desired vocabulary size is reached. To apply BPE to discrete audio features, we first convert the feature sequence to Unicode text, and then apply BPE training and encoding to the Unicode text. The specific process is as follows... Figure 1 As shown, it includes the following steps:
[0056] 1. We use a pre-trained HuBERT model and k-means clustering to discretize the audio.
[0057] 2. Next, we convert the feature sequence into Unicode text by mapping integers to common Chinese characters located in the Unicode region 4E00~9FFF. This region contains 20992 Chinese characters, which is sufficient to meet our requirements.
[0058] 3. Train a BPE model with a specific vocabulary size on the obtained Unicode text.
[0059] 4. Use the trained BPE model to encode Unicode text into acoustic BPE sequences.
[0060] Speech Language Model
[0061] A language model (with parameter θ) aims to compute a given sequence x = {x1, ..., x...} t The probability of} can be expressed as:
[0062]
[0063] Given a sequence, a language model can compute the probability of that sequence or predict the probability distribution of the next discrete feature, thereby performing tasks such as continuation writing. Speech language models (SLMs) extend language modeling to discrete audio features. However, unlike previous methods that directly utilize discrete audio features, our speech language model leverages acoustic BPE (Broadcast Precision Estimation). This approach inherits the advantages of BPE encoding, such as enhanced sequence modeling and improved semantic capture capabilities. We use a self-attention-based model architecture and briefly describe the training and continuation writing generation processes.
[0064] Continue to refer to Figure 2 This illustrates the acoustic BPE training and encoding. Figure 2 In (a), the audio is discretized into discrete audio features using a pre-trained HuBERT model and k-means clustering. Figure 2 (b) in the example converts discrete features into Unicode text for BPE training. Figure 2 (c) In this step, the trained BPE model is used for encoding to obtain an acoustic BPE sequence.
[0065] Training: We first discretize the audio using HuBERT and k-means clustering, then encode these word blocks into acoustic BPE sequences. We optimize the SLM (with parameter θ) to maximize the joint probability of the acoustic BPE sequences x:
[0066]
[0067] Generate continuation: Speech language models can generate continuation by analyzing the next feature x. {i+1}Perform autoregressive sampling, given the clue p = {p1, ..., p...} l Continue writing after}
[0068] x i+1 ~p(xi|x <i ,p,θ) (3)
[0069] Re-scoring synthesized speech from a prosodic synthesis model using SLM
[0070] State-of-the-art TTS systems can generate speech with diverse prosody. However, the naturalness of these synthesized speeches can vary greatly. To strike a balance between diversity and naturalness, one approach is to generate multiple candidate speeches from the TTS system and then manually select the most natural one. However, this manual evaluation process is time-consuming and laborious. To address this issue, we propose a novel rescore method that utilizes SLM to select the optimal synthesized speech, ensuring both diversity and naturalness in the synthesized speech.
[0071] For multiple synthesized speech u1, ..., u n We use SLM for rescore, first calculating the u of each speech unit. i The probability y i Then, the synthesized speech with the highest probability is selected as the optimal speech, as follows:
[0072]
[0073]
[0074]
[0075] The feature extractor first discretizes the audio into features using HuBERT and k-means, and then further encodes them into acoustic BPE features.
[0076] experiment
[0077] Datasets: We used two datasets for our experiments: LibriTTS, containing 580 hours of speech data, and a subset of LibriLight, containing 6,000 hours of speech. For LibriLight, we used the official script to segment the original long recordings into 60-second segments. All speech waveforms were downsampled to 16kHz before training and inference.
[0078] Acoustic BPE: We extract speech features from the last layer of the pre-trained Hubert Large model. Then, we randomly select a 100-hour subset of speech features from the LibriTTS train-960 dataset and train it using k-means clustering with 2000 centroids. Figure 2 As shown in (a), all speech data from LibriTTS and LibriLight are encoded into discrete audio features by a trained k-means model.
[0079] The acoustic BPE model was trained on a subset of LibriTTS train-960. We first converted the discrete features to Unicode text, then trained the acoustic BPE model using SentencePiece with the required vocabulary size described earlier. We then used the trained BPE model to encode the discrete features of LibriLight into acoustic BPE features. In the experiments below, we compared four acoustic BPE variants: no acoustic BPE and acoustic BPE with vocabulary sizes of 5k, 10k, and 20k.
[0080] Speech Language Model: We used a self-attention-based model architecture with 12 layers, 16 attention heads, 1024 embedding dimensions, and T5-style relative position encoding. During training, we employed random pruning, setting the equivalent input length to 15 seconds. We trained the model on the LibriLight6k subset for 10 epochs on four acoustic BPE variants, with the learning rate linearly increasing from 0 to 1×10⁻⁶ in the first epoch. -5 The learning rate then cosine decays to 1×10⁻⁶ during training. -6 .
[0081] Decoding Discrete Features into Waveforms: We trained a CTX-vec2wav vocoder to decode discrete features into sound waveforms. The CTX-vec2wav frame shift was 20ms, and its HifiGAN upsampling layer kernel size was (16, 10, 8, 4). It was trained on a subset of LibriTTS train-960. During decoding, the BPE phonemes were first decoded and then synthesized into waveforms. In all experiments, we used the same speaker (speaker ID 121) as the synthesis cue.
[0082] Next, we will compare the speech continuation inference speed using various acoustic BPE variants, then examine the semantic capture capability of SLM, and further investigate the diversity and richness of the audio generated by speech continuation. Finally, we demonstrate the effectiveness of our new rescoring method, confirming the advantages of acoustic BPE.
[0083] Inference acceleration
[0084] Acoustic BPE combines frequently occurring features into a single unit, thus significantly reducing sequence length. Please refer to [reference needed]. Figure 4 It shows a comparison of sequence length and inference speed for four acoustic BPE variants. The English and Chinese translations are as follows: Encoding, Number of Acoustic BPEs: Num.aBPE, Average Sequence Length: Avg sequence, Inference Speedup: Inferencespeedup, No Acoustic BPE: w / o aBPE, With Acoustic BPE: aBPE.
[0085] exist Figure 4 In our study, we compared the average sequence length of LibriLight-6k across four acoustic BPE variants: without acoustic BPE and with acoustic BPEs of 5k, 10k, and 20k vocabulary sizes. Furthermore, we evaluated their inference speed in SLM continuation. We used the first 3 seconds of randomly selected speech from the LibriTTS test-clean subset as cues and generated 10 continuations, each 20 seconds long, on an NVIDIA V100 GPU equipped with 32GB of memory. Figure 4 The speed improvement brought about by acoustic BPE was compared.
[0086] from Figure 4 As can be seen, using acoustic BPE can compress sequences by 1.6 to 2.4 times, thus simplifying sequence modeling. Shorter sequences also speed up the autoregressive inference process by 2.8 to 5.0 times.
[0087] Semantic capture capabilities when using acoustic BPE
[0088] Following GLSM and AudioLM, we evaluate the semantic modeling capability of SLM by measuring its ability to distinguish between a pair of semantically correct and incorrect statements. Similar to the rescore method mentioned above, we consider statements with higher probabilities as syntactically correct. To construct such test pairs, we filter out statements that are too short from the LibriTTS test-all subset, and accept the remaining 5497 statements as syntactically correct. To create meaningless test cases, we randomly shuffle words in syntactically correct text. Both semantically correct and incorrect statements are synthesized into waveforms using VQTTS trained on LibriTTS train-960. Subsequently, SLM is used to classify the two speech pairs in each test case as semantically correct and incorrect. This classification accuracy is called semantic accuracy.
[0089] Please refer to Figure 5The paper compares the semantic detection accuracy and generative diversity of four acoustic BPE (aBPE) variants. It demonstrates that the addition of acoustic BPE enhances the speech language model's ability to accurately capture syntactic structures and effectively simulates more diverse syntactic patterns.
[0090] In addition, we examined the content diversity of the generated speech continuation. We randomly selected three sentences from the LibriTTS test-clean subset, using the first three seconds of each sentence as prompts, and generated a 20-second continuation using SLM. For each prompt, we repeated the continuation process 50 times, generating a total of 150 sentences. We used the open-source Whisper software to transcribe the speech into text. To assess the diversity of these texts, we used the n-gram VERT metric. This metric is the geometric mean of the n-gram self-BLEU and n-gram auto-BLEU scores, used to measure n-gram diversity across sentences and within sentences. A higher VERT value indicates lower diversity. Figure 5 The results for 3-gram VERT are listed.
[0091] The improvement of information rate by acoustic BPE
[0092] With acoustic BPE, SLM can generate a variety of outputs with diverse content. In this section, we will explore other improvements brought by acoustic BPE, especially in terms of output generation, where it can convey more information within a limited time.
[0093] To quantify the information content of the SLM, we used the cross-entropy H of the speech-text content generated by the SLM relative to a pre-trained text language model. The text language model can be considered as an approximation of the true distribution of all meaningful text content. Therefore, treating each SLM with different acoustic BPE as an information source and assuming that speech recognition does not introduce much error, we can consider the cross-entropy as a measure of the information contained in the synthesized speech. The cross-entropy is calculated as follows: First, we generate a set of prompt continuations {u1, ..., u1} from the SLM (with θ as a parameter). n Then, we transcribe it into text using ASR. Next, for each continuation, we compute the log probability of its transcribed text using a pre-trained text language model (parameter γ). Cross-entropy H(SLM) θ TextLM γ The value is obtained by averaging the negative logarithmic probabilities of all generated continuous text.
[0094] {u1,...,u n}~p(u|θ) (7)
[0095]
[0096] In our experiments, we randomly selected the first 3 seconds of the LibriTTS test-clean subset as prompts and generated 150 continuations, each 20 seconds long. We used Whisper to transcribe the speech into text and calculated the cross-entropy using a pre-trained text-language model. The results are shown below. Figure 6 .
[0097] Please refer to Figure 6 It shows a comparison of cross-entropy for four acoustic BPE (aBPE) variants.
[0098] Using acoustic BPE increases the amount of information that SLM can transmit in a limited time, indicating that acoustic BPE helps to improve the content richness of generated speech.
[0099] Rescoring of prosodic synthesized speech using SLM
[0100] The advantages of acoustic BPE discussed above highlight its potential applications in speech generation tasks. Here, we introduce a novel rescore method that selects the best synthesized speech from numerous candidate speech samples generated by a prosodic TTS system. This method aims to strike a balance between preserving diversity and ensuring the naturalness of the generated speech.
[0101] We conducted our experiments using a prosodic TTS model trained on the LibriTTS train-960 subset. We randomly selected 106 sentences from the LibriTTS test-clean subset, synthesizing each sentence five times using the prosodic TTS model. Before using SLM rescore, we conducted a subjective listening test, having 10 listeners rank the five synthesized speech samples of the same sentence in each group according to naturalness. Subsequently, as mentioned earlier, we used SLM to rescore each group of five synthesized speech samples. To quantitatively evaluate the degree of agreement between rescore selection and human preferences, we calculated the top-x accuracy, i.e., the success rate of the rescore selection appearing in the top x positions according to human preferences. Figure 7 The accuracy of the top 1 to top 3 of four acoustic BPE variants was compared.
[0102] Please refer to Figure 7 It shows the rescoring accuracy of various acoustic BPEs (aBPEs).
[0103] As shown above, the rescore selection aligns with human preferences. The use of an acoustic BPE further improves rescore performance. Additionally, we conducted two preference tests to validate the effectiveness of the rescore and acoustic BPE. The first preference test involved random selection and rescore selection using an acoustic BPE 10k from an SLM. The second test involved selection from two acoustic BPE variants: no acoustic BPE and an acoustic BPE 10k. The results are as follows... Figure 8 As shown.
[0104] Figure 8 A preference test is shown. Among them, Figure 8 The figure above shows a comparison between random selection and the use of acoustic BPE (10k) (aBPE10k); Figure 8 The figure below shows a comparison between not using acoustic BPE (w\o aBPE) and aBPE 10k. The preference test above further confirms the effectiveness of the rescoring method and highlights the potential of acoustic BPE in speech generation tasks.
[0105] In this application, we introduce acoustic BPE into the speech generation task. We trained a speech language model using acoustic BPE and investigated its properties in depth. These studies revealed significant advantages of acoustic BPE. First, acoustic BPE can shorten sequences, facilitating sequence modeling and accelerating autoregressive inference. Second, by utilizing the morphological information present in the feature sequences, it alleviates the burden on the speech language model to construct feature relevance, thereby enhancing its semantic capture ability and the diversity and richness of the generated content. Furthermore, we demonstrate the application of acoustic BPE in a novel TTSrescore method that selects the optimal synthesis from multiple prosodic TTS synthesizers, achieving a balance between diversity and naturalness. Experimental results provide evidence for the effectiveness of acoustic BPE. These findings suggest the possibility of applying acoustic BPE to other speech generation tasks such as text-to-speech synthesis, which can be explored in future research.
[0106] In other embodiments, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the audio coding method for speech generation task in any of the above method embodiments;
[0107] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:
[0108] The semantic features of the audio waveform are extracted using a pre-trained model, and the semantic features are discretized into a discrete feature sequence.
[0109] The discrete feature sequences are mapped to Unicode strings, and then the BPE model is trained on the Unicode strings;
[0110] The trained byte-pair encoding model is used to encode the Unicode string to obtain acoustic BPE features.
[0111] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function, and the stored data area may store data created based on the use of the means for audio encoding for the speech generation task. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely disposed relative to a processor, which can be connected via a network to the means for audio encoding for the speech generation task. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0112] This invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the above-described audio encoding methods for speech generation tasks.
[0113] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 9 As shown, the device includes one or more processors 910 and a memory 920. Figure 9 Taking a processor 910 as an example, the device for the audio encoding method for the speech generation task may further include an input device 930 and an output device 940. The processor 910, memory 920, input device 930, and output device 940 can be connected via a bus or other means. Figure 9 Taking a bus connection as an example, the memory 920 is the aforementioned non-volatile computer-readable storage medium. The processor 910 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 920, thereby implementing the audio encoding method for speech generation tasks described in the above method embodiment. The input device 930 can receive input digital or character information, as well as key signal inputs related to user settings and function control of the device for audio encoding for speech generation tasks. The output device 940 may include a display device such as a screen.
[0114] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.
[0115] In one embodiment, the above-described electronic device is applied in an audio encoding apparatus for a speech generation task, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:
[0116] The semantic features of the audio waveform are extracted using a pre-trained model, and the semantic features are discretized into a discrete feature sequence.
[0117] The discrete feature sequences are mapped to Unicode strings, and then the BPE model is trained on the Unicode strings;
[0118] The trained byte-pair encoding model is used to encode the Unicode string to obtain acoustic BPE features.
[0119] The electronic devices described in this application exist in various forms, including but not limited to:
[0120] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.
[0121] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.
[0122] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players (e.g., iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.
[0123] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.
[0124] (5) Other electronic devices with data interaction functions.
[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.
[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An audio encoding method for a speech generation task, comprising: The semantic features of the audio waveform are extracted using a pre-trained model, and the semantic features are discretized into a discrete feature sequence. The discrete feature sequences are mapped to Unicode strings, and then the BPE model is trained on the Unicode strings; The trained byte-pair encoding model is used to encode the Unicode string to obtain acoustic BPE features; The pre-trained model is HuBERT, and the step of discretizing the semantic features into a discrete feature sequence includes using the k-means clustering method to discretize the semantic features into a discrete feature sequence. The process of training a speech language model using the acoustic BPE features, wherein the speech language model is a Decoder-only Transformer, includes: Encode the audio waveform into an acoustic BPE sequence; The speech language model is trained using the acoustic BPE sequence; The likelihood of multiple TTS synthesis results of the prosodic TTS system is calculated using the speech language model, and the synthesis result with the highest likelihood among the multiple TTS synthesis results is selected.
2. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method of claim 1.
3. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method of claim 1.
Citation Information
Patent Citations
Speech synthesis method and device, electronic equipment and storage medium
CN115641834A
Phishing URL detection method and system based on pre-training language model
CN115994224A
