Text-to-speech generation method
By obtaining voice self-supervising features and discretization processing, combining the two-stage model architecture and non-autoregressive mask generation model, the existing text-to-speech system's slow generation speed and dependence on alignment information is solved, and efficient and accurate voice is achieved Synthesis, adapt to different languages and pronunciation characteristics.
Patent Information
- Application Number
- CN202411539820.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-10-31
AI Technical Summary
The existing text-to-speech system generates slowly and depends on the alignment information between text and speech, making it difficult to balance the generation of content and rhythm, especially in tone-rich languages, which may lead to information loss, affecting the quality of speech and the accuracy of rhythm.
By obtaining the self-supervised features of reference speech, discretization is performed to obtain the reference semantic marker sequence, and a two-stage model architecture is adopted: the first stage predicts the target semantic marker from text marker and reference semantic marker, and the second stage predicts the target semantic marker from target semantic marker. Acoustic marking, ultimately generating target speech, uses a non-autoregressive mask generation model and vector quantization method to improve synthesis quality and efficiency.
It improves the accuracy and nature of speech synthesis, simplifies the model training process, reduces the dependence on aligned information, adapts to different languages and speech characteristics, and significantly improves the synthesis speed and efficiency, avoids information loss.
Smart Images

Figure CN119049447A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a text-to-speech generation method, belonging to the technical field of speech generation. Background Art
[0002] In the existing technology, many text-to-speech (TTS) systems use autoregressive models, such as sequence-to-sequence models (Seq2Seq). These models predict token by token when generating speech, resulting in slow synthesis speed and being susceptible to long-distance dependency problems.
[0003] Some TTS systems rely on alignment information between text and speech, such as alignment of phonemes to speech frames. This reliance increases the complexity of the system, requires additional alignment tools or algorithms, and may not be flexible enough when dealing with different languages or speech characteristics.
[0004] In addition, traditional TTS systems usually use a single-stage model to generate speech directly from text. This model may not be sophisticated enough to handle complex speech characteristics and it is difficult to simultaneously generate content and prosody. Moreover, in the discretization process of semantic representation, especially in tonal-rich languages, existing technologies such as k-means methods may cause information loss, affecting speech quality and prosody accuracy. Summary of the invention
[0005] The present invention provides a text-to-speech generation method, which can solve the problems of low speech synthesis efficiency and reliance on alignment information in existing text-to-speech systems.
[0006] The present invention provides a text-to-speech generation method, the method comprising: S1, obtaining self-supervisory features of reference speech, and discretizing the self-supervisory features to obtain a reference semantic tag sequence; S2. Obtaining a target semantic tag sequence according to the text tag sequence and the reference semantic tag sequence; S3. Predict the target semantic tag sequence to obtain a multi-layer acoustic tag sequence, and generate a target speech according to the multi-layer acoustic tag sequence.
[0007] Optionally, the step of obtaining the self-supervisory features of the reference speech in S1 is specifically: The speech self-supervision model is used to obtain the self-supervision features of the reference speech.
[0008] Optionally, the step of discretizing the self-supervisory features in S1 to obtain a reference semantic tag sequence specifically includes: Encoding the self-supervisory features using an encoder in a speech semantic codec to obtain encoded features; The encoded features are discretized using a vector quantization module in a speech semantic codec to obtain a reference semantic tag sequence.
[0009] Optionally, S2 is specifically: According to the text tag sequence, the reference semantic tag sequence is predicted by using a text-to-semantic mask generation model to obtain a target semantic tag sequence.
[0010] Optionally, the step of predicting the target semantic tag sequence in S3 to obtain a multi-layer acoustic tag sequence is specifically: The target semantic tag sequence is conditionally predicted using a semantic-to-acoustic mask generation model to obtain a multi-layer acoustic tag sequence.
[0011] Optionally, the step of predicting the target semantic tag sequence in S3 to obtain a multi-layer acoustic tag sequence is specifically: The target semantic tag sequence is predicted using an adversarial neural network model to obtain a multi-layer acoustic tag sequence.
[0012] Optionally, generating the target speech according to the multi-layer acoustic marker sequence in S3 is specifically: The multi-layer acoustic marker sequence is reconstructed using a speech acoustic codec to generate the target speech.
[0013] Optionally, the speech acoustic codec uses a residual vector quantization method to perform discretization processing.
[0014] Optionally, the text-to-semantic mask generation model is constructed using a non-autoregressive mask generation modeling approach.
[0015] Optionally, the semantic-to-acoustic mask generation model is constructed using a non-autoregressive mask generation modeling approach. The beneficial effects that the present invention can produce include: The text-to-speech generation method provided by the present invention obtains reference semantic tags by referring to the self-supervisory features of speech, and then adopts a two-stage model architecture. In the first stage, the target semantic tags are predicted from the text tags and the reference semantic tags, and in the second stage, the acoustic tags are predicted from the target semantic tags, and finally the target speech is generated. This staged approach enables the model to focus on the generation of content and rhythm respectively, improves the accuracy and naturalness of the synthesis, and does not rely on the alignment information between text and speech in both stages, which simplifies the training process of the model and reduces the dependence on external alignment tools. This design makes the model more flexible and can adapt to different language and speech characteristics.
[0016] The present invention adopts a non-autoregressive mask generation model for text-to-speech synthesis, which significantly improves the synthesis speed and efficiency compared with the traditional autoregressive model. At the same time, the non-autoregressive mask generation model avoids the dependency of sequence generation, enables the model to be processed in parallel, and thus accelerates the synthesis process.
[0017] The speech semantic codec provided by the present invention minimizes information loss through a vector quantization method and improves the quality of reference semantic tags. The design of the codec helps to accurately reconstruct high-quality speech, especially in tonal-rich languages. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 A block diagram of a text-to-speech generation method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a speech semantic codec provided by an embodiment of the present invention; Figure 3 A schematic diagram of a text-to-semantic mask generation model provided in an embodiment of the present invention; Figure 4 A schematic diagram of a semantic-to-acoustic mask generation model provided in an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a speech acoustic codec provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0019] The present invention is described in detail below in conjunction with embodiments, but the present invention is not limited to these embodiments.
[0020] The embodiment of the present invention provides a text-to-speech generation method, such as Figures 1 to 5 As shown, the method includes: S1. Obtain the self-supervisory features of the reference speech, and discretize the self-supervisory features to obtain a reference semantic tag sequence.
[0021] Specifically, refer to Figure 2 As shown, first, the speech self-supervision model is used to obtain the self-supervision features of the reference speech; then, the encoder in the speech semantic codec is used to encode the self-supervision features to obtain the encoded features; finally, the vector quantization module in the speech semantic codec is used to discretize the encoded features to obtain the reference semantic tag sequence.
[0022] During the training phase, the self-supervised features are passed through the encoder, vector quantization module and decoder to obtain reconstructed self-supervised features. The smaller the difference between the reconstructed self-supervised features and the self-supervised features before reconstruction, the smaller the information loss, and the better the effect of the speech semantic codec.
[0023] The speech semantic codec obtains reference semantic tags by discretizing the self-supervised features extracted by the speech self-supervised model (SSL), and reconstructs the reference semantic tags from the speech self-supervised model SSL using a vector quantization codebook. The encoder and decoder minimize information loss by optimizing the reconstruction loss, codebook loss, and commitment loss.
[0024] In practical applications, the above speech self-supervision model can use the w2v-bert-2.0 model.
[0025] S2. Obtain a target semantic tag sequence according to the text tag sequence and the reference semantic tag sequence.
[0026] refer to Figure 3 As shown, specifically: according to the text tag sequence, the reference semantic tag sequence is predicted using the text-to-semantic mask generation model to obtain the target semantic tag sequence.
[0027] Among them, the text-to-semantic mask generation model is constructed using a non-autoregressive mask generation modeling approach.
[0028] The text-to-semantic mask generation model uses a non-autoregressive mask generation transformer, does not require any text-to-speech alignment information, and randomly extracts the prefix of the reference semantic tag sequence as a hint, and combines the text tag sequence for conditional prediction. In the inference stage, the model can generate a target semantic tag sequence of any specified length based on the text tag sequence and the reference semantic tag sequence.
[0029] S3. Predict the target semantic tag sequence to obtain a multi-layer acoustic tag sequence, and generate the target speech according to the multi-layer acoustic tag sequence.
[0030] Specifically, refer to Figure 4 As shown, the semantic-to-acoustic mask generation model is used to conditionally predict the target semantic tag sequence to obtain a multi-layer acoustic tag sequence.
[0031] Among them, the semantic to acoustic mask generation model is constructed using a non-autoregressive mask generation modeling approach.
[0032] The semantic-to-acoustic mask generation model is based on a mask generation encoder-decoder transformer, and conditionally predicts the target semantic tag sequence to generate a multi-layer acoustic tag sequence. During the training process, the model randomly selects a layer of the acoustic tag sequence for mask prediction, and generates each layer of acoustic tags from coarse to fine through iterative parallel decoding during the inference phase.
[0033] In practical applications, the adversarial neural network model can also be used to replace the semantic to acoustic mask generation model to obtain a multi-layer acoustic label sequence.
[0034] For further reference, Figure 5 As shown, the multi-layer acoustic marker sequence is reconstructed using a speech acoustic codec to generate the target speech.
[0035] Among them, the speech acoustic codec uses a residual vector quantization method for discretization processing.
[0036] The speech acoustic codec uses the residual vector quantization (RVQ) method to compress the speech waveform with a sampling rate of 24K into 12 layers of discrete tokens, each layer contains 1024 entries, and the dimension of each entry is 8.
[0037] Preferably, the reconstruction quality can be further enhanced by using a time domain / frequency domain discriminator to determine whether the target speech is real or fake.
[0038] The present invention obtains reference semantic tags by using the self-supervisory features of reference speech, and then adopts a two-stage model architecture. In the first stage, the target semantic tags are predicted from the text tags and the reference semantic tags. In the second stage, the acoustic tags are predicted from the target semantic tags, and finally the target speech is generated. This staged approach enables the model to focus on the generation of content and rhythm respectively, improving the accuracy and naturalness of the synthesis, and does not rely on the alignment information between text and speech in both stages, which simplifies the training process of the model and reduces the dependence on external alignment tools. This design makes the model more flexible and can adapt to different language and speech characteristics.
[0039] The present invention adopts a non-autoregressive mask generation model for text-to-speech synthesis, which significantly improves the synthesis speed and efficiency compared with the traditional autoregressive model. At the same time, the non-autoregressive mask generation model avoids the dependency of sequence generation, enables the model to be processed in parallel, and thus accelerates the synthesis process.
[0040] The speech semantic codec provided by the present invention minimizes information loss through a vector quantization method and improves the quality of reference semantic tags. The design of the codec helps to accurately reconstruct high-quality speech, especially in tonal-rich languages.
[0041] The text-to-speech generation method provided by the present invention can realize high-quality zero-sample speech synthesis. Compared with the previous autoregression-based method, the present invention can improve the generation efficiency on the basis of having better effect. The text-to-speech generation method proposed by the present invention can freely control the total duration of the generated speech.
[0042] The above are only a few embodiments of the present application and do not constitute any form of limitation to the present application. Although the present application is disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technician familiar with the profession, without departing from the scope of the technical solution of the present application, using the technical content disclosed above to make slight changes or modifications are equivalent to equivalent implementation cases and fall within the scope of the technical solution.
Claims
1. A text-to-speech generation method, It is characterized in that The method comprises: S1, obtaining self-supervisory features of reference speech, and discretizing the self-supervisory features to obtain a reference semantic tag sequence; S2. Obtaining a target semantic tag sequence according to the text tag sequence and the reference semantic tag sequence; S3. Predict the target semantic tag sequence to obtain a multi-layer acoustic tag sequence, and generate a target speech according to the multi-layer acoustic tag sequence.
2. The method according to claim 1, It is characterized in that The self-supervisory features of the reference speech obtained in S1 are specifically: The speech self-supervision model is used to obtain the self-supervision features of the reference speech.
3. The method according to claim 1, It is characterized in that The discretization processing of the self-supervisory features in S1 to obtain a reference semantic tag sequence specifically includes: Encoding the self-supervisory features using an encoder in a speech semantic codec to obtain encoded features; The encoded features are discretized using a vector quantization module in a speech semantic codec to obtain a reference semantic tag sequence.
4. The method according to claim 1, It is characterized in that The S2 is specifically: According to the text tag sequence, the reference semantic tag sequence is predicted by using a text-to-semantic mask generation model to obtain a target semantic tag sequence.
5. The method according to claim 1, It is characterized in that The step S3 predicts the target semantic tag sequence to obtain a multi-layer acoustic tag sequence, specifically: The target semantic tag sequence is conditionally predicted using a semantic-to-acoustic mask generation model to obtain a multi-layer acoustic tag sequence.
6. The method according to claim 1, It is characterized in that The step S3 predicts the target semantic tag sequence to obtain a multi-layer acoustic tag sequence, specifically: The target semantic tag sequence is predicted using an adversarial neural network model to obtain a multi-layer acoustic tag sequence.
7. The method according to claim 1, It is characterized in that The step of generating the target speech according to the multi-layer acoustic marker sequence in S3 is specifically: The multi-layer acoustic marker sequence is reconstructed using a speech acoustic codec to generate the target speech.
8. The method according to claim 7, It is characterized in that The speech acoustic codec adopts a residual vector quantization method to perform discretization processing.
9. The method according to claim 4, It is characterized in that The text-to-semantic mask generation model is constructed using a non-autoregressive mask generation modeling approach.
10. The method according to claim 5, It is characterized in that The semantic-to-acoustic mask generation model is constructed using a non-autoregressive mask generation modeling approach.
Citation Information
Patent Citations
Speech generation method and device based on pre-training language model, equipment and medium
CN116364055A
Non-autoregression modeling method based on discretization self-supervision representation enhancement
CN116935826A
Speech synthesis method, electronic equipment and storage medium
CN117012177A
Audio coding method for voice generation task, electronic equipment and storage medium
CN117253493A
Speech synthesis model training method and device and speech synthesis method and device
CN117316139A
Cited By
Target audio generation method and device, electronic equipment and medium
CN120164486A
Target audio generation method and apparatus, electronic device, and medium
CN120164486B
