A text-to-speech generation method

Abstract

The present invention discloses a text-to-speech generation method, which belongs to the field of speech generation technology and can solve the problems of low speech synthesis efficiency and dependence on alignment information in existing text-to-speech systems. The method comprises: S1, obtaining self-supervisory features of reference speech, and discretizing the self-supervisory features to obtain a reference semantic tag sequence; S2, obtaining a target semantic tag sequence according to the text tag sequence and the reference semantic tag sequence; S3, predicting the target semantic tag sequence to obtain a multi-layer acoustic tag sequence, and generating a target speech according to the multi-layer acoustic tag sequence. The present invention is used to generate a target speech.

Description

Technical Field

[0001] The invention relates to a text-to-speech generation method, belonging to the technical field of speech generation. Background Art

[0002] In the existing technology, many text-to-speech (TTS) systems use autoregressive models, such as sequence-to-sequence models (Seq2Seq). These models predict token by token when generating speech, resulting in slow synthesis speed and being susceptible to long-distance dependency problems.

[0003] Some TTS systems rely on alignment information between text and speech, such as alignment of phonemes to speech frames. This reliance increases the complexity of the system, requires additional alignment tools or algorithms, and may not be flexible enough when dealing with different languages ​​or speech characteristics.

[0004] In addition, traditional TTS systems usually use a single-stage model to generate speech directly from text. This model may not be sophisticated enough to handle complex speech characteristics and it is difficult to simultaneously generate content and prosody. Moreover, in the discretization process of semantic representation, especially in tonal-rich languages, existing technologies such as k-means methods may cause information loss, affecting speech quality and prosody accuracy. Summary of the invention

[0005] The present invention provides a text-to-speech generation method, which can solve the problems of low speech synthesis efficiency and reliance on alignment information in existing text-to-speech systems.

[0006] The present invention provides a text-to-speech generation method, the method comprising:

[0007] S1, obtaining self-supervisory features of reference speech, and discretizing the self-supervisory features to obtain a reference semantic tag sequence;

[0008] S2. Obtaining a target semantic tag sequence according to the text tag sequence and the reference semantic tag sequence;

[0009] S3. Predict the target semantic tag sequence to obtain a multi-layer acoustic tag sequence, and generate a target speech according to the multi-layer acoustic tag sequence.

[0010] Optionally, the step of obtaining the self-supervisory features of the reference speech in S1 is specifically:

[0011] The speech self-supervision model is used to obtain the self-supervision features of the reference speech.

[0012] Optionally, the step of discretizing the self-supervisory features in S1 to obtain a reference semantic tag sequence specifically includes:

[0013] Encoding the self-supervisory features using an encoder in a speech semantic codec to obtain encoded features;

[0014] The encoded features are discretized using a vector quantization module in a speech semantic codec to obtain a reference semantic tag sequence.

[0015] Optionally, S2 is specifically:

[0016] According to the text tag sequence, the reference semantic tag sequence is predicted by using a text-to-semantic mask generation model to obtain a target semantic tag sequence.

[0017] Optionally, the step of predicting the target semantic tag sequence in S3 to obtain a multi-layer acoustic tag sequence is specifically:

[0018] The target semantic tag sequence is conditionally predicted using a semantic-to-acoustic mask generation model to obtain a multi-layer acoustic tag sequence.

[0019] Optionally, the step of predicting the target semantic tag sequence in S3 to obtain a multi-layer acoustic tag sequence is specifically:

[0020] The target semantic tag sequence is predicted using an adversarial neural network model to obtain a multi-layer acoustic tag sequence.

[0021] Optionally, generating the target speech according to the multi-layer acoustic marker sequence in S3 is specifically:

[0022] The multi-layer acoustic marker sequence is reconstructed using a speech acoustic codec to generate the target speech.

[0023] Optionally, the speech acoustic codec uses a residual vector quantization method to perform discretization processing.

[0024] Optionally, the text-to-semantic mask generation model is constructed using a non-autoregressive mask generation modeling approach.

[0025] Optionally, the semantic-to-acoustic mask generation model is constructed using a non-autoregressive mask generation modeling approach.

[0026] The beneficial effects that the present invention can produce include:

[0027] The text-to-speech generation method provided by the present invention obtains reference semantic tags by referring to the self-supervisory features of speech, and then adopts a two-stage model architecture. In the first stage, the target semantic tags are predicted from the text tags and the reference semantic tags, and in the second stage, the acoustic tags are predicted from the target semantic tags, and finally the target speech is generated. This staged approach enables the model to focus on the generation of content and rhythm respectively, improves the accuracy and naturalness of the synthesis, and does not rely on the alignment information between text and speech in both stages, which simplifies the training process of the model and reduces the dependence on external alignment tools. This design makes the model more flexible and can adapt to different language and speech characteristics.

[0028] The present invention adopts a non-autoregressive mask generation model for text-to-speech synthesis, which significantly improves the synthesis speed and efficiency compared with the traditional autoregressive model. At the same time, the non-autoregressive mask generation model avoids the dependency of sequence generation, enables the model to be processed in parallel, and thus accelerates the synthesis process.

[0029] The speech semantic codec provided by the present invention minimizes information loss through a vector quantization method and improves the quality of reference semantic tags. The design of the codec helps to accurately reconstruct high-quality speech, especially in tonal-rich languages. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A block diagram of a text-to-speech generation method provided by an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of the structure of a speech semantic codec provided by an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of a text-to-semantic mask generation model provided in an embodiment of the present invention;

[0033] Figure 4 A schematic diagram of a semantic-to-acoustic mask generation model provided in an embodiment of the present invention;

[0034] Figure 5 A schematic diagram of the structure of a speech acoustic codec provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below in conjunction with embodiments, but the present invention is not limited to these embodiments.

[0036] The embodiment of the present invention provides a text-to-speech generation method, such as Figures 1 to 5 As shown, the method includes:

[0037] S1. Obtain the self-supervisory features of the reference speech, and discretize the self-supervisory features to obtain a reference semantic tag sequence.

[0038] Specifically, refer to Figure 2 As shown, first, the speech self-supervision model is used to obtain the self-supervision features of the reference speech; then, the encoder in the speech semantic codec is used to encode the self-supervision features to obtain the encoded features; finally, the vector quantization module in the speech semantic codec is used to discretize the encoded features to obtain the reference semantic tag sequence.

[0039] During the training phase, the self-supervised features are passed through the encoder, vector quantization module and decoder to obtain reconstructed self-supervised features. The smaller the difference between the reconstructed self-supervised features and the self-supervised features before reconstruction, the smaller the information loss, and the better the effect of the speech semantic codec.

[0040] The speech semantic codec obtains reference semantic tags by discretizing the self-supervised features extracted by the speech self-supervised model (SSL), and reconstructs the reference semantic tags from the speech self-supervised model SSL using a vector quantization codebook. The encoder and decoder minimize information loss by optimizing the reconstruction loss, codebook loss, and commitment loss.

[0041] In practical applications, the above speech self-supervision model can use the w2v-bert-2.0 model.

[0042] S2. Obtain a target semantic tag sequence according to the text tag sequence and the reference semantic tag sequence.

[0043] refer to Figure 3 As shown, specifically: according to the text tag sequence, the reference semantic tag sequence is predicted using the text-to-semantic mask generation model to obtain the target semantic tag sequence.

[0044] Among them, the text-to-semantic mask generation model is constructed using a non-autoregressive mask generation modeling approach.

[0045] The text-to-semantic mask generation model uses a non-autoregressive mask generation transformer, does not require any text-to-speech alignment information, and randomly extracts the prefix of the reference semantic tag sequence as a hint, and combines the text tag sequence for conditional prediction. In the inference stage, the model can generate a target semantic tag sequence of any specified length based on the text tag sequence and the reference semantic tag sequence.

[0046] S3. Predict the target semantic tag sequence to obtain a multi-layer acoustic tag sequence, and generate the target speech according to the multi-layer acoustic tag sequence.

[0047] Specifically, refer to Figure 4 As shown, the semantic-to-acoustic mask generation model is used to conditionally predict the target semantic tag sequence to obtain a multi-layer acoustic tag sequence.

[0048] Among them, the semantic to acoustic mask generation model is constructed using a non-autoregressive mask generation modeling approach.

[0049] The semantic-to-acoustic mask generation model is based on a mask generation encoder-decoder transformer, and conditionally predicts the target semantic tag sequence to generate a multi-layer acoustic tag sequence. During the training process, the model randomly selects a layer of the acoustic tag sequence for mask prediction, and generates each layer of acoustic tags from coarse to fine through iterative parallel decoding during the inference phase.

[0050] In practical applications, the adversarial neural network model can also be used to replace the semantic to acoustic mask generation model to obtain a multi-layer acoustic label sequence.

[0051] For further reference, Figure 5 As shown, the multi-layer acoustic marker sequence is reconstructed using a speech acoustic codec to generate the target speech.

[0052] Among them, the speech acoustic codec uses a residual vector quantization method for discretization processing.

[0053] The speech acoustic codec uses the residual vector quantization (RVQ) method to compress the speech waveform with a sampling rate of 24K into 12 layers of discrete tokens, each layer contains 1024 entries, and the dimension of each entry is 8.

[0054] Preferably, the reconstruction quality can be further enhanced by using a time domain / frequency domain discriminator to determine whether the target speech is real or fake.

[0055] The present invention obtains reference semantic tags by using the self-supervisory features of reference speech, and then adopts a two-stage model architecture. In the first stage, the target semantic tags are predicted from the text tags and the reference semantic tags. In the second stage, the acoustic tags are predicted from the target semantic tags, and finally the target speech is generated. This staged approach enables the model to focus on the generation of content and rhythm respectively, improving the accuracy and naturalness of the synthesis, and does not rely on the alignment information between text and speech in both stages, which simplifies the training process of the model and reduces the dependence on external alignment tools. This design makes the model more flexible and can adapt to different language and speech characteristics.

[0056] The present invention adopts a non-autoregressive mask generation model for text-to-speech synthesis, which significantly improves the synthesis speed and efficiency compared with the traditional autoregressive model. At the same time, the non-autoregressive mask generation model avoids the dependency of sequence generation, enables the model to be processed in parallel, and thus accelerates the synthesis process.

[0057] The speech semantic codec provided by the present invention minimizes information loss through a vector quantization method and improves the quality of reference semantic tags. The design of the codec helps to accurately reconstruct high-quality speech, especially in tonal-rich languages.

[0058] The text-to-speech generation method provided by the present invention can realize high-quality zero-sample speech synthesis. Compared with the previous autoregression-based method, the present invention can improve the generation efficiency on the basis of having better effect. The text-to-speech generation method proposed by the present invention can freely control the total duration of the generated speech.

[0059] The above are only a few embodiments of the present application and do not constitute any form of limitation to the present application. Although the present application is disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technician familiar with the profession, without departing from the scope of the technical solution of the present application, using the technical content disclosed above to make slight changes or modifications are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A text-to-speech generation method, characterized in that: The method comprises: S1, obtaining self-supervisory features of reference speech, and encoding the self-supervisory features using an encoder in a speech semantic codec to obtain encoded features, and discretizing the encoded features using a vector quantization module in the speech semantic codec to obtain a reference semantic tag sequence; S2. Obtaining a target semantic tag sequence according to the text tag sequence and the reference semantic tag sequence; S3. Predict the target semantic tag sequence to obtain a multi-layer acoustic tag sequence, and generate a target speech according to the multi-layer acoustic tag sequence.

2. The method according to claim 1, characterized in that The self-supervisory features of the reference speech obtained in S1 are specifically: The speech self-supervision model is used to obtain the self-supervision features of the reference speech.

3. The method according to claim 1, characterized in that The S2 is specifically: According to the text tag sequence, the reference semantic tag sequence is predicted by using a text-to-semantic mask generation model to obtain a target semantic tag sequence.

4. The method according to claim 1, characterized in that: The step S3 predicts the target semantic tag sequence to obtain a multi-layer acoustic tag sequence, specifically: The target semantic tag sequence is conditionally predicted using a semantic-to-acoustic mask generation model to obtain a multi-layer acoustic tag sequence.

5. The method according to claim 1, characterized in that The step S3 predicts the target semantic tag sequence to obtain a multi-layer acoustic tag sequence, specifically: The target semantic tag sequence is predicted using an adversarial neural network model to obtain a multi-layer acoustic tag sequence.

6. The method according to claim 1, characterized in that The step of generating the target speech according to the multi-layer acoustic marker sequence in S3 is specifically: The multi-layer acoustic marker sequence is reconstructed using a speech acoustic codec to generate the target speech.

7. The method according to claim 6, characterized in that The speech acoustic codec adopts a residual vector quantization method to perform discretization processing.

8. The method according to claim 3, characterized in that The text-to-semantic mask generation model is constructed using a non-autoregressive mask generation modeling approach.

9. The method according to claim 4, characterized in that The semantic-to-acoustic mask generation model is constructed using a non-autoregressive mask generation modeling approach.

Images