Speech synthesis method and apparatus, device, medium, product

Abstract

The application provides a speech synthesis method and device, equipment, medium and product. The semantic module is used to extract a semantic token sequence of a target text, semantic information and prosodic features can be deeply identified, and target audio that fits the text sentiment and prosody can be accurately generated. Moreover, the token fusion vector is used as the autoregressive modeling unit of the semantic module, so that the information amount of multiple tokens can be covered in each reasoning step of the semantic module, the reasoning step number is greatly reduced, and the model reasoning efficiency is improved. Meanwhile, the causal constraint and context dependency capability of autoregression are completely retained by maintaining the causal constraint and dependency of the token fusion vector layer between each generation step, so that the reasoning delay is effectively reduced under the premise of guaranteeing the semantic coherence and prosodic naturalness of the synthesized speech, and the speech synthesis quality and reasoning efficiency are taken into account.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, specifically providing a speech synthesis method, apparatus, device, medium, and product. Background Technology

[0002] TTS (Text-to-Speech) technology refers to converting text content into natural speech that can be recognized by the human ear. Currently, TTS is widely used in products such as intelligent voice assistants, navigation, and smart speakers. However, TTS systems often lack an understanding of the emotion, context, and complex pragmatic information in the text, resulting in a lack of rhythm and emotion in the synthesized speech, a mechanical delivery, and poor speech generation quality. Summary of the Invention

[0003] To address at least one of the aforementioned technical problems, embodiments of this specification provide a speech synthesis method and apparatus, an electronic device, a storage medium, and a computer program product.

[0004] Firstly, the embodiments of this specification provide a speech synthesis method, including: A semantic word sequence is predicted based on the target text using autoregressive inference. The semantic word sequence implicitly encodes at least one of the prosodic information, sentiment information, and speech style information corresponding to the target text. In each non-first step of word generation in the autoregressive inference, the word fusion vector of the current generation step is predicted and output based on the target text and the previously generated word fusion vectors. The word fusion vector is then decoded to obtain the word subsequence corresponding to the word fusion vector. The word subsequence includes k words, k≥2. The word subsequences obtained from each word generation step are concatenated to obtain the semantic word sequence. Based on the semantic word sequence, the phoneme sequence of the target text, and the timbre embedding vector, the target speech corresponding to the target text is predicted and generated. The timbre embedding vector is used to characterize the timbre information of the target speech. The timbre embedding vector is determined from a pre-trained preset timbre library based on the input timbre identifier, or the timbre embedding vector is determined based on the input reference audio data.

[0005] In some possible implementations, the step of predicting semantic lexical sequences based on target text using autoregressive inference includes: inputting the target text into the semantic module of a speech synthesis model, performing autoregressive inference on the target text through the backbone model included in the semantic module to obtain a lexical fusion vector for each lexical generation step, and decoding the lexical fusion vector through the lexical decoding module included in the semantic module to obtain a lexical subsequence corresponding to the lexical fusion vector.

[0006] In some possible implementations, predicting and generating the target speech corresponding to the target text based on the semantic word sequence, the phoneme sequence of the target text, and the timbre embedding vector includes: the acoustic module of the speech synthesis model predicting and generating the target speech based on the semantic word sequence, the phoneme sequence of the target text, and the timbre embedding vector.

[0007] In some possible implementations, the process by which the acoustic module generates the target speech based on the semantic lexical sequence includes: Whenever the number of words decoded by the word decoding module reaches a first number, the first number of words are input into the acoustic module as a word stream block; The acoustic module predicts and outputs an audio segment corresponding to the current lexical streaming block based on the current lexical streaming block and the lexical streaming blocks above and / or below the current lexical streaming block, and splices the audio segments in the generation order to obtain the target speech.

[0008] In some possible implementations, the method further includes: The acoustic module performs window function weighted superposition processing on the transition region of two adjacent audio segments to obtain the target speech.

[0009] In some possible implementations, the acoustic module includes a text encoder, a streaming model, and an audio decoder; The text encoder is used to extract and fuse features from the semantic word sequence and the phoneme sequence of the target text to obtain multimodal fusion features; The streaming model generates target speech distribution features based on the multimodal fusion features and the timbre embedding vector. The timbre embedding vector is determined from a pre-trained preset timbre library based on the input timbre identifier, or the timbre embedding vector is determined based on the input reference audio data. The audio decoder generates the target speech based on the target speech distribution features and the timbre embedding vector.

[0010] In some possible implementations, the text encoder includes a first convolutional layer, a second convolutional layer, and a cross-attention fusion layer; The first convolutional layer is used to extract phoneme sequence features from the target text to obtain a phoneme embedding vector; The second convolutional layer is used to extract semantic embedding vectors from the semantic word sequence features; The cross-attention fusion layer is used to perform cross-attention fusion on the phoneme embedding vector and the semantic embedding vector to obtain the multimodal fusion feature.

[0011] In some possible implementations, the training process of the semantic module includes: Obtain sample data, which includes text samples and their corresponding sample word sequences, wherein the sample word sequences include multiple words arranged in chronological order; The sample word sequence is grouped according to word sequence to obtain multiple word subsequences, each word subsequence including k words; Each subsequence of words contains k words, which are then input into the word encoding module. The word encoding module is used to fuse the words to obtain a word fusion vector. Using the text sample and the historical word fusion vectors preceding the current generation step as input, the backbone model is trained to predict and output the word fusion vector of the current generation step. Furthermore, using the word fusion vectors as input, the word decoding module is trained to predict and output the word subsequences corresponding to each word fusion vector.

[0012] In some possible implementations, the target text contains one or more phonetic annotations, which are embedded in the original text in a preset format to specify the pronunciation of the corresponding character.

[0013] In some possible implementations, the acoustic module predicts and outputs an audio segment corresponding to the current lexical streaming block based on the current lexical streaming block and the lexical streaming blocks preceding and / or following the current lexical streaming block, including: The acoustic module acquires a second number of historical lexical units preceding the current lexical unit streaming block and / or a third number of future lexical units following the current lexical unit streaming block; The acoustic module decodes the current word stream block, the historical words, and / or the future words together, and only outputs the audio segment corresponding to the current word stream block.

[0014] In some possible implementations, the training process of the acoustic module includes: Input the text sample into the pre-trained semantic module to obtain the semantic word sequence output by the semantic module; The semantic word sequence and the phoneme sequence corresponding to the text sample are input into the acoustic module to be trained. The audio corresponding to the text sample is used as the supervision label. The acoustic module is trained to predict the target speech based on the semantic word sequence and the phoneme sequence.

[0015] Secondly, embodiments of this specification provide a speech synthesis apparatus, including a speech synthesis model, the speech synthesis model being configured to generate and output corresponding target speech based on target text; The speech synthesis model includes a semantic module and an acoustic module. The semantic module is used to convert the target text into a semantic word sequence containing semantic information, and the acoustic module generates the target speech based on the semantic word sequence. The semantic module includes a backbone model based on autoregressive inference and a lexical decoding module connected to the backbone model. The semantic module is configured as follows: In each lexical generation step, the backbone model predicts and outputs the lexical fusion vector for the current generation step based on the target text and the previously generated lexical fusion vectors. The word decoding module decodes the word fusion vector to obtain the word subsequence corresponding to the word fusion vector. The word subsequence includes k words, where k ≥ 2. By concatenating the subsequences of each lexical generation step, the semantic lexical sequence is obtained.

[0016] Thirdly, embodiments of this specification provide an electronic device, including: processor; The memory stores computer instructions that cause the processor to perform the method described in any of the above embodiments.

[0017] Fourthly, embodiments of this specification provide a storage medium storing computer instructions for implementing the methods described in any of the above embodiments.

[0018] Fifthly, embodiments of this specification provide a computer program product for implementing the methods described in any of the above embodiments.

[0019] The speech synthesis method described in this specification utilizes a semantic module to extract semantic word sequences from the target text, enabling deep recognition of semantic information and prosodic features, thereby accurately generating target audio that matches the emotion and prosody of the text. Furthermore, by using word fusion vectors as autoregressive modeling units of the semantic module, each inference step of the semantic module can cover the information of multiple words, significantly reducing the number of inference steps and improving model inference efficiency. Simultaneously, causal constraints are maintained between each generation step at the word fusion vector level, fully preserving the autoregressive causal constraints and contextual dependencies. Therefore, while ensuring the semantic coherence and prosodic naturalness of the synthesized speech, inference latency is effectively reduced, balancing speech synthesis quality and inference efficiency. Moreover, by injecting timbre embedding vectors into the streaming model and / or audio decoder, decoupling control of prosody, semantic content, and timbre is achieved during speech synthesis, allowing the same text to be played with any target timbre, and timbre switching does not affect pronunciation accuracy or prosodic naturalness. Attached Figure Description

[0020] Figure 1 This is an architecture diagram of the speech synthesis model in some embodiments of this specification.

[0021] Figure 2 This is an architecture diagram of the semantic module in some implementations of this specification.

[0022] Figure 3 This is a flowchart illustrating the training process of the semantic module in some implementations of this specification.

[0023] Figure 4 This describes the training principle of the semantic module in some implementations of this specification.

[0024] Figure 5 This is a flowchart illustrating the training process of the acoustic module in some implementations of this specification.

[0025] Figure 6 This is a flowchart of a speech synthesis method in some embodiments of this specification.

[0026] Figure 7 This is a flowchart of a speech synthesis method in some other embodiments of this specification.

[0027] Figure 8 This is a flowchart of a speech synthesis method in some other embodiments of this specification.

[0028] Figure 9 These are structural block diagrams of electronic devices in some embodiments of this specification. Detailed Implementation

[0029] TTS (Text-to-Speech) technology combines speech synthesis, artificial intelligence, and natural language generation technologies to convert plain text content into natural speech that can be recognized by the human ear. Currently, TTS has been widely used in products such as intelligent voice broadcasting, navigation, smart speakers, and voice assistants.

[0030] Traditional TTS models primarily learn the mapping relationship between text and acoustic features, lacking an understanding of textual emotion, context, and complex pragmatic information. This results in synthesized speech lacking rhythm and emotion, and exhibiting a mechanical delivery. Furthermore, they have poor cross-domain and cross-language generalization capabilities, failing to capture long-term contextual information for continuous long sentences and segments, leading to problems such as tone deviation from context or semantic breaks in speech delivery.

[0031] Based on this, some embodiments of this specification provide a speech synthesis method, speech synthesis apparatus, electronic device, storage medium, and computer program product, which aim to use language models to predict the semantic and emotional information of input text through autoregression, thereby generating speech with rich emotional expression and natural rhythm.

[0032] In the embodiments described in this specification, a text-to-speech (TTS) model is used to convert text into speech. That is, the input of the speech synthesis model includes at least the target text, which refers to the natural language text that needs to be converted into a speech waveform. The output of the speech synthesis model is the speech waveform after converting the target text, which represents the audio corresponding to the target text.

[0033] It is worth noting that, unlike traditional TTS models, the speech synthesis model provided in this manual includes two-stage model modules: a semantic module and an acoustic module.

[0034] Semantic models are used for text-to-semantic processing. The input to the semantic module includes at least the target text, and it is used to extract deep semantic information and prosodic features from the target text. Semantic information refers to the semantic understanding of the target text content, such as the recognition of text content and the technical field to which the text content belongs. Prosodic features refer to the rhythm of the text content when it is spoken, such as stress, pauses, speech rate, and emotion.

[0035] For example, in one scenario, the target text might be, "I'm sorry, I didn't understand your request before, which led to giving you the wrong answer." The text is an apology to the user, and the corresponding emotion should be gentle and reassuring, making the user feel comforted. In this example, the semantic model is used to extract the semantic and prosodic features contained in the target text, thereby identifying the emotional layers and realism of the speech.

[0036] The output of a semantic model is a sequence of semantic tokens, which includes discrete token representations obtained from feature extraction of the target text. In the field of natural language processing, a token is the smallest semantic unit after the text has been segmented, and also the smallest computational unit that the model can process. The semantic token sequence output by the semantic model is a discrete representation of the semantic and prosodic features contained in the target text.

[0037] In some implementations, the semantic module may include a language model based on autoregressive inference, such as a Large Language Model (LLM). Unlike traditional large language models, the output of the large language model in this specification is not natural language text, but rather a discrete sequence of semantic lexical units.

[0038] The acoustic module is used for Semantic-to-Audio processing. The input of the acoustic module at least includes the aforementioned semantic token sequence, or the input of the acoustic module includes the semantic token sequence and the phoneme sequence of the target text. The phoneme sequence refers to a sequence obtained by converting each character / word in the target text into the corresponding pronunciation phoneme and then splicing the phonemes according to the text order. For example, the phoneme sequence corresponding to the text "ni hao" can be "n i3 h ao3". The acoustic module outputs target speech with corresponding emotional expression and prosodic style according to the deep semantic information and prosodic features carried by the semantic token sequence as well as the pronunciation information carried by the phoneme sequence. In some implementations, the acoustic module may adopt a neural vocoder (such as HiFi-GAN, WaveNet, etc.) or a diffusion model.

[0039] It can be understood from the above that in the solution of the present specification, the semantic module is used to extract the semantic token sequence of the target text, which can deeply identify semantic information and prosodic features, such as stress positions, intonation trends, speech speed changes, emotional colors, etc. Therefore, when generating speech audio, the acoustic module can accurately generate speech audio that fits the emotion, speech speed, intonation, pause rhythm, etc. of the text content based on the semantic and prosodic features carried by the semantic token sequence. Unlike traditional TTS systems that can only generate speech of fixed templates, the speech realism is improved.

[0040] In addition, the implementations of the present specification use a large language model with an autoregressive architecture as the backbone model. Compared with traditional TTS models, there is no need to design front-end modules such as a duration predictor and a prosody predictor, which reduces the number of model components and the workload of hyperparameter tuning, and lowers the complexity of training and deployment. Moreover, by utilizing the strong modeling capability of the large language model, complex nonlinear mapping relationships among phoneme duration and text content, speaker style, and emotional expression can be automatically learned, so that the learning capability is significantly improved compared with the traditional TTS models. Through the end-to-end implicit modeling manner, the stiffness and discontinuity caused by explicit duration prediction are avoided, so that the generated speech is more natural and fluent in terms of pauses, speech speed changes, stress distribution, etc., and the prosodic naturalness and expressiveness of the synthesized speech are significantly improved.

[0041] It should be noted that when generating the semantic token sequence, the semantic module adopts an Autoregressive (AR) generation manner, that is, the model predicts and outputs only one token each time, and the token generation process is strictly serial. For example, in the current generation step, the model predicts and outputs a token t1, then in the next generation step, the token t1 needs to be spliced with the original target text as an input to predict and generate a token t2, then in the next generation step, the tokens {t1, t2} are spliced with the original target text as an input to predict and generate a token t3, and so on, until a complete semantic token sequence is obtained.

[0042] In speech synthesis scenarios, a single speech segment can contain hundreds to thousands of lexical units, resulting in numerous inference steps and high cumulative latency in model output. This latency issue is particularly prominent in real-time speech interaction and streaming speech broadcasting. For example, in online teaching support systems, when a student asks a question and the system converts the response text into speech output, a considerable wait time often occurs, impacting the smoothness of the teaching process and the user experience.

[0043] Non-autoregressive (NAR) generation methods can predict all lexical units at once to accelerate reasoning. However, non-autoregressive methods disrupt the causal dependencies between lexical contexts, leading to a significant decrease in speech synthesis quality and defects such as semantic incoherence and prosodic breaks, making it difficult to meet the speech quality requirements of practical applications.

[0044] Therefore, in some embodiments of this specification, the semantic module uses the lexical fusion vector of multiple tokens as the modeling unit to perform autoregressive inference, and uses the lexical decoding module to decode and obtain multiple token outputs, thereby achieving single-step multi-token output while retaining the autoregressive causal constraints of the large language model, significantly improving the inference efficiency of speech synthesis.

[0045] For example Figure 1 This specification shows an architecture diagram of a speech synthesis model for some embodiments. The following is a summary of the diagrams. Figure 1 Please provide an explanation.

[0046] like Figure 1 As shown, in this example, the speech synthesis model includes a semantic module 10 and an acoustic module 20. The semantic module 10 includes a backbone model and a lexical decoding module. The backbone model can be a large language model based on autoregressive inference. This large language model can be a model obtained by using a general large language model as a base model and then fine-tuning and training the base model. In this example, the backbone model performs autoregressive inference using the lexical fusion vectors of k (k≥2) lexical units as modeling units. Therefore, its output lexical fusion vector represents the fusion representation of k lexical units. The lexical decoding module is connected to the output of the backbone model. The lexical decoding module is used to decode the lexical fusion vector, thereby decoding the fusion representation into k discrete lexical units.

[0047] In some implementations, both the backbone model and the lexical decoding module can adopt a transformer-based model. For example, the backbone model may use a multi-layer transformer model, and the lexical decoding module may use a lightweight transformer branch model. This will be understood by those skilled in the art, and will not be elaborated further in this specification.

[0048] It is understandable that in traditional autoregressive inference, each word generation step (one frame) can only predict one word, requiring n inference steps to generate n words. However, in the implementation of this specification, the backbone model's autoregressive inference outputs a word fusion vector of k words. Therefore, each word generation step can infer k words, and similarly, generating n words requires only n / k inference steps. Furthermore, the generation of the word fusion vector in each generation step is strictly sequential, meaning the current word fusion vector depends on previously generated word fusion vectors. Therefore, the causal constraints of autoregression are still preserved, resulting in better speech synthesis quality.

[0049] It is worth noting that the input of the semantic module 10 may include target text, which refers to the text information that needs to be synthesized into speech. For example, the target text may be text information described in natural language.

[0050] In some implementations, the input to the semantic module 10 may further include a timbre identifier (spk ID) and / or a language identifier (lang). The timbre identifier (spk ID) specifies the timbre of the speaker corresponding to the speech. For example, in one exemplary implementation, multiple speaker timbre embedding vectors can be pre-trained, such as adult male voice, adult female voice, child male voice, and child female voice. These timbre embedding vectors are then stored in the model, with a unique timbre identifier assigned to each vector. When a user wants to specify the timbre for the speech playback, the timbre identifier (spk ID) can be embedded in the input to the speech synthesis model. The corresponding timbre embedding vector is then found based on the timbre identifier (spk ID), and the timbre embedding vector is fused into the audio decoding process to output the target speech played with that timbre.

[0051] The language identifier `lang` is used to specify the language of the target speech. In some implementations, the speech synthesis model can be trained on data from multiple different languages, such as Chinese, English, and French. When a user wants to specify the language of the synthesized speech, they can embed the language identifier `lang` into the input of the speech synthesis model, and the model can then generate the target speech in that language.

[0052] In some embodiments, the input to the semantic module 10 may also include reference audio data. The reference audio data may be an audio sample of a specific speaker, carrying the speaker's timbre characteristics, pronunciation habits, and prosodic style. When a user wants to synthesize speech with a specific speaker's style, a piece of reference audio data can be embedded in the input to the semantic module 10, thus eliminating the need to pre-configure timbre identifiers for each speaker. Those skilled in the art will understand this, and it will not be elaborated further in this specification.

[0053] After receiving the reference audio data, the semantic module 10 can associate the reference audio data with the target text. In one example, the input sequence of the semantic module 10 includes the reference audio data corresponding to the reference audio data, the reference text, the target text, and the language identifier. The reference text refers to the text content corresponding to the reference audio data. For example, if the content of the reference audio data is a speech phrase "The weather is nice today," then the reference text is the text information of "The weather is nice today." By inputting both the reference text and the reference audio data, the semantic module 10 can learn the speaker's prosodic expression habits for different texts, thereby making the target speech generated based on the reference audio data more prosodic in style than that of the speaker.

[0054] When the input to a speech synthesis model includes one or more of the following: timbre identifier (spk ID), language identifier (lang), and reference audio data, the speech synthesis model can extract corresponding features based on each input and fuse them internally. For example, a user can specify the timbre of the target speaker using the timbre identifier (spk ID) and the language identifier (lang) to specify the language of the synthesized speech, enabling the model to generate target speech with the target timbre and conforming to the pronunciation rules of the target language. As another example, a user can also specify the prosodic style of the target speech using reference audio data and the language identifier (lang) to specify the language of the synthesized speech, enabling the model to generate target speech with the prosodic style of the reference speaker and conforming to the pronunciation rules of the target language.

[0055] In the embodiments described in this specification, the speech synthesis model supports combinations of various control conditions such as timbre identifier spk ID, language identifier lang, and reference audio data. Users can select one or more control conditions to input into the speech synthesis model according to their actual needs, thereby achieving diverse speech synthesis effects.

[0056] In the embodiments described in this specification, the backbone model can be a large language model based on autoregressive inference. The backbone model outputs a lexical fusion vector C at each generation step through autoregressive inference. For example, in the current generation step, the lexical fusion vector C1 is predicted and output. Then, in the next generation step, the lexical fusion vector C1 is combined with the original target text as input to predict and output the lexical fusion vector C2. Subsequently, in the next generation step, the lexical fusion vectors {C1, C2} are combined with the original target text as input to predict and output the lexical fusion vector C3, and so on. These lexical fusion vectors are concatenated sequentially to obtain the fusion vector sequence {C1, C2, ..., C...}. n}

[0057] It can be understood that the word fusion vector C refers to the fused representation of k words; that is, the word fusion vector C is the feature vector obtained by fusing and encoding k words. In the implementation method of this specification, for each word fusion vector C... i This can be input into the word decoding module, which then processes the word fusion vector C. i Decoding is performed to obtain the word subsequence it includes, which contains k words.

[0058] Taking a certain generation step as an example, the backbone module fuses the previously generated word vectors {C1,C2,…,C}. i-1 The algorithm takes the original target text as input and predicts the word fusion vector C for this generation step. i Then, the word fusion vector C is... i Inputting the token decoding module, the corresponding token subsequence can be obtained by decoding it, denoted as {C}. i_1 C i_2 ,…,C i_k}

[0059] For each generation step, after obtaining the corresponding subsequence of words through the backbone model and the word decoding module, the subsequence of words is concatenated according to the generation time order to obtain the complete semantic word sequence.

[0060] As can be seen, in the scheme described in this specification, k words can be generated in one reasoning step. When generating the same number of words, the efficiency is improved by k times compared with the traditional autoregressive reasoning theory. Moreover, the word fusion vector between each generation step is strictly sequential, that is, the later word fusion vector depends on the previously generated word fusion vector, thus still retaining the autoregressive causal constraints, which can better integrate contextual dependencies and ensure the quality of speech synthesis.

[0061] In some implementations, the input to the acoustic module 20 includes at least a semantic word sequence and a phoneme sequence corresponding to the target text. The semantic word sequence is generated by the aforementioned semantic module 10 and carries deep semantic information and prosodic features of the target text. The phoneme sequence refers to the sequence obtained by converting each character / word in the target text into its corresponding phoneme and then concatenating them in the text order; it carries precise pronunciation information of the target text. The acoustic module 20 fuses the semantic word sequence and the phoneme sequence to generate the target speech corresponding to the target text.

[0062] In some implementations, the phoneme sequence can be obtained by conversion using a G2P (Grapheme-to-Phoneme) module. For example, the front end of the acoustic module 20 includes a G2P module, and inputting the target text into the G2P module will yield the phoneme sequence corresponding to the target text.

[0063] Combination Figure 1 As shown, the acoustic module 20 includes a text encoder, a stream model, and an audio decoder. The text encoder is used to extract features from semantic word sequences and phoneme sequences separately, and then fuses the two types of features to output multimodal fusion features. For example... Figure 2 A schematic diagram of the text encoder is shown below. Figure 2 Please provide an explanation.

[0064] exist Figure 2 In the example, the text encoder includes a first convolutional layer (Conv 1), a second convolutional layer (Conv 2), and a cross-attention fusion layer. The input to the first convolutional layer is the phoneme sequence of the target text. The first convolutional layer extracts features from the phoneme sequence to obtain a phoneme embedding vector T_Emb. The input to the second convolutional layer is a sequence of semantic words. The second convolutional layer extracts features from the semantic word sequence to obtain a semantic embedding vector C_Emb. The cross-attention fusion layer is a feature fusion layer utilizing a cross-attention mechanism. Its inputs are the phoneme embedding vector T_Emb and the semantic embedding vector C_Emb. The cross-attention fusion layer performs cross-attention fusion on the phoneme embedding vector T_Emb and the semantic embedding vector C_Emb, aligning phoneme information (pronunciation content) and semantic information (prosodic style) in the feature space. The output is a fused multimodal feature, which represents the fusion feature expression of pronunciation content and prosodic style. In the example provided in this manual, by introducing a cross-attention mechanism, the phoneme embedding vector and the semantic embedding vector can be fused, so that the acoustic module can take into account both pronunciation accuracy and prosodic naturalness.

[0065] The streaming model connects to the output of the text encoder and is used to generate target speech distribution features based on multimodal fusion features. In some implementations, the streaming model can be implemented using structures such as WAVENET networks, Transformer-based networks, or Normalizing Flow. The input of the streaming model includes at least the multimodal fusion features output by the text encoder. The streaming model is used to transform the multimodal fusion features output by the text encoder into a more complex speech feature distribution that better reflects the distribution of real speech data. It can be understood that the speech distribution corresponding to the multimodal fusion features output by the text encoder is usually relatively simple (such as a Gaussian distribution), while the distribution of real speech data often has higher complexity and nonlinearity. The streaming model maps the simple distribution to a complex distribution through a series of reversible transformations, enabling the subsequent audio decoder to generate more realistic and natural speech waveforms based on this distribution.

[0066] In some implementations, a projection layer can be placed between the streaming model and the cross-attention fusion layer. The projection layer is used to extract the target speech distribution from the multimodal fusion features. For example, the projection layer maps the multimodal fusion features to low-dimensional speech distribution parameters through one or more linear or convolutional transformations. These speech distribution parameters can include the mean and variance, representing the probability distribution of the target speech in the feature space, such as a Gaussian distribution. It can be understood that by compressing the high-dimensional multimodal fusion features output by the text encoder into a compact speech distribution representation through the projection layer, the computational complexity of the subsequent streaming model is reduced, improving the model's processing efficiency. In some implementations, the projection layer can be implemented using an S-layer Transformer encoder or a convolutional layer.

[0067] An audio decoder is connected to the output of the streaming model and is used to generate target speech based on the target speech distribution features. In some implementations, the audio decoder may include M transposed convolutional blocks for upsampling the target speech distribution features, restoring the frame-level speech features to a sample-point-level audio waveform, thus obtaining playable target speech. For example, if the speech distribution features output by the streaming model are frame-level features (e.g., each frame corresponds to 10ms of audio), the audio decoder progressively upsamples through multiple transposed convolutional blocks, restoring the frame-level features to temporal sampling points (e.g., a 16kHz sampling rate), thereby obtaining a continuous audio waveform audible to the human ear.

[0068] In some implementations, the inputs to the audio decoder and the streaming model also include a timbre embedding vector spk_Emb. For example, the acoustic module 20 may also include a voice encoder, which generates the timbre embedding vector spk_Emb and inputs it into the audio decoder and the streaming model. For example, in one instance where the input to the speech synthesis model includes a timbre identifier spk ID, the voice encoder can look up the corresponding timbre embedding vector spk_Emb from a pre-trained timbre embedding vector library based on the timbre identifier spk ID. This library pre-stores multiple timbre embedding vectors corresponding to different timbre identifiers. In another instance, where the input to the speech synthesis model includes reference audio data, the voice encoder performs acoustic feature extraction on the reference audio data and generates the timbre embedding vector spk_Emb based on the extracted acoustic features.

[0069] In the embodiments described in this specification, by injecting timbre embedding vectors into the stream model and / or audio decoder, the decoupling control of prosody, semantic content and timbre in the speech synthesis process is realized, so that the same text can be broadcast with any target timbre, and the timbre switching does not affect the pronunciation accuracy and prosodic naturalness.

[0070] Based on the above speech synthesis model, the training process of the speech synthesis model will be explained.

[0071] In the embodiments described in this specification, the first step is to prepare sample data for training the speech synthesis model. For example, corpus data of multiple speakers can be obtained, including audio and the corresponding text. Here, audio refers to the audio data generated by the speaker reading natural language text, and the corresponding text refers to the text content read aloud in the audio. It can be understood that the more speakers included in the sample data and the longer the total audio duration of each speaker, the richer the timbre diversity and pronunciation styles the model can learn.

[0072] The corpus of sample data can come from open-source speech datasets, such as publicly available internet datasets. It can also be recorded data, such as audio recordings of a specific speaker taken in a recording studio or quiet environment. Furthermore, it can be audio resources obtained from the internet, such as public speeches and audiobooks. Understandably, this specification does not limit the source of the sample data.

[0073] In some implementations, the obtained sample data may have problems such as inconsistent audio quality, mismatch between text and audio, and missing text corresponding to audio. Therefore, the sample data can be further preprocessed.

[0074] For example, in one possible implementation, each audio segment can be divided into short audio clips of S seconds (the value of S can be set). Then, a speech recognition tool is used to recognize each short audio clip, obtaining the corresponding recognized text for each short audio clip. The speech recognition tool can use an open-source speech recognition model, such as FunASR or SenseVoice. Afterward, each short audio clip is associated with the recognized text to obtain an audio-text data pair.

[0075] For example, in another possible implementation, the audio-text data pairs can be further filtered. For instance, a quality score can be calculated for each audio segment, filtering out segments with poor sound quality. Alternatively, the similarity between the identified text and the original text corresponding to the audio can be calculated, filtering out audio segments with significant differences, thereby ensuring the matching of text and audio content.

[0076] In another possible implementation, the audio-text data pairs can be further normalized. For example, unconventional characters in the text can be converted to standard pronunciations, such as converting URLs to full pinyin or symbols to phonetic representations. This text normalization allows the model to learn a unified text representation, improving its ability to recognize unconventional characters.

[0077] In some embodiments of this specification, the training process of the speech synthesis model can adopt a two-stage training strategy, that is, training the semantic module and the acoustic module separately. The training process of the semantic module is described below.

[0078] like Figure 3 As shown, in some embodiments, the speech synthesis method exemplified in this specification includes the following process for training a semantic model: S310. Obtain sample data.

[0079] In the embodiments described in this specification, the sample data refers to the sample dataset obtained in the aforementioned manner. This sample data includes a large number of audio-text data pairs, and the text in these data pairs is the text sample described in this specification.

[0080] It is worth noting that the backbone model included in semantic module 10 uses word fusion vectors as output for modeling. Therefore, when training the backbone model, the word fusion vectors corresponding to the text samples need to be used as supervision labels.

[0081] For example, in some implementations, a pre-trained neural text encoder can be used to encode the text sample to obtain its corresponding discretized representation of the token sequence. This token sequence is the sample word sequence described in this specification, represented as {t1,t2,t3,…tn}. The sample word sequence includes multiple words arranged in temporal order, and its sequence length depends on the length of the text sample.

[0082] S320. The sample word sequence is grouped according to word time order to obtain multiple word subsequences, each word subsequence including k words.

[0083] In the embodiments described in this specification, the sample word sequence can be grouped according to word sequence, dividing every k words into a word subsequence, where k ≥ 2 and is an integer. For example, see [example provided]. Figure 4 As shown, the sample word sequence is {t1,t2,t3,…t12}. With k=4, the sample word sequence is grouped into 3 groups of word subsequences, namely {t1,t2,t3,t4}, {t5,t6,t7,t8}, and {t9,t10,t11,t12}.

[0084] Of course, those skilled in the art will understand that the value of k is not limited to the above example, and can take any other value suitable for implementation, which is not limited in this specification.

[0085] S330. Input the k words included in each word subsequence into the word encoding module, and fuse them through the word encoding module to obtain the word fusion vector.

[0086] Combination Figure 4As shown, during the training phase, a word encoding module can be deployed. The input of the word encoding module is the k words included in the word subsequence, and its output is the word fusion vector after fusing the k words.

[0087] In one example, the lexical encoding module can employ a lightweight Transformer network or an attention-based encoder network, with input dimensions of k encoded lexical units and output dimensions of a single lexical fusion vector. For instance, for a lexical subsequence {t1,t2,t3,t4}, the lexical encoding module encodes and fuses its four constituent lexical units to obtain the lexical fusion vector C1.

[0088] The lexical encoding module can use a self-attention mechanism to model the dependencies between k lexical units, ensuring that the generated lexical fusion vector fully represents the overall information of the k lexical units in the subsequence. The above fusion operation is performed on each lexical subsequence, resulting in a lexical fusion vector sequence {C1, C2, ..., Cn}, where n is the number of lexical subsequences. It can be understood that the lexical fusion vector is a compressed representation of the k lexical units after joint encoding, rather than a simple concatenation or averaging; it preserves the semantic and temporal dependencies between the k lexical units.

[0089] S340. Using text samples and historical word fusion vectors preceding the current generation step as input, train the backbone model to predict and output the word fusion vector of the current generation step. Also, using the word fusion vector as input, train the word decoding module to predict and output the word subsequences corresponding to each word fusion vector.

[0090] Combination Figure 4 As shown in the embodiments of this specification, the backbone model is based on an autoregressive inference architecture, and its input includes two parts: one is the original text sample, which serves as a global semantic condition throughout each generation step; the other is the previously generated word fusion vector, which serves as the contextual condition for autoregression.

[0091] During training, taking the current generation step i as an example, the backbone model's input includes text samples and the word fusion vectors {C1, C2, ..., Ci-1} already generated in previous generation steps. Based on the input, the backbone model predicts and generates the word fusion vector Ci' for the current generation step i. It can be understood that Ci' is the word fusion vector predicted by the backbone model for the current generation step, while Ci in the word fusion vector sequence is the actual word fusion vector corresponding to the current generation step i. Therefore, Ci in the word fusion vector sequence can be used as a supervision label. The loss between the word fusion vector Ci' and the supervision label Ci can be calculated, such as cross-entropy loss or mean squared error loss, and the backbone model's parameters can be updated through backpropagation.

[0092] The above example illustrates the supervised training process of the backbone model using a generation step i as an example. For each generation step, the above training process can be repeated, and multiple sample data can be used for iterative training until the backbone model meets the convergence condition, thus completing the iterative training process of the backbone model.

[0093] As shown above, this training method enables the backbone model to perform autoregressive inference using word fusion vectors as modeling units, maintaining strict causal dependencies between adjacent generation steps, thus preserving the autoregressive model's ability to model sequence dependencies. Furthermore, since each word fusion vector corresponds to k words, each step of the backbone model's inference can cover the information of k words. Compared to traditional word-by-word autoregression, the number of inference steps required by the backbone model is significantly reduced, improving the model's inference efficiency.

[0094] Continue to refer to Figure 4 As shown, for the token encoding module, its input can be the token fusion vectors included in the token fusion vector sequence {C1, C2, ..., Cn}. For example, taking the token fusion vector Ci as an example, Ci is input to the token decoding module, which decodes Ci and predicts the k tokens corresponding to the token fusion vector Ci. For example... Figure 4 In the example, taking the word fusion vector C1 as an example, after inputting it into the word decoding module, the word decoding module predicts and outputs the word subsequence {t1',t2',t3',t4'}. It can be understood that {t1',t2',t3',t4'} is the word subsequence of the word fusion vector C1 predicted by the word decoding module, while {t1,t2,t3,t4} is the actual word subsequence of the word fusion vector C1. Therefore, {t1,t2,t3,t4} can be used as a supervision label to calculate the loss between the word subsequence {t1',t2',t3',t4'} and the supervision label {t1,t2,t3,t4}, such as cross-entropy loss or mean squared error loss, and update the model parameters of the word decoding module through backpropagation.

[0095] The above example illustrates the supervised training process of the word decoding module using a word fusion vector. For each word fusion vector, the above training process can be repeated, and multiple sample data can be used for iterative training until the word decoding module meets the convergence condition, thus completing the iterative training process of the word decoding module.

[0096] Through the above training process, the backbone model acquires autoregressive generation capabilities using word fusion vectors as modeling units, while the word decoding module acquires the decoding capability to reconstruct word fusion vectors into multiple words. During the inference phase, the backbone model can directly predict and output word fusion vector sequences from the target text, without needing the word encoding module. The word decoding module can decode and expand each word fusion vector into k words, ultimately obtaining a complete semantic word sequence. After training, the semantic module significantly reduces the number of inference steps for word sequences while preserving autoregressive causal constraints, thus improving inference efficiency.

[0097] like Figure 5 As shown, in some embodiments, the speech synthesis method exemplified in this specification includes the following process for training a model of the acoustic module: S510. Input the text sample into the pre-trained semantic module to obtain the semantic word sequence output by the semantic module.

[0098] S520. Input the semantic word sequence and the phoneme sequence corresponding to the text sample into the acoustic module to be trained, and use the audio corresponding to the text sample as the supervision label to train the acoustic module to predict and output the target speech based on the semantic word sequence and the phoneme sequence.

[0099] In this embodiment of the specification, after training the semantic module 10 through the aforementioned implementation process, the model parameters of the semantic module 10 can be fixed, and the acoustic module 20 can be trained.

[0100] First, the text sample is input into the trained semantic module to obtain the semantic word sequence {t1,t2,t3,…tn} corresponding to the text sample predicted by the semantic module. This semantic word sequence carries the deep semantic information and prosodic features of the text sample.

[0101] Next, the semantic word sequence and the phoneme sequence corresponding to the text sample are input into the acoustic module 20. As mentioned earlier, the phoneme sequence is obtained by phoneme conversion of the text sample. The acoustic module 20 predicts and outputs the target speech based on the semantic word sequence and the phoneme sequence.

[0102] It can be understood that the target speech is the audio data predicted by acoustic module 20. In the sample data, the text samples correspond to actual audio. Therefore, the audio in the audio-text data pair can be used as the supervision label. The loss between the target speech and the supervision label, such as L1 loss or L2 loss, is calculated, and the model parameters of acoustic module 20 are updated through backpropagation. The above process is repeated to iteratively train the acoustic module using the sample data until the acoustic module meets the convergence condition, thus completing the iterative training process of acoustic module 20.

[0103] In some implementations, during the training process of the speech synthesis model described above, the input of the speech synthesis model may also include one or more of the following: timbre identifier spk ID, language identifier lang, reference audio data, and text data corresponding to the reference audio data. This allows the model to learn speech synthesis capabilities for different timbres and languages, thereby improving the model's generalization ability.

[0104] For example, in one scenario, the input to the speech synthesis model can include a text sample, a timbre identifier (spk ID), and a language identifier (lang). The spk ID represents the timbre characteristics of a specified speaker, and the lang indicates the language corresponding to the text sample. Through this training process, the speech synthesis model can learn the timbre and prosody of multiple speakers. During the inference phase, when a user wants to specify a particular speaker's timbre for speech synthesis, the corresponding spk ID can be included in the input sequence of the speech synthesis model. The acoustic module, during audio decoding, can then embed the corresponding timbre embedding vector based on the spk ID to generate target speech that matches the specified speaker's timbre.

[0105] The above describes the construction and training process of the speech synthesis model in some embodiments of this specification. The following describes the speech synthesis method based on the speech synthesis model.

[0106] like Figure 6 As shown, in some embodiments, the speech synthesis method exemplified in this specification includes: S610. Input the target text into the semantic module of the speech synthesis model to obtain the semantic word sequence output by the semantic module.

[0107] S620. Input the semantic word sequence and the phoneme sequence corresponding to the target text into the acoustic module of the speech synthesis model to obtain the target speech output by the acoustic module.

[0108] During the model inference phase, the input to the speech synthesis model can include target text, which is the text content of the speech to be synthesized.

[0109] As can be seen from the foregoing, in some implementations, users can specify the timbre and language of the generated target speech. Therefore, in addition to the target text, the input sequence of the speech synthesis model may also include one or more of the following: timbre identifier spk ID, language identifier lang, reference audio data, and text data corresponding to the reference audio data.

[0110] For example, in one instance, the input sequence for the speech synthesis model includes the target text, a user-specified timbre identifier (spk ID), and a language identifier (lang). Combined with... Figure 1As shown, the semantic module 10 generates a corresponding semantic word sequence based on the target text. Then, the acoustic module 20 takes the semantic word sequence and the phoneme sequence of the target text as input, and embeds the corresponding timbre embedding vector based on the timbre identifier spk ID in the stream model and / or audio decoder stage, thereby predicting and outputting the target speech with the user-specified timbre.

[0111] For example, in another example, the input sequence of the speech synthesis model includes target text, reference audio data, text data corresponding to the reference audio data, and language identifier (lang). In this example, if a user wants to imitate the timbre or rhythm of a speaker, they can input an audio clip of that speaker and its corresponding text as reference data into the model. Then, during the audio decoding stage in the acoustic module 20, the corresponding timbre embedding vector can be extracted based on the reference audio data and text data. Combined with the phoneme sequence and semantic word sequence of the target text, the corresponding target speech is generated. This target speech is the audio that imitates the timbre and rhythm of the speaker in the reference audio data.

[0112] As can be seen from the above, in the embodiments of this specification, the speech synthesis model provides input of various control conditions. Users can specify a certain speaker timbre, or imitate the desired speaker timbre by referring to audio, so as to achieve diverse speech synthesis effects.

[0113] like Figure 7 As shown, in some embodiments, the speech synthesis method exemplified in this specification includes the following reasoning process of the semantic module: S710. In each lexical generation step, the backbone model predicts and outputs the lexical fusion vector of the current generation step based on the target text and the historically generated lexical fusion vectors.

[0114] As mentioned above, the backbone model is based on an autoregressive inference architecture, which generates word fusion vectors representing the fusion features of k word terms step by step, and the word fusion vector generated in each step depends on all the word fusion vectors that have been generated before.

[0115] In the embodiments described in this specification, the generation process of each word fusion vector is referred to as a word generation step. Taking the current generation step i as an example, the input of the backbone model includes the target text and the word fusion vectors {C1, C2, ..., Ci-1} already generated in previous generation steps. The backbone model predicts and generates the word fusion vector Ci for the current generation step i based on the input. Similarly, for the next generation step i+1, the word fusion vector Ci needs to be concatenated with the historical word fusion vectors, with the target text and the word fusion vectors {C1, C2, ..., Ci-1, Ci} as input, and the predicted output word fusion vector Ci+1 corresponding to generation step i+1 is generated. In summary, the reasoning process of each generation step relies on the previously generated word fusion vectors to achieve an autoregressive reasoning process.

[0116] S720, the word decoding module decodes the word fusion vector to obtain the word subsequence corresponding to the word fusion vector.

[0117] It can be understood that the word fusion vector predicted by the backbone model represents the fusion features of k words, and the role of the word decoding module is to decode the word fusion vector into k discrete words.

[0118] For example, taking the current generation step i as an example, after obtaining the word fusion vector Ci, the word fusion vector Ci is input into the word decoding module. The word decoding module expands the word fusion vector Ci in reverse into k words through decoding. The sequence represented by these k words is the word subsequence. For example, in one example, the word subsequence can be represented as {t1,t2,…,tk}.

[0119] S730. Concatenate the subsequences of each lexical generation step to obtain the semantic lexical sequence.

[0120] The above explanation uses a single generation step i as an example. By repeating the above process for each generation step, the word subsequence corresponding to each generation step can be obtained. Then, these word subsequences are concatenated according to the generation order to obtain the complete semantic word sequence corresponding to the target text. The semantic word sequence represents a feature sequence carrying the deep semantic information and prosodic features of the target text.

[0121] It is understood that in the scheme of this specification, each generation step can generate k words simultaneously. When generating the same number of words, the efficiency is improved by k times compared with the traditional autoregressive inference theory. Moreover, the word fusion vector between each generation step is strictly serial, that is, the later word fusion vector depends on the previously generated word fusion vector, thus still retaining the autoregressive causal constraints, which can better integrate contextual dependencies.

[0122] In some implementations, the value of k can be selected according to the specific needs of the scenario. It can be understood that a larger value of k results in higher model inference efficiency but lower model accuracy, and vice versa. Therefore, the value of k can be selected by balancing model efficiency and accuracy. For example, in an exemplary implementation, k=3 or k=4 can effectively balance model inference efficiency and accuracy.

[0123] As described above, in this embodiment, by using the fusion vector of k word units as the autoregressive modeling unit of the semantic module, each inference step of the semantic module can cover the information of k word units. The number of inference steps is reduced from the number of word units n to n / k, theoretically improving efficiency by a factor of k compared to traditional word-by-word autoregressive inference. Simultaneously, causal constraints are maintained between each generation step at the word unit fusion vector level; the prediction of the subsequent word unit fusion vector depends on the previously generated word unit fusion vector, fully preserving the causal constraints and contextual dependencies of autoregression. Therefore, while ensuring the semantic coherence and prosodic naturalness of the synthesized speech, inference latency is effectively reduced, balancing speech synthesis quality and inference efficiency.

[0124] In some implementations, the speech synthesis module can employ a streaming inference mechanism, meaning that the acoustic module 20 does not need to wait for the semantic module 10 to generate a complete semantic lexical sequence before starting to synthesize audio. Instead, the audio segment is synthesized and output synchronously during the process of the semantic module 10 generating the semantic lexical sequence.

[0125] like Figure 8 As shown, in some embodiments, the speech synthesis method exemplified in this specification includes: S810. Whenever the number of words decoded by the word decoding module reaches a first number, the first number of words are input into the acoustic module as a word stream block.

[0126] S820: The acoustic module predicts and outputs the audio segment corresponding to the current word stream block based on the current word stream block and the word stream blocks above and / or below the current word stream block, and splices the audio segments in the generation order to obtain the target speech.

[0127] In some implementations, the lexical decoding module decodes each lexical fusion vector to obtain a corresponding lexical subsequence, each lexical subsequence comprising k lexical units. The reasoning process of the semantic module and the audio synthesis process of the acoustic module are executed in parallel streaming. For example, whenever the cumulative number of lexical units decoded by the lexical decoding module reaches a first number, the first number of lexical units are input into the acoustic module as a lexical streaming block.

[0128] For example, in one instance, assuming the first number is 25, after the word decoding module decodes the first to the 25th words, these 25 words are sent as the current word streaming block to the acoustic module for decoding, without waiting for the word decoding module to continue decoding subsequent words.

[0129] For each word stream block, the acoustic module can further combine the word stream blocks preceding and / or following the current word stream block to jointly predict and output the audio segment corresponding to the current word stream block.

[0130] For example, in some implementations, the acoustic module acquires a second number of historical lexical units preceding the current lexical unit streaming block and / or a third number of future lexical units following the current lexical unit streaming block. Historical lexical units refer to lexical units that have been decoded before the current lexical unit streaming block and are used to provide contextual information for the synthesis of the current audio segment. Future lexical units refer to subsequent lexical units that have been decoded by the semantic module after the current lexical unit streaming block and are used to provide contextual information for the synthesis of the current audio segment.

[0131] For example, assuming the first, second, and third quantities are all 25, when the acoustic module decodes the current word stream block, its input includes the current 25 words, the 25 historical words preceding the current 25 words, and the 25 future words following the current 25 words, totaling 75 words. The acoustic module performs decoding processing based on the above 75 word inputs, but only uses the audio segment corresponding to the current 25 words as the valid output.

[0132] It is understandable that introducing historical lexical units can ensure that the prosodic style and speech features of the current audio segment are consistent with those of the previously synthesized segments, avoiding abrupt changes in intonation between segments. At the same time, introducing future lexical units can ensure that the prosodic direction of the current audio segment (such as sentence ending tone, pause position, etc.) matches the subsequent content, avoiding prosodic deviations caused by a lack of information about the following text.

[0133] In the above manner, the acoustic module synthesizes audio segments one by one according to the word generation progress of the semantic module, and obtains the audio segments corresponding to each word streaming block. The audio segments are then spliced ​​together in the generation order to obtain the complete target speech corresponding to the target text.

[0134] As can be seen from the above, in the embodiments of this specification, the reasoning process of the semantic module and the audio synthesis process of the acoustic module are executed in streaming parallel, so that the speech synthesis system can start outputting audio without waiting for all tokens to be generated, which effectively reduces the speech synthesis latency. At the same time, the prediction of audio segments combines the context information of the current streaming block to ensure the consistency of prosody style and speech features. Moreover, each token streaming block is obtained based on the reasoning of the complete target text, thus the prosody coherence of the complete text is preserved. Compared with the streaming method of text segmentation, the problem of prosody discontinuity caused by text segmentation can be effectively avoided.

[0135] In some embodiments, the acoustic module may further perform window function weighted superposition processing on the transition region between two adjacent audio segments to obtain the target speech. For example, the acoustic module may apply a window function (such as a Hamming window or a Hanning window) to the transition region between two adjacent audio segments for weighted superposition processing, so that the tail of the previous audio segment and the head of the subsequent audio segment are smoothly fused within the transition region. In one example, taking the adjacent first audio segment and second audio segment as an example, an overlapping region with a preset length may be provided. Within the overlapping region, the sampling points of the first audio segment and the second audio segment are respectively multiplied by corresponding weight coefficients and then added, so as to obtain an audio signal with smooth transition. In the embodiments of this specification, the transition region is subjected to weighted superposition processing through the window function, which eliminates the splicing traces caused by independent decoding between token streaming blocks, and makes the finally output target speech auditorily continuous and natural.

[0136] In some embodiments, the target text may further include pronunciation annotations for one or more words, and the phonemes of polyphonic characters, interchangeable characters and special characters can be corrected through the pronunciation annotations.

[0137] For example, the pronunciation annotation may be embedded in the target text in a preset format. In one example, the target text is "学而时习之,不亦说乎?", wherein the character "说" is an interchangeable character, which is interchangeable with "悦" and should be pronounced as "yue4" instead of "shuo1". In the embodiments of this specification, the target text may be modified to "学而时习之,不亦" by means of pronunciation annotation <yue4>"hu", in which, <yue4>is a pronunciation annotation for a specific character, indicating that in this context, "说" should be pronounced as "yue4".

[0138] After receiving the target text, the speech synthesis model parses the target text, identifies and extracts the pronunciation annotations and their corresponding pronunciation information. Taking "学而时习之,不亦 <yue4>Take "hu?" as an example, the speech synthesis model parses and recognizes the pronunciation annotation "yue4", and extracts the pronunciation information "yue4" corresponding to the pronunciation annotation. According to the pronunciation information, the target phoneme of the corresponding character in the target text is determined, that is, the target phoneme of the character "yue" is "yue4". For characters in the target text that do not contain pronunciation annotations, processing can be performed according to the default phoneme conversion method, so as to obtain the phoneme sequence corresponding to the target text.

[0139] It can be seen from the above that in the embodiments of the present specification, characters with specific pronunciations can be corrected in pronunciation through annotation according to user requirements, which is especially suitable for scenarios where automatic conversion may make mistakes, such as rare characters, interchangeable characters, polyphonic characters, proper nouns, etc., and improves the effect of speech synthesis.

[0140] In some embodiments, the training process of the semantic module may include general base model training and a fine-tuning process of the base model. For example, when training the semantic module 10 through the aforementioned training process, sample data of multiple speakers can be used for training first to obtain a base model, which represents the general mapping rules for speakers with various timbres and prosodies. Then, in the fine-tuning stage, a specific target speaker can be selected from the multiple speakers, and the sample data of the target speaker is used to perform fine-tuning training on the semantic model. The fine-tuned model has a high degree of restoration of the timbre and pronunciation habits of the target speaker.

[0141] In addition, during the training process of the speech synthesis model, a dynamic batch size strategy can be adopted for training. The batch size refers to the number of samples used in a single iteration of model training. The larger the batch size, the higher the parallel computing efficiency of hardware resources (such as GPU) and the faster the training speed, but the higher the requirements for hardware capabilities, and vice versa. Since the text lengths of different training samples vary greatly, if a fixed batch size is adopted, short text samples cannot fully utilize the hardware computing power, while long text samples may exceed the upper limit of hardware capabilities.

[0142] Therefore, in the embodiments of the present specification, when loading each batch of sample data, the size of the batch size can be dynamically adjusted according to the text length of the samples in the current batch. For example, a larger batch size is used for short text samples, and the batch size is automatically reduced for long text samples, so that each batch of data occupies hardware resources as much as possible, thereby maximizing resource utilization and improving the model training speed.

[0143] In some embodiments, the present specification provides a speech synthesis apparatus, for example, refer to the aforementioned Figure 1 and Figure 2 As shown, the device includes a speech synthesis model, which is configured to generate and output corresponding target speech based on the target text. The speech synthesis model includes a semantic module and an acoustic module. The semantic module is used to convert the target text into a semantic word sequence containing semantic information, and the acoustic module generates the target speech based on the semantic word sequence. The semantic module includes a backbone model based on autoregressive inference and a lexical decoding module connected to the backbone model. The semantic module is configured as follows: In each lexical generation step, the backbone model predicts and outputs the lexical fusion vector for the current generation step based on the target text and the previously generated lexical fusion vectors. The word decoding module decodes the word fusion vector to obtain the word subsequence corresponding to the word fusion vector. The word subsequence includes k words, where k ≥ 2. By concatenating the subsequences of each lexical generation step, the semantic lexical sequence is obtained.

[0144] In some embodiments, this specification provides an electronic device comprising: processor; The memory stores computer instructions that cause the processor to perform the method described in any of the above embodiments.

[0145] In some embodiments, this specification provides a storage medium storing computer instructions for implementing the methods described in any of the above embodiments.

[0146] In some embodiments, this specification provides a computer program product for implementing the methods described in any of the above embodiments.

[0147] Figure 9 A schematic diagram of the structure of an electronic device 600 suitable for implementing the methods of this specification is shown. Figure 9 The architecture shown can realize the corresponding functions of the above-mentioned server and storage media.

[0148] like Figure 9 As shown, the electronic device 600 includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a memory 602 or a program loaded into the memory 602 from a storage section 608. The memory 602 also stores various programs and data required for the operation of the electronic device 600. The processor 601 and the memory 602 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0149] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0150] In particular, according to embodiments of this disclosure, the above-described method process can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the above-described methods. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611.

[0151] Obviously, the above embodiments are merely examples for clear illustration and are not intended to limit the embodiments. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all embodiments here. However, obvious variations or modifications derived therefrom remain within the scope of protection created by this specification.

Claims

1. A speech synthesis method characterized by, include: A semantic word sequence is predicted based on the target text using autoregressive inference. The semantic word sequence implicitly encodes at least one of the prosodic information, sentiment information, and speech style information corresponding to the target text. In each non-first step of word generation in the autoregressive inference, the word fusion vector of the current generation step is predicted and output based on the target text and the previously generated word fusion vectors. The word fusion vector is then decoded to obtain the word subsequence corresponding to the word fusion vector. The word subsequence includes k words, k≥2. The word subsequences obtained from each word generation step are concatenated to obtain the semantic word sequence. Based on the semantic word sequence, the phoneme sequence of the target text, and the timbre embedding vector, the target speech corresponding to the target text is predicted and generated. The timbre embedding vector is used to characterize the timbre information of the target speech. The timbre embedding vector is determined from a pre-trained preset timbre library based on the input timbre identifier, or the timbre embedding vector is determined based on the input reference audio data.

2. The method of claim 1, wherein, The method of obtaining a semantic word sequence based on target text prediction using autoregressive inference includes: inputting the target text into the semantic module of the speech synthesis model, performing autoregressive inference on the target text through the backbone model included in the semantic module to obtain the word fusion vector for each word generation step, and decoding the word fusion vector through the word decoding module included in the semantic module to obtain the word subsequence corresponding to the word fusion vector. The step of predicting and generating target speech corresponding to the target text based on the semantic word sequence, the phoneme sequence of the target text, and the timbre embedding vector includes: the acoustic module of the speech synthesis model predicts and generates the target speech based on the semantic word sequence, the phoneme sequence of the target text, and the timbre embedding vector.

3. The method of claim 2, wherein, Also includes: Whenever the number of words decoded by the word decoding module reaches a first number, the first number of words are input into the acoustic module as a word stream block; The acoustic module predicts and outputs an audio segment corresponding to the current lexical streaming block based on the current lexical streaming block and the lexical streaming blocks above and / or below the current lexical streaming block. It then performs window function weighted superposition processing on the transition region of two adjacent audio segments and splices the audio segments in the generation order to obtain the target speech.

4. The method of claim 2, wherein, The acoustic module includes a text encoder, a stream model, and an audio decoder; The text encoder is used to extract and fuse features from the semantic word sequence and the phoneme sequence of the target text to obtain multimodal fusion features; The streaming model generates target speech distribution features based on the multimodal fusion features and timbre embedding vectors; The audio decoder generates the target speech based on the target speech distribution features and the timbre embedding vector.

5. The method of claim 4, wherein, The text encoder includes a first convolutional layer, a second convolutional layer, and a cross-attention fusion layer; The first convolutional layer is used to extract phoneme sequence features from the target text to obtain a phoneme embedding vector; The second convolutional layer is used to extract semantic embedding vectors from the semantic word sequence features; The cross-attention fusion layer is used to perform cross-attention fusion on the phoneme embedding vector and the semantic embedding vector to obtain the multimodal fusion feature.

6. The method according to any one of claims 2 to 5, characterized in that, The training process of the semantic module includes: Obtain sample data, which includes text samples and their corresponding sample word sequences, wherein the sample word sequences include multiple words arranged in chronological order; The sample word sequence is grouped according to word sequence to obtain multiple word subsequences, each word subsequence including k words; Each subsequence of words contains k words, which are then input into the word encoding module. The word encoding module is used to fuse the words to obtain a word fusion vector. Using the text sample and the historical word fusion vectors preceding the current generation step as input, the backbone model is trained to predict and output the word fusion vector of the current generation step. Furthermore, using the word fusion vectors as input, the word decoding module is trained to predict and output the word subsequences corresponding to each word fusion vector.

7. A speech synthesis device, characterized in that, This includes a speech synthesis model, which comprises a semantic module and an acoustic module; The semantic module is used to predict a semantic word sequence based on the target text using autoregressive inference. The semantic word sequence implicitly encodes at least one of the prosodic information, sentiment information, and speaking style information corresponding to the target text. In each non-first step of word generation in the autoregressive inference, based on the target text and the previously generated word fusion vectors, the word fusion vector of the current generation step is predicted and output. The word fusion vector is then decoded to obtain the word subsequence corresponding to the word fusion vector. The word subsequence includes k words, k≥2. The word subsequences obtained from each word generation step are concatenated to obtain the semantic word sequence. The acoustic module is used to predict and generate target speech corresponding to the target text based on the semantic word sequence, the phoneme sequence of the target text, and the timbre embedding vector. The timbre embedding vector is used to characterize the timbre information of the target speech. The timbre embedding vector is determined from a pre-trained preset timbre library based on the input timbre identifier, or the timbre embedding vector is determined based on the input reference audio data.

8. An electronic device, characterized in that, include: processor; A memory storing computer instructions for causing a processor to perform the method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The device stores computer instructions for implementing the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product is used to implement the method as described in any one of claims 1 to 6.

Images