Speech synthesis method and device

Abstract

The invention discloses a speech synthesis method, and belongs to the technical field of speech processing. The method comprises the steps that audio data and text data are acquired and preprocessed, and a token dictionary is obtained; inputting the token dictionary into a text-voice model to obtain a text-voice token set; inputting the text-voice token set into a causal flow matching model to obtain a Mel spectrogram; and inputting the Mel spectrogram into the vocoder model, and outputting target voice. According to the method, the speech synthesis speed and quality are improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech synthesis method and device. Background Art

[0002] With the rapid development of artificial intelligence technology, speech synthesis technology has also made considerable progress. However, existing speech synthesis methods have some defects in pronunciation accuracy. Faced with complex language phenomena, such as polyphones and polysemous words in Chinese and the special pronunciation rules of various languages, traditional algorithms have a high pronunciation error rate and are difficult to accurately output correct pronunciation that conforms to the context, which greatly affects the quality and comprehensibility of the synthesized speech.

[0003] When it comes to adapting speech rate and tone, traditional algorithms lack the flexibility to meet diverse demands. In real-world application scenarios, such as teaching, entertainment, and customer service, different speech rates and tones are required. Traditional algorithms only provide a superficial understanding of text semantics, failing to delve into deeper information such as emotion and context. Furthermore, they struggle to adjust speech naturally and fluently across multiple speeds. They also struggle to express different tones, such as seriousness, gentleness, and cheerfulness, resulting in a lack of realism and impact in synthesized speech. Regarding real-time synthesis performance, traditional algorithms suffer from slow synthesis speeds and significant latency when synthesizing long texts. This severely impacts user experience in demanding real-time scenarios, such as real-time voice interaction and live streaming, preventing timely speech synthesis and causing information lags. Traditional autoregressive models suffer from significant latency when synthesizing long texts. While non-autoregressive models can reduce latency, they still score low for prosodic naturalness.

[0004] Existing speech synthesis methods have problems such as low naturalness and accuracy of speech synthesis and poor real-time performance when facing complex language phenomena, adjusting speech speed and tone, and real-time response capabilities. Summary of the Invention

[0005] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a speech synthesis method and apparatus, which improves the speed and quality of speech synthesis.

[0006] In a first aspect, the present application provides a speech synthesis method, the method comprising:

[0007] Obtain audio data and text data and preprocess them to obtain a token dictionary;

[0008] Inputting the token dictionary into a text-to-speech model to obtain a text-to-speech token set;

[0009] Inputting the text-to-speech token set into a causal stream matching model to obtain a mel-spectrogram;

[0010] The mel-spectrogram is input into a vocoder model to output the target speech.

[0011] According to one embodiment of the present application, the acquiring of audio data and text data and preprocessing thereof to obtain a token dictionary includes:

[0012] Input the audio data and text data into the BERT model for annotation to obtain semantic features;

[0013] Resampling and channel averaging the audio data to obtain a voice token;

[0014] Normalize the text data to obtain a text token;

[0015] Based on a dynamic feature fusion algorithm, the semantic features, voice tokens and text tokens are fused to obtain a token dictionary.

[0016] According to one embodiment of the present application, inputting the audio data and text data into the BERT model for annotation to obtain semantic features includes:

[0017] Input the audio data and text data into the semantic feature layer, capture the long-distance dependency between the data through the attention mechanism, and obtain the first feature;

[0018] Inputting the first feature into a prosodic feature layer, marking the prosodic boundaries and stress positions of the text data, and obtaining a second feature;

[0019] The second feature is input into the special instruction layer to obtain a semantic feature.

[0020] According to one embodiment of the present application, inputting the token dictionary into a text-to-speech model to obtain a text-to-speech token set includes:

[0021] Input the token dictionary into a text-to-speech model, and generate a text-to-speech token set of a preset proportion based on the next token prediction method by autoregression;

[0022] The text-to-speech model is a Qwen2.5-0.5B model that does not include a text encoder and an embedding layer.

[0023] According to one embodiment of the present application, inputting the text-to-speech token set into a causal flow matching model to obtain a mel-spectrogram includes:

[0024] Upsampling and forward convolution are performed on the text-to-speech token set to obtain a speech token sequence;

[0025] Inputting the speech token sequence into multiple perceptual causal convolutional Transformer UNets for feature alignment to obtain a representation space for speech tokens;

[0026] The representation space of the speech token, the reference speech and the speaker embedding vector are input into the causal flow matching model for matching to obtain a mel-spectrogram.

[0027] According to one embodiment of the present application, inputting the mel-spectrogram into a vocoder model and outputting a target speech includes:

[0028] Inputting the Mel-spectrogram into a vocoder model for phase recovery to obtain an audio waveform;

[0029] A target speech is generated based on the audio waveform and outputted.

[0030] According to one embodiment of the present application, the calculation formula of the causal convolution Transforer UNet is as follows:

[0031]

[0032] in, is the interpolation function based on optimal transmission, v t is the vector field predicted by UNet, UNet θ is a causal convolutional TransformerUNet with parameters θ, t is the time step, v is the speaker embedding vector, and {μ} 1:L is the upsampled speech token sequence, is the masked mel-spectrogram.

[0033] In a second aspect, the present application provides a speech synthesis device, the device comprising:

[0034] The acquisition module is used to obtain audio data and text data and preprocess them to obtain a token dictionary;

[0035] A first processing module, configured to input the token dictionary into a text-to-speech model to obtain a text-to-speech token set;

[0036] A second processing module is configured to input the text-to-speech token set into a causal flow matching model to obtain a mel-spectrogram;

[0037] The output module is used to input the mel-spectrogram into the vocoder model and output the target speech.

[0038] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech synthesis method as described in the first aspect above when executing the computer program.

[0039] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech synthesis method as described in the first aspect above.

[0040] In a fifth aspect, the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the speech synthesis method as described in the first aspect.

[0041] In a sixth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the speech synthesis method as described in the first aspect above.

[0042] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application.

[0043] The speech synthesis method provided by the present invention has the following advantages over the prior art:

[0044] (1) The present invention obtains audio data and text data and preprocesses them to obtain a token dictionary, inputs it into a text-to-speech model, combines the causal flow matching model and the vocoder model to generate a mel-spectrogram and converts it into target speech. It can better simulate the speech characteristics of natural language, enhance the user experience of voice interaction, effectively improve the naturalness and fluency of speech synthesis, enhance the realism and clarity of speech synthesis, and is suitable for different application scenarios. It achieves a comprehensive breakthrough in real-time, accuracy, and adaptability of speech processing, and improves the performance and application effect of the speech synthesis system.

[0045] (2) The present invention obtains voice tokens and text tokens by inputting audio data and text data into the BERT model for annotation, and combines resampling, channel averaging processing and normalization processing of text data, and fuses them with semantic features through a dynamic feature fusion algorithm to obtain a token dictionary, which effectively improves the accuracy and efficiency of the speech synthesis system when processing multimodal data, can more accurately capture the relationship between audio and text, improves the naturalness and coherence of speech synthesis, realizes multi-language dynamic pronunciation adaptation, meets the special usage requirements of complex scenarios, and enhances the expressiveness of voice interaction and user experience.

[0046] (3) The present invention can effectively improve the efficiency and quality of text-to-speech synthesis by inputting a token dictionary into a text-to-speech model and combining it with an autoregressive next token prediction method to generate a text-to-speech token set of a preset proportion. By adopting the Qwen2.5-0.5B model that does not include a text encoder and an embedding layer, the complex preprocessing steps are reduced and the model structure is simplified. Through the streaming inference mechanism of the hybrid token sequence, not only can the details of speech synthesis be accurately captured, but also the fluency and naturalness of speech generation can be significantly improved, meeting the needs of real-time speech generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0048] Figure 1 This is one of the flow charts of the speech synthesis method provided in the embodiment of the present application;

[0049] Figure 2 Schematic diagram of the text-to-speech model provided in the embodiment of the present application;

[0050] Figure 3 It is a structural diagram of the causal flow matching model provided in the embodiment of the present application;

[0051] Figure 4 This is the second flow chart of the speech synthesis method provided in the embodiment of the present application;

[0052] Figure 5 is a structural diagram of a speech synthesis device provided in an embodiment of the present application;

[0053] Figure 6 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0055] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0056] The speech synthesis method, speech synthesis device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0057] The speech synthesis method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0058] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0059] In the following embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse, and a joystick.

[0060] The speech synthesis method provided in the embodiments of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the speech synthesis method. The electronic devices mentioned in the embodiments of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The speech synthesis method provided in the embodiments of the present application is described below using an electronic device as an example of the execution entity.

[0061] Figure 1 This is one of the flow charts of the speech synthesis method provided in the embodiment of the present application. Figure 1 As shown, the speech synthesis method includes: step 110, step 120, step 130 and step 140.

[0062] Step 110: Acquire audio data and text data and pre-process them to obtain a token dictionary;

[0063] It is easy to understand that by obtaining the audio data and text data input by the user, inputting the text data and audio data into the BERT model for annotation, and obtaining semantic features, after BERT preprocessing, the synthesized speech is more natural and fluent in emotional expression and more consistent with the text content.

[0064] In addition, the audio data and text data are processed separately to obtain audio tokens and text tokens. Based on the dynamic feature fusion algorithm, the semantic features, audio tokens and text tokens are organically spliced together to obtain a token dictionary.

[0065] In some embodiments, an adversarial sample generation mechanism can be introduced in the annotation stage. By generating challenging adversarial samples, the BERT model can be enabled to demonstrate greater robustness and adaptability when faced with ambiguous text content.

[0066] Optionally, the BERT model is continuously updated with the help of a continuous learning mechanism so that the model can promptly adapt to newly emerging complex scenario terms. The update step size of incremental learning is set to α = 0.001, and the learning rate attenuation coefficient is set to β = 0.95, so that the performance of the model can be continuously improved with the passage of time and the accumulation of data.

[0067] Optionally, jointly train subtasks such as sentiment classification and prosody prediction to improve the generalization ability of BERT model features through multi-task collaborative learning. During the multi-task learning process, a weighted loss function is used for training. The loss function calculation formula is as follows:

[0068] L=λ1L sentiment +λ2L prosody

[0069] λ1+λ2=1

[0070] Among them, L is the total loss value, λ1 and λ2 are the weights of sentiment classification and rhythm prediction tasks respectively, and L sentiment is the loss function value of the sentiment classification task, L prosody is the loss function value of the rhythm prediction task.

[0071] In some embodiments, λ1 = 0.6, λ2 = 0.4.

[0072] Step 120: input the token dictionary into a text-to-speech model to obtain a text-to-speech token set;

[0073] Figure 2 This is a schematic diagram of the structure of the text-speech model provided in the embodiment of the present application. Figure 2As shown in Figure 1, the token dictionary is input into the text-to-speech model, and an autoregressive method is used to generate mixed text tokens and speech tokens to obtain a text-to-speech token set.

[0074] Step 130: Input the text-to-speech token set into a causal flow matching model to obtain a mel-spectrogram;

[0075] Furthermore, the text-to-speech token set is input into the causal stream matching model, and the mel-spectrogram of the specified speaker is obtained through look-ahead pre-convolution, causal upsampling and causal Transformer encoder.

[0076] Step 140: Input the mel-spectrogram into a vocoder model to output the target speech.

[0077] Finally, the phase is restored through the pre-trained vocoder model, the Mel-spectrogram obtained by the causal flow matching model is processed, the Mel-spectrogram is converted back to the original audio signal, and the target speech is generated and output.

[0078] According to the speech synthesis method provided in the embodiment of the present application, by acquiring audio data and text data and performing preprocessing, a token dictionary is obtained, which is input into a text-to-speech model, and a mel-spectrogram is generated by combining a causal flow matching model and a vocoder model and converted into a target speech. This method can better simulate the speech characteristics of natural language, enhance the user experience of voice interaction, effectively improve the naturalness and fluency of speech synthesis, and enhance the realism and clarity of speech synthesis. It is suitable for different application scenarios, achieves a comprehensive breakthrough in real-time, accuracy, and adaptability of speech processing, and improves the performance and application effect of the speech synthesis system.

[0079] In some embodiments, the acquisition of audio data and text data and preprocessing to obtain a token dictionary includes:

[0080] Input the audio data and text data into the BERT model for annotation to obtain semantic features;

[0081] Resampling and channel averaging the audio data to obtain a voice token;

[0082] Normalize the text data to obtain a text token;

[0083] Based on a dynamic feature fusion algorithm, the semantic features, voice tokens and text tokens are fused to obtain a token dictionary.

[0084] For example, the sampling specification is set to a standard sampling rate of 16kHz, covering two-person voice interaction scenarios, single-person voice + ambient noise scenarios, two-person voice (transgender) + background noise scenarios, etc. Audio data and text data of different scenarios are collected as data sets and preprocessed. The preprocessing process is as follows:

[0085] (1) Standardize the parameters of the data set. The parameters that need to be adjusted are sampling rate, bit depth, and number of channels. First, the sampling rate of all WAV sound data is uniformly set to the standard sampling rate recommended by the system, for example, 16000Hz.

[0086] (2) The bit depth of audio data is unified to 16 bits to ensure the accuracy and consistency of sound data.

[0087] (3) Convert all audio to mono to reduce the complexity and potential information interference brought by multi-channel audio, and at the same time detect whether there are problems such as audio loss, long silent segments, and excessive noise.

[0088] Resample and average the audio data so that the audio channel and sampling rate meet the requirements. For example, if the target channel is mono and the target sampling rate is 22050Hz, first use the load function in the torchaudio library to load the sound data to obtain the audio waveform data, whose shape is (number of channels, number of sampling points). Then, average the data on dimension 0 to convert the multi-channel audio into single-channel audio. Finally, create a resampling transformer and apply it to the audio data to convert the audio sampling rate and obtain the processed speech token.

[0089] At the same time, the input text data is normalized, including removing blank characters at the beginning and end of the data; replacing blanks, superscripts, punctuation marks, and removing brackets; and splitting the text into paragraphs to form text tokens.

[0090] (4) An advanced annotation system is used to annotate audio data and text data. Table 1 is a schematic diagram of the annotation system provided in an embodiment of the present application. As shown in Table 1, not only the text annotation of the speech content is performed, but also the speech features such as emotion, speaking speed, and rhythm of the speech are marked with speech features.

[0091] Table 1

[0092]

[0093] Optionally, the text data and audio data are annotated using the BERT model to extract semantic features, parts of speech, sentiment, and other information of the text and audio, and special tokens are inserted into the text data and audio data. Table 2 is an example of tokens provided in an embodiment of the present application. As shown in Table 2, by inserting tokens, the intonation, speaking speed, and stress of the speech can be adjusted to improve the quality of the synthesized audio.

[0094] Table 2

[0095]

[0096]

[0097] Based on the dynamic feature fusion algorithm, the semantic features, voice tokens, and text tokens are fused to obtain the token dictionary. The process is as follows:

[0098] (a) Perform dimensional alignment on text data and audio data. Through linear transformation, the 768-dimensional feature vector output by BERT is accurately mapped to the same embedding space as the text token to ensure the consistency of the two dimensions and lay the foundation for subsequent fusion operations. Let the linear transformation matrix be W, then the dimensional calculation process is:

[0099] X aligned =X BERT W

[0100] Among them, X aligned is the aligned feature vector, X BERT is the feature vector output by BERT, and W is the linear transformation matrix.

[0101] (2) Sine position coding technology is used for encoding. The encoding formula is as follows:

[0102]

[0103] Among them, PE (pos，2i) is the value of the even dimension in the position encoding vector, PE (pos，2i+1) is the value of the odd dimension in the position encoding vector, i is the dimension in the position encoding vector, pos is the position of the word or token in the text, d model This effectively preserves the order information of the original text, allowing the model to fully utilize the contextual information of the text.

[0104] (3) An adaptive gating unit is introduced to construct a gating mechanism. The aligned feature vector is concatenated with the speech token and text token based on the gating formula. The weight distribution of semantic features, speech tokens and text tokens is dynamically adjusted to fuse the semantic features with the speech token and text token.

[0105] In some embodiments, a training set of 20,000 sentences, a test set of 3,000 sentences, and a validation set of 5,000 sentences were constructed. The data sources include a large amount of voice data obtained from open source platforms, as well as personalized voice samples uploaded by users. To verify the impact of different datasets on the performance of the BERT model, A. Basic dataset, B. Enhanced dataset (adding domain-specific data and dialect data), and C. Personalized dataset (including user-uploaded data) were constructed. Experimental results show that models trained with the enhanced datasets significantly improve pronunciation accuracy and semantic understanding, while personalized datasets enable the model to quickly adapt to the specific user's voice style, generating speech that better meets user needs.

[0106] In this embodiment, by inputting audio data and text data into the BERT model for annotation, combining resampling, channel averaging processing and normalization processing of text data, voice tokens and text tokens are obtained, and they are fused with semantic features through a dynamic feature fusion algorithm to obtain a token dictionary, which effectively improves the accuracy and efficiency of the speech synthesis system when processing multimodal data, can more accurately capture the relationship between audio and text, improve the naturalness and coherence of speech synthesis, realize multi-language dynamic pronunciation adaptation, meet the special usage needs of complex scenarios, and enhance the expressiveness of voice interaction and user experience.

[0107] In some embodiments, inputting the audio data and text data into a BERT model for annotation to obtain semantic features includes:

[0108] Input the audio data and text data into the semantic feature layer, capture the long-distance dependency between the data through the attention mechanism, and obtain the first feature;

[0109] Inputting the first feature into a prosodic feature layer, marking the prosodic boundaries and stress positions of the text data, and obtaining a second feature;

[0110] The second feature is input into the special instruction layer to obtain a semantic feature.

[0111] For example, a bidirectional Transformer encoder is used to construct a semantic annotation module to efficiently extract multi-level features of text content and obtain information such as semantic features, part of speech, and sentiment tendency of the input text. The specific process is as follows:

[0112] (1) Input audio data and text data into the semantic feature layer, capture the long-distance dependency between data through the attention mechanism, accurately extract deep semantic features such as entity relationships and sentiment polarity, and provide support for understanding the semantic picture of the text. The first feature is obtained. The formula of the attention mechanism is as follows:

[0113]

[0114] Among them, Q, K, and V are query matrix, key matrix, and value matrix respectively. T is the transposed matrix of K, d k is the dimension of the key vector.

[0115] (2) The first feature is input into the prosodic feature layer to obtain the second feature. The prosodic feature layer can be closely integrated with the phonetic rule library to accurately mark the prosodic boundaries (such as the boundaries defined by punctuation marks such as commas and periods) and stress positions of the text, ensuring that the prosody of the generated result is natural and smooth.

[0116] (3) The second feature is input into the special instruction layer, which has a strong recognition capability and can recognize the fine-grained instructions input by the user (e.g. <laughter>, [breath]), and generates corresponding control token sequences to provide more detailed control information for speech generation.

[0117] In this embodiment, by inputting audio data and text data into the BERT model for annotation, combined with layer-by-layer processing of semantic feature layers, prosodic feature layers, and special instruction layers, the deep semantic relationship between audio and text can be effectively captured. By capturing long-distance dependencies between data through the attention mechanism and annotating the prosodic boundaries and stress positions of the text, the accuracy and naturalness of speech synthesis are further improved. It can provide more precise semantic features during multimodal data processing, making the synthesized speech more natural and fluent, improving the accuracy of emotion matching and prosodic rationality, and significantly enhancing the effectiveness of the speech synthesis system and user experience.

[0118] In some embodiments, inputting the token dictionary into a text-to-speech model to obtain a text-to-speech token set includes:

[0119] Input the token dictionary into a text-to-speech model, and generate a text-to-speech token set of a preset proportion based on the next token prediction method by autoregression;

[0120] The text-to-speech model is a Qwen2.5-0.5B model that does not include a text encoder and an embedding layer.

[0121] It is easy to understand that by utilizing the language understanding ability of the text-to-speech model, processing the token dictionary, and autoregressively generating hybrid text-to-speech tokens, the text-to-speech model uses the pre-trained Qwen2.5-0.5B as the backbone and removes the text encoder and speaker embedding layer, which can reduce information leakage and enhance context understanding capabilities.

[0122] During training, the Qwen2.5-0.5B model uses the next token prediction method to autoregressively generate speech tokens based on the input text. In streaming mode, text and speech tokens are mixed in a predefined ratio of N:M, for example, N and M are set to 5 and 15 respectively. If the next token is a text token, a filler token will be predicted, prompting the next N text tokens to be connected during the inference phase. When the text tokens are used up, the "speech conversion" token T and the remaining speech tokens are connected in sequence to form a text-to-speech token set. If there are insufficient text tokens, virtual tokens are automatically generated to maintain inference continuity.

[0123] In this embodiment, by inputting the token dictionary into the text-speech model and combining the autoregressive next token prediction method to generate a preset proportion of text-speech token sets, the efficiency and quality of text-speech synthesis can be effectively improved. By adopting the Qwen2.5-0.5B model that does not include a text encoder and embedding layer, the complex preprocessing steps are reduced and the model structure is simplified. Through the streaming inference mechanism of the hybrid token sequence, not only can the details of speech synthesis be accurately captured, but also the fluency and naturalness of speech generation can be significantly improved, meeting the needs of real-time speech generation.

[0124] In some embodiments, inputting the text-to-speech token set into a causal flow matching model to obtain a mel-spectrogram comprises:

[0125] Upsampling and forward convolution are performed on the text-to-speech token set to obtain a speech token sequence;

[0126] Inputting the speech token sequence into multiple perceptual causal convolutional Transformer UNets for feature alignment to obtain a representation space for speech tokens;

[0127] The representation space of the speech token, the reference speech and the speaker embedding vector are input into the causal flow matching model for matching to obtain a mel-spectrogram.

[0128] Figure 3 This is a schematic diagram of the structure of the causal flow matching model provided in the embodiment of the present application. Figure 3 As shown in the figure, a mel-spectrogram with a frame rate of 50 Hz and a sampling rate of 24,000 is used as the acoustic feature. The speech token has a frame rate of 25 Hz, which does not match the frame rate of the mel-spectrogram. The speech token is first upsampled by a factor of 2 to match its frame rate with the mel-spectrogram. Before upsampling, a look-ahead convolutional layer is added to provide future information for the subsequent causal module. The look-ahead layer is implemented as a right-padded one-dimensional convolution with a padding size of P and a kernel size of P + 1.

[0129] After upsampling and forward convolution, the speech tokens enter multiple block-aware causal transformer blocks. This aligns the speech token representation space with the mel-spectrogram. During text-to-speech conversion, the representation space of speech tokens and mel-spectrograms may differ from the acoustic features in terms of feature space. The block-aware causal transformer blocks capture the temporal characteristics and contextual information of the speech signal, thereby bringing the feature representation of speech tokens closer to the acoustic features.

[0130] Using the causal flow matching model, the mel spectrogram is sampled with speech tokens, reference speech, and speaker embedding as conditions. The distribution of the target mel spectrogram is described by the probability density path from the prior distribution p0(x) to the data distribution q(x). The optimal transmission (OT) flow and ordinary differential equation (ODE) are used to match the vector field. The ODE is learned by training the causal convolutional Transformer UNet. The vector field ω t The calculation formula is as follows:

[0131]

[0132] X0~p0(X)=N(0,1)

[0133] X1~q(X)

[0134] Among them, ω t is a time-dependent vector field, It is an interpolation function based on optimal transmission. X0 is the sample obtained by sampling from the prior distribution p0(x). The prior distribution p0(x) obeys a normal distribution with mean 0 and the unit matrix as the covariance. X1 is the sample obtained by sampling from the data distribution q(x). t represents the time step, and its value range is [0,1].

[0135] In some embodiments, the calculation formula of the causal convolution Transforer UNet is as follows:

[0136]

[0137] in, is the interpolation function based on optimal transmission, v t is the vector field predicted by UNet, UNet θ is a causal convolutional Transformer UNet with parameters θ, t is the time step, v is the speaker embedding vector, and {μ} 1:L is the upsampled speech token sequence, is the masked mel-spectrogram.

[0138] It should be noted that during the training phase, the masked mel-spectrogram is obtained by randomly masking 70%-100% of the final frames in X1. During inference, it is provided by the mel-spectrogram extracted from the reference speech.

[0139] In this embodiment, using a causal convolutional Transformer UNet for vector field prediction effectively improves the accuracy of time-step feature alignment during speech generation. By feeding the upsampled speech token sequence, speaker embedding vector, and masked mel-spectrogram into the causal convolutional Transformer UNet for feature matching and fine-tuning, speech details can be more accurately reproduced, improving the naturalness and accuracy of speech synthesis.

[0140] In this embodiment, by inputting a text-to-speech token set into a causal stream matching model and combining it with upsampling, look-ahead convolution, and a perceptual causal convolution Transformer UNet for feature alignment, the accuracy and naturalness of speech synthesis can be effectively improved. By performing feature matching on the representation space of speech tokens and combining it with reference speech and speaker embedding vectors to generate a mel-spectrogram, speech details can be more accurately reproduced, improving the fluency and naturalness of speech synthesis.

[0141] In some embodiments, inputting the mel-spectrogram into a vocoder model to output a target speech comprises:

[0142] Inputting the Mel-spectrogram into a vocoder model for phase recovery to obtain an audio waveform;

[0143] A target speech is generated based on the audio waveform and outputted.

[0144] Figure 4 This is the second flow chart of the speech synthesis method provided in the embodiment of the present application. Figure 4 As shown in the figure, audio data and text data are input into the BERT model for data annotation, and then fused with text tokens and speech tokens to obtain a token dictionary. The token dictionary is input into the text-speech model to obtain a text-speech token set. The text-speech token set is input into the causal stream matching model to obtain a Mel-spectrogram. The Mel-spectrogram is input into the vocoder model to output the target speech.

[0145] In some embodiments, a speech synthesis system is constructed, which includes a BERT model, a text-to-speech model, a causal flow matching model, and a vocoder model. The speech synthesis system is optimized by fine-tuning parameters such as batchsize, learning rate, and number of iterations. During the fine-tuning process, a small-batch stochastic gradient descent algorithm is used to continuously adjust the weights of each model to improve the performance of the speech synthesis system on a specific data set. For example, the batchsize of the text-to-speech model is adjusted from the original 32 to 64, and the learning rate is appropriately reduced. This allows the speech synthesis system to converge more stably during training and reduce overfitting. The fine-tuned speech synthesis system exhibits better adaptability and accuracy when processing complex text data and diverse voice styles.

[0146] In this embodiment, by inputting the mel-spectrogram into a vocoder model for phase recovery, the accuracy of audio waveform restoration can be effectively improved, and the target speech can be generated based on the restored audio waveform. This method improves the quality and naturalness of the audio waveform through the phase recovery process, enhancing the clarity and realism of speech synthesis.

[0147] The speech synthesis method provided in the embodiment of the present application can be executed by a speech synthesis device. In the embodiment of the present application, the speech synthesis device provided in the embodiment of the present application is described by taking the speech synthesis device executing the speech synthesis method as an example.

[0148] The present application also provides a speech synthesis device, such as Figure 5 As shown, the speech synthesis device includes: an acquisition module 510 , a first processing module 520 , a second processing module 530 and an output module 540 .

[0149] An acquisition module 510 is used to acquire audio data and text data and perform preprocessing to obtain a token dictionary;

[0150] A first processing module 520 is configured to input the token dictionary into a text-to-speech model to obtain a text-to-speech token set;

[0151] A second processing module 530 is configured to input the text-to-speech token set into a causal flow matching model to obtain a mel-spectrogram;

[0152] The output module 540 is configured to input the mel-spectrogram into a vocoder model and output the target speech.

[0153] According to the speech synthesis method provided in the embodiment of the present application, by acquiring audio data and text data and performing preprocessing, a token dictionary is obtained, which is input into a text-to-speech model, and a mel-spectrogram is generated by combining a causal flow matching model and a vocoder model and converted into a target speech. This method can better simulate the speech characteristics of natural language, enhance the user experience of voice interaction, effectively improve the naturalness and fluency of speech synthesis, and enhance the realism and clarity of speech synthesis. It is suitable for different application scenarios, achieves a comprehensive breakthrough in real-time, accuracy, and adaptability of speech processing, and improves the performance and application effect of the speech synthesis system.

[0154] The speech synthesis device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented in the speech synthesis method embodiment are not described here.

[0155] In some embodiments, as Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, the various processes of the above-mentioned speech synthesis method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0156] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0157] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned speech synthesis method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0158] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0159] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned speech synthesis method when executed by a processor.

[0160] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0161] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned speech synthesis method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0162] It should be understood that the chip mentioned in the embodiments of the present application can also be called a device-level chip, a device chip, a chip device, or an on-chip device chip, etc.

[0163] It should be noted that, in this document, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the speech synthesis method of each embodiment of the present application.

[0165] In the description of this application, "first feature" and "second feature" may include one or more such features.

[0166] In the description of this application, "plurality" means two or more.

[0167] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0168] Throughout this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "example," "specific example," or "some examples" indicate that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present application. In this specification, illustrative uses of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0169] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and intent of the present application, and that the scope of the present application is defined by the claims and their equivalents.< / laughter>

Claims

1. A speech synthesis method, characterized in that: The method comprises: Obtain audio data and text data and preprocess them to obtain a token dictionary; Inputting the token dictionary into a text-to-speech model to obtain a text-to-speech token set; Inputting the text-to-speech token set into a causal stream matching model to obtain a mel-spectrogram; The mel-spectrogram is input into a vocoder model to output the target speech.

2. The speech synthesis method according to claim 1, wherein: The audio data and text data are obtained and preprocessed to obtain a token dictionary, including: Input the audio data and text data into the BERT model for annotation to obtain semantic features; Resampling and channel averaging the audio data to obtain a voice token; Normalize the text data to obtain a text token; Based on a dynamic feature fusion algorithm, the semantic features, voice tokens and text tokens are fused to obtain a token dictionary.

3. The speech synthesis method according to claim 2, wherein: The audio data and text data are input into the BERT model for annotation to obtain semantic features, including: Input the audio data and text data into the semantic feature layer, capture the long-distance dependency between the data through the attention mechanism, and obtain the first feature; Inputting the first feature into a prosodic feature layer, marking the prosodic boundaries and stress positions of the text data, and obtaining a second feature; The second feature is input into the special instruction layer to obtain a semantic feature.

4. The speech synthesis method according to claim 1, wherein: The token dictionary is input into a text-to-speech model to obtain a text-to-speech token set, including: Input the token dictionary into a text-to-speech model, and generate a text-to-speech token set of a preset proportion based on the next token prediction method by autoregression; The text-to-speech model is a Qwen2.5-0.5B model that does not include a text encoder and an embedding layer.

5. The speech synthesis method according to claim 1, wherein: The text-to-speech token set is input into a causal flow matching model to obtain a mel-spectrogram, including: Upsampling and forward convolution are performed on the text-to-speech token set to obtain a speech token sequence; Inputting the speech token sequence into multiple perceptual causal convolutional Transformer UNets for feature alignment to obtain a representation space for speech tokens; The representation space of the speech token, the reference speech and the speaker embedding vector are input into the causal flow matching model for matching to obtain a mel-spectrogram.

6. The speech synthesis method according to claim 1, wherein: The step of inputting the mel-spectrogram into a vocoder model and outputting a target speech comprises: Inputting the Mel-spectrogram into a vocoder model for phase recovery to obtain an audio waveform; A target speech is generated based on the audio waveform and outputted.

7. The speech synthesis method according to claim 5, characterized in that The calculation formula of the causal convolution Transforer UNet is as follows: in, is the interpolation function based on optimal transmission, v t is the vector field predicted by UNet, UNet θ is a causal convolutional TransformerUNet with parameters θ, t is the time step, v is the speaker embedding vector, and {μ} 1:L is the upsampled speech token sequence, is the masked mel-spectrogram.

8. A speech synthesis device, implemented by the speech synthesis method according to any one of claims 1 to 7, characterized in that: The device comprises: The acquisition module is used to obtain audio data and text data and preprocess them to obtain a token dictionary; A first processing module, configured to input the token dictionary into a text-to-speech model to obtain a text-to-speech token set; A second processing module is configured to input the text-to-speech token set into a causal flow matching model to obtain a mel-spectrogram; The output module is used to input the mel-spectrogram into the vocoder model and output the target speech.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the speech synthesis method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 7 is implemented.

Images