Speech synthesis method, device, equipment and storage medium

Abstract

The embodiments of the present application relate to the field of data processing technology, and more specifically, to a speech synthesis method, apparatus, device, and storage medium, which are intended to enhance the expressiveness of speech synthesis. The method comprises: receiving a target text; obtaining corresponding text features from the target text, wherein the text features include phoneme information, word boundary information, and sentence information; inputting the text features into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features; performing multi-level representation fusion on the semantic features to obtain multi-level semantic features; aligning the multi-level semantic features with words and durations, and aligning the multi-level semantic features with phonemes and durations through a multi-head attention mechanism to obtain the aligned multi-level semantic features; and obtaining the speech corresponding to the target text based on the aligned multi-level semantic features.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and specifically, to a speech synthesis method, apparatus, device, and storage medium. Background Art

[0002] Text-to-speech (TTS) is a technology that converts text into speech. It plays a crucial role in interpersonal interaction and is currently a hotly researched area. Existing speech synthesis technologies primarily extract information from the input sentence, convert the sentence's textual features into a speech representation, then convert the speech representation into an acoustic representation, ultimately generating the corresponding acoustic waveform to complete speech synthesis.

[0003] The speech synthesis systems in the existing technology often ignore the rich representation of text and context, and the expressiveness of speech synthesis is insufficient. Summary of the Invention

[0004] The embodiments of the present application provide a speech synthesis method, apparatus, device, and storage medium, aiming to improve the expressiveness of speech synthesis.

[0005] A first aspect of an embodiment of the present application provides a speech synthesis method, the method comprising:

[0006] receiving target text;

[0007] Acquire corresponding text features from the target text, wherein the text features include phoneme information, word boundary information, and sentence information;

[0008] Inputting the text features into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features;

[0009] Performing multi-level representation fusion on the semantic features to obtain multi-level semantic features;

[0010] Aligning the multi-level semantic features by words and durations, and aligning the multi-level semantic features by phonemes and durations through a multi-head attention mechanism to obtain the aligned multi-level semantic features;

[0011] The speech corresponding to the target text is obtained based on the aligned multi-level semantic features.

[0012] Optionally, obtaining corresponding text features from the target text, wherein the text features include phoneme information, word boundary information, and sentence information, includes:

[0013] Recognizing the phonemes of each word in the target text to obtain the phoneme information;

[0014] Identifying the number of phonemes included in each word in the target text to obtain word boundary information;

[0015] Each sentence contained in the target text is identified to obtain the sentence information.

[0016] Optionally, the step of training the semantic model includes:

[0017] Collecting training data, wherein the training data includes text data in at least one field;

[0018] Classifying the training data according to sentence length to obtain classified training data;

[0019] performing mask processing on the classified training data to obtain processed training data;

[0020] Design corresponding symbol id for the training data;

[0021] The semantic model is trained using the processed training data and the symbol ID to obtain the semantic model.

[0022] Optionally, performing multi-level representation fusion on the semantic features to obtain multi-level semantic features includes:

[0023] Extracting corresponding sentence features from the semantic features;

[0024] The sentence features are fused with the phoneme features and word features included in the semantic features to obtain the multi-level semantic features.

[0025] Optionally, extracting corresponding sentence features from the semantic features includes:

[0026] The semantic features are arranged in a sentence sequence, and adjacent sentences are regarded as a sentence pair to obtain at least one sentence pair;

[0027] Extracting semantic features corresponding to the sentence pairs through a pre-trained cross-coding model to obtain corresponding sentence features;

[0028] The sentence features are processed using a multi-head attention mechanism to obtain processed sentence features.

[0029] Optionally, aligning the multi-level semantic features by words and durations, and aligning the multi-level semantic features by phonemes and durations through a multi-head attention mechanism to obtain the aligned multi-level semantic features, includes:

[0030] performing hard alignment on words and durations in the multi-level semantic features;

[0031] Softly aligning the phonemes and durations in the multi-level semantic features through a multi-head attention mechanism;

[0032] When the words and the phonemes are aligned with the duration, the speech features are obtained.

[0033] Optionally, obtaining the speech corresponding to the target text according to the aligned multi-level semantic features includes:

[0034] Inputting the aligned multi-level semantic features into the decoder;

[0035] Processing the multi-level semantic features by the decoder to obtain corresponding speech features;

[0036] The speech features are input into a vocoder to obtain the speech.

[0037] According to a second aspect of the present application, a speech synthesis device is provided, comprising:

[0038] A text receiving module, used for receiving target text;

[0039] A text feature extraction module is used to obtain corresponding text features from the target text, wherein the text features include phoneme information, word boundary information and sentence information;

[0040] A semantic feature extraction module, configured to input the text features into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features;

[0041] A multi-level semantic feature acquisition module is used to perform multi-level representation fusion on the semantic features to obtain multi-level semantic features;

[0042] a speech feature acquisition module, configured to align words and durations of the multi-level semantic features, and to align phonemes and durations of the multi-level semantic features using a multi-head attention mechanism, thereby obtaining the aligned multi-level semantic features;

[0043] The speech acquisition module is used to obtain the speech corresponding to the target text according to the multi-level semantic features after alignment processing.

[0044] Optionally, the text feature extraction module includes:

[0045] A phoneme information acquisition submodule, configured to recognize the phonemes of each word in the target text to obtain the phoneme information;

[0046] A word boundary information acquisition submodule is used to identify the number of phonemes included in each word in the target text to obtain the word boundary information;

[0047] The sentence recognition submodule is used to recognize each sentence contained in the target text to obtain the sentence information.

[0048] Optionally, the step of training the semantic model includes:

[0049] Collecting training data, wherein the training data includes text data in at least one field;

[0050] Classifying the training data according to sentence length to obtain classified training data;

[0051] performing mask processing on the classified training data to obtain processed training data;

[0052] Design corresponding symbol id for the training data;

[0053] The semantic model is trained using the processed training data and the symbol ID to obtain the semantic model.

[0054] Optionally, the multi-level semantic feature acquisition module includes:

[0055] A sentence feature acquisition submodule is used to extract corresponding sentence features from the semantic features;

[0056] The feature fusion submodule is used to fuse the sentence feature with the phoneme feature and word feature included in the semantic feature to obtain the multi-level semantic feature.

[0057] Optionally, the sentence feature acquisition submodule includes:

[0058] A sentence pair acquisition submodule is used to obtain at least one sentence pair by taking the semantic features into account according to the sentence sequence and treating adjacent sentences as a sentence pair;

[0059] A sentence feature extraction submodule is used to extract semantic features corresponding to the sentence pairs through a pre-trained cross-coding model to obtain corresponding sentence features;

[0060] The sentence feature processing submodule is used to process the sentence features using a multi-head attention mechanism to obtain processed sentence features.

[0061] Optionally, the speech feature acquisition module includes:

[0062] An alignment submodule, configured to hard align words and durations in the multi-level semantic features;

[0063] a soft alignment submodule, configured to softly align phonemes and durations in the multi-level semantic features through a multi-head attention mechanism;

[0064] The speech feature acquisition submodule is used to obtain the speech feature when the word and the phoneme are aligned with the duration.

[0065] Optionally, the voice acquisition module includes:

[0066] A decoding submodule, configured to input the aligned multi-level semantic features into a decoder;

[0067] A voice code acquisition submodule, configured to process the multi-level semantic features through the decoder to obtain corresponding speech features;

[0068] The audio acquisition submodule is used to input the speech features into a vocoder to obtain the speech.

[0069] A third aspect of an embodiment of the present application provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the method described in the first aspect of the present application are implemented.

[0070] A fourth aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described in the first aspect of the present application are implemented.

[0071] The speech synthesis method proposed in the present application is adopted to receive a target text; obtain corresponding text features from the target text, wherein the text features include phoneme information, word boundary information and sentence information; input the text features into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features; perform multi-level representation fusion on the semantic features to obtain multi-level semantic features; align the words and durations of the multi-level semantic features, and align the phonemes and durations of the multi-level semantic features through a multi-head attention mechanism to obtain the aligned multi-level semantic features; and obtain the speech corresponding to the target text based on the aligned multi-level semantic features.

[0072] In this application, a pre-trained semantic model is used to process text features containing phoneme information, word boundary information, and sentence information, so that semantic features at the phoneme and word levels can be obtained, thereby improving the ability to capture fine-grained features in speech synthesis. The semantic features are then fused with multi-level representations to obtain multi-level semantic features, enabling the system to effectively capture the semantics of the context and improve the naturalness and expressiveness of the synthesized speech. The multi-level semantic features are then aligned with the phonemes and durations through a multi-head attention mechanism, which can improve the alignment accuracy, avoid noise caused by unstable phoneme boundaries, and regenerate the corresponding speech, effectively improving the expressiveness of speech synthesis in all aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0074] Figure 1 This is a flow chart of a speech synthesis method proposed in one embodiment of the present application;

[0075] Figure 2 This is a flow chart of an end-to-end text-to-speech synthesis system with multi-level language and speech representation proposed in one embodiment of the present application;

[0076] Figure 3 This is a structural diagram of a speech synthesis model proposed in one embodiment of the present application;

[0077] Figure 4 This is a schematic diagram of the speech synthesis model processing flow proposed in one embodiment of the present application;

[0078] Figure 5 is a schematic diagram of a speech synthesis device proposed in one embodiment of the present application;

[0079] Figure 6 FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0080] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0081] refer to Figure 1 , Figure 1This is a flow chart of the speech synthesis method proposed in one embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0082] S11: Receive target text.

[0083] In this embodiment, the target text is the text that needs to be synthesized into speech.

[0084] Text-to-speech (TTS), also known as text-to-speech conversion, is a technology that converts text into speech. Since speech synthesis is the final step in human-computer interaction, it converts the text processed in previous steps into target audio and feeds it back to the user. Therefore, the goal of speech synthesis is to enable machines to produce natural and pleasant sounds, thereby resolving the difficulties inherent in human-computer interaction. Consequently, speech synthesis has garnered widespread attention from researchers and technicians. After years of development, end-to-end speech synthesis has become the mainstream technology in speech synthesis research and application. The goal of speech synthesis is to generate intelligible and natural audio. Naturalness, in turn, depends largely on the expressiveness of the voice.

[0085] The speech synthesis system mainly consists of front-end analysis, acoustic model and vocoder. The process is as follows: the input text is converted into language representation through front-end analysis, and then these language representations are converted into intermediate acoustic representations such as mel spectrum through the acoustic model. Finally, the acoustic representation is converted into a waveform through the vocoder. From this, it can be intuitively seen that the burden of the acoustic model is greater than that of the vocoder. This is because the vocoder only needs to restore the phase information of the given spectrogram, while the acoustic model needs to narrow the gap between the text and the spectrogram. The information gap between the text and the spectrogram is large, making the task of the acoustic model more difficult than that of the vocoder. Moreover, most of the existing expressive speech synthesis still focuses on modeling, decoupling, controlling and migrating the rhythm and emotional style of the reference audio, while ignoring the rich representations contained in the text.

[0086] Furthermore, most speech synthesis systems only consider information within the current sentence, while ignoring contextual textual information. As the requirements for naturalness and expressiveness of speech synthesis systems continue to increase, these systems lack the ability to produce paragraph-level speech that is sufficiently natural and expressive. This is because human speech is contextually coherent within a paragraph, rather than each sentence being independent and context-independent.

[0087] In this embodiment, speech synthesis is performed using a speech synthesis model. This model is based on FastPitch, but replaces the speech codec with a pre-trained semantic model (PhonemeBERT), and then combines it with a multi-head attention mechanism. This model first needs to receive the target text.

[0088] For example, the target text may be a text in any language, such as Chinese, English, etc., which includes characters, words, sentences, etc.

[0089] S12: Acquire corresponding text features from the target text, where the text features include phoneme information, word boundary information, and sentence information.

[0090] In this embodiment, phoneme information, a phoneme is the smallest unit that makes up a pronunciation, a word boundary is the number of phonemes contained in a word, and the position of the first and last phonemes of a word can be determined based on the number of phonemes, and sentence information is the number of sentences contained in a text and other information.

[0091] In this embodiment, the target text is analyzed by the front-end analysis module to obtain corresponding text features.

[0092] In this embodiment, the specific steps of obtaining corresponding text features from the target text, wherein the text features include phoneme information, word boundary information, and sentence information, include:

[0093] S12-1: Recognize the phonemes of each word in the target text to obtain the phoneme information.

[0094] In this embodiment, the phoneme information includes information such as all phonemes appearing in the target text and the number of each phoneme.

[0095] In this embodiment, the phonemes of each word in the target text are firstly recognized to obtain the corresponding phoneme information.

[0096] S12-2: Identify the number of phonemes included in each word in the target text to obtain the word boundary information.

[0097] In this embodiment, the front-end analysis module identifies the number of phonemes included in each word in the text. By determining the number of phonemes included in each word, the first and last phonemes of each word can be determined, and word boundary information can be obtained.

[0098] For example, the target text contains the word "I am", which includes the phonemes "w, o, s, h, i".

[0099] S12-3: Recognize each sentence contained in the target text to obtain the sentence information.

[0100] In this embodiment, each sentence contained in the target text is identified by the front-end analysis module to obtain sentence information.

[0101] S13: Inputting the text features into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features.

[0102] In this embodiment, the pre-trained semantic model is the PhonemeBERT model. PhonemeBERT is a pre-trained model based on the BERT architecture. It aims to capture rich contextual information through phoneme-level modeling and generate fine-grained phoneme representations to improve the sound quality and expressiveness of the speech synthesis system.

[0103] In this example, pre-training is performed based on the BERT model to build semantic representations at the word and phoneme levels, enhancing the system's generalization capabilities. This approach ensures that the model can achieve good synthesis results even under low-resource data conditions, improving the accuracy of speech generation.

[0104] In this example, the phoneme encoder in FastPitch is replaced with the pre-trained PhonemeBERT model to leverage its fine-grained phoneme representations. Combined with word boundary information, word-level semantic representations can be obtained, enhancing the speech synthesis model's ability to model speech details and making the synthesized speech more expressive.

[0105] In this embodiment, text features are input into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features. The semantic model extracts phoneme-level semantic features from the text features in combination with word boundary information.

[0106] In this embodiment, phoneme-level and word-level semantic representations are introduced, and the phoneme encoder in FastPitch is replaced by PhonemeBERT to improve the fine-grained feature capture capability of speech synthesis.

[0107] S14: Perform multi-level representation fusion on the semantic features to obtain multi-level semantic features.

[0108] In this embodiment, the multi-level semantic features include phoneme-level features, word-level features, and sentence-level features.

[0109] In this embodiment, multi-level context is taken into account simultaneously, the factors and words of the previous sentence and the current sentence are connected, and the coarse-grained sentence-level semantic representation of adjacent sentences is extracted through a pre-trained cross-utterance encoder, thereby capturing the semantic relevance between context sentences.

[0110] In this embodiment, the specific steps of performing multi-level representation fusion on semantic features to obtain multi-level semantic features include:

[0111] S14-1: Extracting corresponding sentence features from the semantic features.

[0112] In this embodiment, firstly, the corresponding sentence features are extracted from the semantic features. The specific steps include:

[0113] S14-1-1: Calculate the semantic features according to the sentence sequence, and treat adjacent sentences as a sentence pair to obtain at least one sentence pair.

[0114] In this embodiment, each semantic feature has a corresponding sentence feature. The received semantic features are arranged in a sentence sequence, and adjacent sentences are treated as a sentence pair to obtain at least one sentence pair.

[0115] For example, the adjacent sentence sequence Divide adjacent sentence sequences into 2L pairs of sentences, where represents the current sentence, and L is the number of sentences considered in the previous and next contexts.

[0116] S14-1-2: Extract the semantic features corresponding to the sentence pairs using a pre-trained cross-coding model to obtain corresponding sentence features.

[0117] In this embodiment, the cross-coding model is a BERT model, and the semantic features corresponding to the sentence pairs are extracted through the pre-trained cross-coding model to obtain corresponding sentence features.

[0118] S14-1-3: Use a multi-head attention mechanism to process the sentence features to obtain processed sentence features.

[0119] In this embodiment, the head attention mechanism is an improved attention mechanism designed to improve the model's expressiveness and learning capabilities by splitting input features into multiple "heads" and processing each head independently. To incorporate sentence-level information into the phoneme-level representation, sentence-level feature extraction is performed using BERT. At this stage, the model aligns the global sentence-level representation with the local phoneme-level representation. Specifically, the phoneme-level representation serves as the query, and the sentence-level representation serves as the key. The multi-head attention mechanism calculates the correlation between each phoneme and the sentence-level information and incorporates global contextual information to enhance the phoneme's features, ensuring that the processed sentence features contain more factor features.

[0120] In this embodiment, after obtaining the sentence features, the sentence features are input into the multi-head attention module of the speech synthesis model, and the sentence features are processed through the multi-head attention mechanism to obtain the processed sentence features.

[0121] S14-2: Fusing the sentence features with the phoneme features and word features included in the semantic features to obtain the multi-level semantic features.

[0122] In this embodiment, after the processed sentence features are obtained, the obtained sentence features are fused with the phoneme features and word features to obtain multi-level semantic features.

[0123] In this example, pre-training is performed based on the BERT model to build semantic representations at the word and phoneme levels, enhancing the system's generalization capabilities. This approach ensures that the model can achieve good synthesis results even under low-resource data conditions, improving the accuracy of speech generation.

[0124] In this implementation, a pre-trained BERT model utilizes a masked language model (MLM) strategy to enhance the system's generalization capabilities and ensure stable synthesis performance under limited resource conditions. A cross-utterance encoder extracts coarse-grained sentence-level semantic information, enabling the system to effectively capture contextual semantics and improve the naturalness and expressiveness of synthesized speech.

[0125] S15: Aligning the words and durations of the multi-level semantic features, and aligning the phonemes and durations of the multi-level semantic features through a multi-head attention mechanism to obtain the aligned multi-level semantic features.

[0126] In this embodiment, when synthesizing speech, it is necessary to align each word and each phoneme with the duration, thereby obtaining the multi-level semantic features after the alignment process.

[0127] In this embodiment, the multi-level semantic features are processed by a multi-head attention mechanism, and the words and phonemes therein are aligned with their respective durations, thereby obtaining the aligned multi-level semantic features. The specific steps include:

[0128] S15-1: Hard aligning the words and durations in the multi-level semantic features.

[0129] In this embodiment, hard alignment is performed on the words and durations in the multi-level semantic features. Hard alignment is to set the duration of each word in the speech to the same time.

[0130] For example, the duration of the word "eat" included in the multi-level semantic features is hard-aligned to 1 second, and each time the word appears in the synthesized speech, it lasts for 1 second.

[0131] S15-2: Softly align the phonemes and durations in the multi-level semantic features through a multi-head attention mechanism.

[0132] In this embodiment, the phonemes and durations in the multi-level semantic features are soft-aligned through a multi-head attention mechanism. Soft alignment is to flexibly set the pronunciation time of each phoneme based on the position and frequency of each phoneme in different words.

[0133] In this embodiment, for example, a phoneme "w" in the word "we" has its duration aligned to 0.1 seconds, and in the word "look at me", has its duration aligned to 0.2 seconds.

[0134] S15-3: When the word and the phoneme are aligned with the duration, the speech feature is obtained.

[0135] In this embodiment, when the words, phonemes and durations are aligned, the multi-level semantic features after the alignment process are obtained.

[0136] In this embodiment, given the uncertainty of phoneme boundaries, traditional hard alignment methods may introduce errors. Therefore, this model uses word-level hard alignment combined with a multi-head attention mechanism to achieve soft alignment of phonemes to durations. This flexible alignment strategy can effectively reduce alignment errors, ensure the accuracy of phoneme duration processing, and thus optimize the quality of decoder input features. Word-level hard alignment combined with a multi-head attention mechanism achieves soft alignment of phonemes to durations, improving alignment accuracy and avoiding noise introduced by the uncertainty of phoneme boundaries.

[0137] S16: Obtain the speech corresponding to the target text according to the aligned multi-level semantic features.

[0138] In this embodiment, after obtaining the speech features, the speech corresponding to the target text is obtained based on the processed multi-level semantic features. The specific steps are: S16-1: inputting the aligned multi-level semantic features into the decoder.

[0139] In this embodiment, the decoder is used to convert semantics, converting semantic features into speech features.

[0140] In this embodiment, the aligned multi-level semantic features are first input into the decoder.

[0141] S16-2: Processing the multi-level semantic features through the decoder to obtain corresponding speech features.

[0142] In this embodiment, the processed multi-level semantic features are decoded by a decoder to obtain corresponding speech features.

[0143] S16-3: Input the speech features into a vocoder to obtain the speech.

[0144] In this embodiment, the obtained speech features are input into a corresponding vocoder for speech synthesis to obtain the corresponding speech.

[0145] In this embodiment, after obtaining the multi-level fusion features, it is necessary to use a pitch predictor to predict the pitch of each phoneme, and use a duration predictor to predict the duration of each word and each phoneme, while combining the multi-head attention mechanism to align the durations.

[0146] In this embodiment, through multi-level semantic representation, the synthesized speech is significantly enhanced in terms of emotional expression and naturalness, adapting to more complex language scenarios. The soft alignment strategy effectively reduces alignment errors, ensuring that each word is clearly distinguishable, especially when dealing with repeated words and complex sentences. The optimized system has a higher processing speed, meets the low latency requirements of streaming TTS systems, and is suitable for real-time applications. The system can be flexibly adjusted according to different text types, improving the application effect in various scenarios and enhancing the practicality of speech synthesis technology.

[0147] In this embodiment, after obtaining the multi-level fusion features, it is necessary to use a pitch predictor to predict the pitch of each phoneme, and use a duration predictor to predict the duration of each word and each phoneme, while combining the multi-head attention mechanism to align the durations.

[0148] In another embodiment of the present application, the step of training the semantic model includes:

[0149] S21: Collect training data, where the training data includes text data in at least one field.

[0150] In this embodiment, training data is first collected, and the collected training data includes text data in at least one field.

[0151] For example, 10GB of open source text data is collected from multiple fields such as dialogue, popular science, and news, and then cleaned and deduplicated to ensure the quality of the training data.

[0152] S22: Classify the training data according to the length of the sentences to obtain classified training data.

[0153] In this embodiment, batch sampling optimization is performed on the data, and the collected training data is classified according to the length of the sentences to obtain classified training data.

[0154] In this embodiment, since the training data comes from a wide range of sources and the text lengths vary greatly, a batch sampling strategy is used to classify data of different lengths, thereby improving the efficiency of model training and avoiding unnecessary computational overhead.

[0155] S23: performing mask processing on the classified training data to obtain processed training data.

[0156] In this embodiment, masking is to hide some data, thereby achieving the purpose of training the model.

[0157] In this embodiment, masking is performed on the classified training data to obtain processed training data. Generally, masking is performed on a portion of the data.

[0158] For example, a masked language model (MLM) is used to enhance the model's generalization capabilities in low-resource scenarios. The specific strategy is: 30% of the input is masked, 80% of the phonemes in the masked portion are masked, 10% are replaced with other phonemes, and the remaining 10% remain unchanged.

[0159] S24: Design corresponding symbol IDs for the training data.

[0160] In this embodiment, the symbol id is a pre-set id corresponding to each phoneme, word, or sentence.

[0161] For example, corresponding symbol ids are designed for the training data, for example, the symbol id of “a” is 1, the symbol id of “o” is 2, and so on.

[0162] S25: Training the semantic model using the processed training data and the symbol ID to obtain the semantic model.

[0163] In this embodiment, the semantic model is iteratively trained using the processed training data and symbol ID. In each round of training, the training data and symbol ID are input into the semantic model. The semantic model automatically extracts the features included in the training data to obtain the corresponding semantic features, and then adjusts the model parameters according to the actual data until the model training is completed and a trained semantic model is obtained.

[0164] In this embodiment, to adapt to streaming applications, the PhonemeBERT model structure can be streamlined and lightweighted to reduce computing resource usage, ensuring high efficiency and scalability in real-time applications, making it suitable for streaming TTS scenarios. For example, the number of model layers can be reduced from 6 to 3, streamlining the model without affecting performance.

[0165] In another embodiment of the present application, the speech synthesis model is also obtained through pre-training. During training, the speech synthesis model is trained using the collected training materials to obtain a trained speech synthesis model. In the final optimization effect evaluation, different types of texts were tested. The results showed that the system not only has a significant improvement in expressiveness and naturalness, but also significantly accelerates the processing speed, meeting the low latency requirements of the streaming TTS system. In particular, in texts containing repeated words and complex sentence structures, the system can clearly read each word, avoiding the problem of ambiguity in repeated words.

[0166] refer to Figure 2 , Figure 2 This is a flowchart of an end-to-end text-to-speech synthesis system with multi-level language and speech representation proposed in one embodiment of the present application. Figure 2 As shown in the figure, the synthesized text should first undergo front-end processing, that is, text regularization to convert it from non-normalized spoken text to normalized written language, and then convert it into pinyin through g2p and process it into phonemes to obtain the features required by the model, and then pass it through the back-end model to obtain the audio content corresponding to the input text.

[0167] refer to Figure 3 , Figure 3 This is a diagram of the speech synthesis model structure proposed in one embodiment of the present application. Figure 3 As shown, it includes a text encoder and a sentence-level encoder. The encoder is the PhonemeBERT model, which generates corresponding semantic features. It also includes a multi-head attention mechanism module. After processing by the multi-head attention module, multi-level fusion features are obtained. The pitch predictor is used to predict the pitch of each phoneme, and the duration predictor is used to predict the duration of each phoneme. The decoder is used to decode the speech features to obtain the corresponding voice code, and the vocoder is used to synthesize the corresponding audio according to the voice code.

[0168] refer to Figure 4 , Figure 4 This is a flowchart of the speech synthesis model processing proposed in one embodiment of the present application. Figure 4As shown, based on the FastPitch model, the text encoder is replaced with PhonemeBERT to extract phoneme-level semantic representations. Based on word boundary information, word-level semantic representations are upsampled and combined with phoneme- and word-level information. The phoneme- and word-level information of two adjacent sentences are aligned using an attention mechanism to extract coarse-grained sentence-level semantic information. For paired sentences, sentence features are extracted using a pre-trained cross encoder. These features are then processed using a multi-head attention mechanism to obtain processed sentence features. These features are then fused with phoneme and word features, and finally, temporally aligned using a multi-head attention mechanism. Pitch prediction is then performed to obtain the corresponding mel-spectrograms. The phoneme- and word-level information of two adjacent sentences are aligned using an attention mechanism to extract coarse-grained sentence-level semantic information. This multi-level representation approach enables the system to perform more natural speech synthesis when handling long sentences, repeated words, and sentences with significant emotional variation, resulting in stronger expressiveness. In particular, the system effectively avoids ambiguity when handling repeated words, ensuring that each word is clearly distinguishable. To ensure more accurate alignment of phonemes to speech features, this example introduces word-level hard alignment and combines it with a multi-head attention mechanism to perform soft alignment of phoneme durations. In practice, this alignment strategy effectively avoids noise and errors caused by uncertainty in phoneme boundaries, thereby improving the clarity and fluency of synthesized speech.

[0169] Based on the same inventive concept, an embodiment of the present application provides a speech synthesis device. Figure 5 , Figure 5 FIG is a schematic diagram of a speech synthesis device 500 proposed in an embodiment of the present application. Figure 5 As shown, the device includes:

[0170] A text receiving module 501 is used to receive a target text;

[0171] A text feature extraction module 502 is used to obtain corresponding text features from the target text, wherein the text features include phoneme information, word boundary information, and sentence information;

[0172] A semantic feature extraction module 503 is configured to input the text feature into a pre-trained semantic model to obtain a phoneme-level semantic feature corresponding to the text feature;

[0173] A multi-level semantic feature acquisition module 504 is used to perform multi-level representation fusion on the semantic features to obtain multi-level semantic features;

[0174] The speech feature acquisition module 505 is configured to align the multi-level semantic features by words and durations, and to align the multi-level semantic features by phonemes and durations using a multi-head attention mechanism to obtain the aligned multi-level semantic features.

[0175] The speech acquisition module 506 is configured to obtain the speech corresponding to the target text according to the aligned multi-level semantic features.

[0176] Optionally, the text feature extraction module includes:

[0177] A phoneme information acquisition submodule, configured to recognize the phonemes of each word in the target text to obtain the phoneme information;

[0178] A word boundary information acquisition submodule is used to identify the number of phonemes included in each word in the target text to obtain the word boundary information;

[0179] The sentence recognition submodule is used to recognize each sentence contained in the target text to obtain the sentence information.

[0180] Optionally, the step of training the semantic model includes:

[0181] Collecting training data, wherein the training data includes text data in at least one field;

[0182] Classifying the training data according to sentence length to obtain classified training data;

[0183] performing mask processing on the classified training data to obtain processed training data;

[0184] Design corresponding symbol id for the training data;

[0185] The semantic model is trained using the processed training data and the symbol ID to obtain the semantic model.

[0186] Optionally, the multi-level semantic feature acquisition module includes:

[0187] A sentence feature acquisition submodule is used to extract corresponding sentence features from the semantic features;

[0188] The feature fusion submodule is used to fuse the sentence feature with the phoneme feature and word feature included in the semantic feature to obtain the multi-level semantic feature.

[0189] Optionally, the sentence feature acquisition submodule includes:

[0190] A sentence pair acquisition submodule is used to obtain at least one sentence pair by taking the semantic features into account according to the sentence sequence and treating adjacent sentences as a sentence pair;

[0191] A sentence feature extraction submodule is used to extract semantic features corresponding to the sentence pairs through a pre-trained cross-coding model to obtain corresponding sentence features;

[0192] The sentence feature processing submodule is used to process the sentence features using a multi-head attention mechanism to obtain processed sentence features.

[0193] Optionally, the speech feature acquisition module includes:

[0194] An alignment submodule, configured to hard align words and durations in the multi-level semantic features;

[0195] a soft alignment submodule, configured to softly align phonemes and durations in the multi-level semantic features through a multi-head attention mechanism;

[0196] The speech feature acquisition submodule is used to obtain the speech feature when the word and the phoneme are aligned with the duration.

[0197] Optionally, the voice acquisition module includes:

[0198] A decoding submodule, configured to input the aligned multi-level semantic features into a decoder;

[0199] A voice code acquisition submodule, configured to process the multi-level semantic features through the decoder to obtain corresponding speech features;

[0200] The audio acquisition submodule is used to input the speech features into a vocoder to obtain the speech.

[0201] Based on the same inventive concept, another embodiment of the present application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments of the present application.

[0202] Based on the same inventive concept, another embodiment of the present application provides an electronic device, Figure 6 This is a schematic diagram of an electronic device 600 proposed in one embodiment of the present application, including a memory 602, a processor 601, and a computer program stored in the memory and executable on the processor. When executed by the processor, the steps in the dual-site data storage method described in any of the above embodiments of the present application are implemented.

[0203] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0204] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0205] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0206] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0207] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0208] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0209] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0210] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0211] The above is a detailed introduction to the speech synthesis method, device, equipment and storage medium provided by this application. Specific examples are used in this article to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core ideas. At the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A speech synthesis method, characterized in that: The method comprises: receiving target text; Acquire corresponding text features from the target text, wherein the text features include phoneme information, word boundary information, and sentence information; Inputting the text features into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features; Performing multi-level representation fusion on the semantic features to obtain multi-level semantic features; Aligning the multi-level semantic features by words and durations, and aligning the multi-level semantic features by phonemes and durations through a multi-head attention mechanism to obtain the aligned multi-level semantic features; Obtaining the speech corresponding to the target text according to the aligned multi-level semantic features; The multi-level semantic features are subjected to multi-level representation fusion to obtain the multi-level semantic features, including: Extracting corresponding sentence features from the semantic features; The sentence features are fused with the phoneme features and word features included in the semantic features to obtain the multi-level semantic features.

2. The speech synthesis method according to claim 1, wherein: The step of obtaining corresponding text features from the target text, wherein the text features include phoneme information, word boundary information, and sentence information, includes: Recognizing the phonemes of each word in the target text to obtain the phoneme information; Identifying the number of phonemes included in each word in the target text to obtain word boundary information; Each sentence contained in the target text is identified to obtain the sentence information.

3. The speech synthesis method according to claim 1, wherein: The training steps of the semantic model include: Collecting training data, wherein the training data includes text data in at least one field; Classifying the training data according to sentence length to obtain classified training data; performing mask processing on the classified training data to obtain processed training data; Design corresponding symbol id for the training data; The semantic model is trained using the processed training data and the symbol ID to obtain the semantic model.

4. The speech synthesis method according to claim 1, wherein: The extracting corresponding sentence features from the semantic features includes: The semantic features are arranged in a sentence sequence, and adjacent sentences are regarded as a sentence pair to obtain at least one sentence pair; Extracting semantic features corresponding to the sentence pairs through a pre-trained cross-coding model to obtain corresponding sentence features; The sentence features are processed using a multi-head attention mechanism to obtain processed sentence features.

5. The speech synthesis method according to claim 1, wherein: The step of aligning the multi-level semantic features with words and durations, and aligning the multi-level semantic features with phonemes and durations through a multi-head attention mechanism to obtain the aligned multi-level semantic features includes: performing hard alignment on words and durations in the multi-level semantic features; Softly aligning the phonemes and durations in the multi-level semantic features through a multi-head attention mechanism; When the words and the phonemes are aligned with the duration, the semantic features are obtained.

6. The speech synthesis method according to claim 1, wherein: Obtaining the speech corresponding to the target text according to the aligned multi-level semantic features includes: Inputting the aligned multi-level semantic features into the decoder; Processing the multi-level semantic features by the decoder to obtain corresponding speech features; The speech features are input into a vocoder to obtain the speech.

7. A speech synthesis device, characterized in that: The device comprises: A text receiving module, used for receiving target text; A text feature extraction module is used to obtain corresponding text features from the target text, wherein the text features include phoneme information, word boundary information and sentence information; A semantic feature extraction module, configured to input the text features into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features; A multi-level semantic feature acquisition module is used to perform multi-level representation fusion on the semantic features to obtain multi-level semantic features; a speech feature acquisition module, configured to align words and durations of the multi-level semantic features, and to align phonemes and durations of the multi-level semantic features using a multi-head attention mechanism, thereby obtaining the aligned multi-level semantic features; A speech acquisition module, configured to obtain the speech corresponding to the target text according to the aligned multi-level semantic features; The multi-level semantic feature acquisition module includes: A sentence feature acquisition submodule is used to extract corresponding sentence features from the semantic features; The feature fusion submodule is used to fuse the sentence feature with the phoneme feature and word feature included in the semantic feature to obtain the multi-level semantic feature.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images