Speech synthesis method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of data processing, in particular to a speech synthesis method and device, equipment and a storage medium, and aims to improve the expressive force of speech synthesis. The method comprises the following steps: receiving a target text; obtaining a corresponding text feature from the target text, wherein the text feature comprises phoneme information, word boundary information and sentence information; inputting the text features into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features; performing multi-level representation fusion on the semantic features to obtain multi-level semantic features; performing word and duration alignment on the multi-level semantic features, and performing phoneme and duration alignment on the multi-level semantic features through a multi-head attention mechanism to obtain the aligned multi-level semantic features; and obtaining voice corresponding to the target text according to the aligned multi-level semantic features.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and more specifically, to a speech synthesis method, apparatus, device and storage medium. Background Art

[0002] Text-to-Speech (TTS) is a technology that converts text into speech. Speech synthesis plays an important role in interpersonal interaction and is a hotly researched technology. Existing speech synthesis technologies mainly extract information from input sentences, convert the text features of the sentences into speech representations, and then convert the speech representations into acoustic representations, thereby obtaining the corresponding acoustic waveforms to complete speech synthesis.

[0003] The speech synthesis systems in the prior art often ignore the rich representation of text and context, and the expressiveness of speech synthesis is insufficient. Summary of the invention

[0004] The embodiments of the present application provide a speech synthesis method, apparatus, device and storage medium, aiming to improve the expressiveness of speech synthesis.

[0005] A first aspect of an embodiment of the present application provides a speech synthesis method, the method comprising: receiving a target text; Acquire corresponding text features from the target text, wherein the text features include phoneme information, word boundary information, and sentence information; Inputting the text features into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features; Performing multi-level representation fusion on the semantic features to obtain multi-level semantic features; Aligning the multi-level semantic features by words and duration, and aligning the multi-level semantic features by phonemes and duration through a multi-head attention mechanism to obtain the aligned multi-level semantic features; According to the aligned multi-level semantic features, the speech corresponding to the target text is obtained.

[0006] Optionally, the acquiring corresponding text features from the target text, wherein the text features include phoneme information, word boundary information and sentence information, includes: Recognize the phonemes of each word in the target text to obtain the phoneme information; Identifying the number of phonemes included in each word in the target text to obtain the word boundary information; Each sentence contained in the target text is identified to obtain the sentence information.

[0007] Optionally, the step of training the semantic model includes: Collecting training data, wherein the training data includes text data in at least one field; Classifying the training data according to the length of the sentences to obtain classified training data; Performing mask processing on the classified training data to obtain processed training data; Designing a corresponding symbol id for the training data; The semantic model is trained using the processed training data and the symbol ID to obtain the semantic model.

[0008] Optionally, the performing multi-level representation fusion on the semantic features to obtain multi-level semantic features includes: Extracting corresponding sentence features from the semantic features; The sentence feature is fused with the phoneme feature and word feature included in the semantic feature to obtain the multi-level semantic feature.

[0009] Optionally, extracting corresponding sentence features from the semantic features includes: The semantic features are arranged in a sentence sequence, and adjacent sentences are regarded as a sentence pair to obtain at least one sentence pair; Extracting semantic features corresponding to the sentence pairs through a pre-trained cross-coding model to obtain corresponding sentence features; The sentence features are processed using a multi-head attention mechanism to obtain processed sentence features.

[0010] Optionally, aligning the multi-level semantic features by words and duration, and aligning the multi-level semantic features by phonemes and duration through a multi-head attention mechanism to obtain the aligned multi-level semantic features, comprises: Hard aligning words and durations in the multi-level semantic features; Softly aligning the phonemes and durations in the multi-level semantic features through a multi-head attention mechanism; When the word and the phoneme are aligned with the duration, the speech feature is obtained.

[0011] Optionally, obtaining the speech corresponding to the target text according to the aligned multi-level semantic features includes: Inputting the aligned multi-level semantic features into a decoder; Processing the multi-level semantic features through the decoder to obtain corresponding speech features; The speech features are input into a vocoder to obtain the speech.

[0012] A second aspect of an embodiment of the present application provides a speech synthesis device, the device comprising: A text receiving module, used for receiving a target text; A text feature extraction module, used to obtain corresponding text features from the target text, wherein the text features include phoneme information, word boundary information and sentence information; A semantic feature extraction module, used for inputting the text feature into a pre-trained semantic model to obtain a phoneme-level semantic feature corresponding to the text feature; A multi-level semantic feature acquisition module, used for performing multi-level representation fusion on the semantic features to obtain multi-level semantic features; A speech feature acquisition module, used to align the multi-level semantic features by words and duration, and to align the multi-level semantic features by phonemes and duration through a multi-head attention mechanism to obtain the multi-level semantic features after alignment; The speech acquisition module is used to obtain the speech corresponding to the target text according to the multi-level semantic features after the alignment process.

[0013] Optionally, the text feature extraction module includes: A phoneme information acquisition submodule, used to recognize the phonemes of each word in the target text to obtain the phoneme information; A word boundary information acquisition submodule is used to identify the number of phonemes included in each word in the target text to obtain the word boundary information; The sentence recognition submodule is used to recognize each sentence contained in the target text to obtain the sentence information.

[0014] Optionally, the step of training the semantic model includes: Collecting training data, wherein the training data includes text data in at least one field; Classifying the training data according to the length of the sentences to obtain classified training data; Performing mask processing on the classified training data to obtain processed training data; Designing a corresponding symbol id for the training data; The semantic model is trained using the processed training data and the symbol ID to obtain the semantic model.

[0015] Optionally, the multi-level semantic feature acquisition module includes: A sentence feature acquisition submodule, used to extract corresponding sentence features from the semantic features; The feature fusion submodule is used to fuse the sentence feature with the phoneme feature and word feature included in the semantic feature to obtain the multi-level semantic feature.

[0016] Optionally, the sentence feature acquisition submodule includes: A sentence pair acquisition submodule, used to obtain at least one sentence pair by taking the semantic features according to the sentence sequence and treating adjacent sentences as a sentence pair; A sentence feature extraction submodule, used to extract semantic features corresponding to the sentence pair through a pre-trained cross-coding model to obtain corresponding sentence features; The sentence feature processing submodule is used to process the sentence features using a multi-head attention mechanism to obtain processed sentence features.

[0017] Optionally, the speech feature acquisition module includes: An alignment submodule is used for hard aligning words and durations in the multi-level semantic features; A soft alignment submodule, used for softly aligning the phonemes and durations in the multi-level semantic features through a multi-head attention mechanism; The speech feature acquisition submodule is used to obtain the speech feature when the word and the phoneme are aligned with the duration.

[0018] Optionally, the voice acquisition module includes: A decoding submodule, used for inputting the aligned multi-level semantic features into a decoder; A voice code acquisition submodule, used for processing the multi-level semantic features through the decoder to obtain corresponding speech features; The audio acquisition submodule is used to input the speech features into a vocoder to obtain the speech.

[0019] A third aspect of an embodiment of the present application provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the method described in the first aspect of the present application are implemented.

[0020] A fourth aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect of the present application are implemented.

[0021] The speech synthesis method proposed in the present application is adopted to receive a target text; obtain corresponding text features from the target text, wherein the text features include phoneme information, word boundary information and sentence information; input the text features into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features; perform multi-level representation fusion on the semantic features to obtain multi-level semantic features; align the words and durations of the multi-level semantic features, and align the phonemes and durations of the multi-level semantic features through a multi-head attention mechanism to obtain the aligned multi-level semantic features; obtain the speech corresponding to the target text based on the aligned multi-level semantic features.

[0022] In this application, a pre-trained semantic model is used to process text features containing phoneme information, word boundary information, and sentence information, so that semantic features at the phoneme and word levels can be obtained, thereby improving the ability to capture fine-grained features in speech synthesis. The semantic features are then represented and fused at multiple levels to obtain multi-level semantic features, which enables the system to effectively capture the semantics of the context and improve the naturalness and expressiveness of the synthesized speech. The multi-level semantic features are then aligned with the phonemes and duration through a multi-head attention mechanism, which can improve the alignment accuracy, avoid noise caused by unstable phoneme boundaries, and regenerate the corresponding speech, effectively improving the expressiveness of speech synthesis from all aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 is a flow chart of a speech synthesis method proposed in one embodiment of the present application; Figure 2 is a flow chart of an end-to-end text-to-speech synthesis system with multi-level language and speech representation proposed in one embodiment of the present application; Figure 3 is a structural diagram of a speech synthesis model proposed in an embodiment of the present application; Figure 4 It is a schematic diagram of the speech synthesis model processing flow proposed in one embodiment of the present application; Figure 5 is a schematic diagram of a speech synthesis device proposed in an embodiment of the present application; Figure 6 It is a schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] refer to Figure 1 , Figure 1 is a flow chart of a speech synthesis method proposed in one embodiment of the present application. Figure 1 As shown, the method comprises the following steps: S11: Receive target text.

[0027] In this embodiment, the target text is the text that needs to be synthesized into speech.

[0028] Text-to-Speech (TTS), also known as text-to-speech conversion, is a technology that converts text into speech. Since speech synthesis is the last link in human-computer interaction, it converts the text processed in the previous link into the target audio and feeds it back to the user. Therefore, the purpose of speech synthesis is to enable the machine to make natural and pleasant sounds, so as to solve the dilemma of human-computer interaction. Therefore, speech synthesis has received widespread attention from researchers and technicians. After a long period of development, speech synthesis technology based on end-to-end speech synthesis has become the mainstream technology for speech synthesis research and application. The goal of speech synthesis is to generate audio with intelligibility and naturalness. Among them, the quality of naturalness depends largely on the expressiveness of the sound.

[0029] The speech synthesis system is mainly composed of front-end analysis, acoustic model and vocoder; the process is to convert the input text into language representation through front-end analysis, and then convert these language representations into intermediate acoustic representations such as Mel spectrum through acoustic model, and finally, convert the acoustic representation into waveform through vocoder. From this, it can be intuitively seen that the burden of acoustic model is greater than that of vocoder, because vocoder only needs to restore the phase information of given spectrogram, while acoustic model needs to narrow the gap between text and spectrogram. The information gap between text and spectrogram is large, which makes the task of acoustic model more difficult than vocoder; moreover, most of the existing expressive speech synthesis is still through the modeling, decoupling, control and migration of the rhythm and emotional style of reference audio, while ignoring the rich representation contained in the text; In addition, most speech synthesis systems only consider the information in the current sentence and ignore the contextual text information. As the requirements for naturalness and expressiveness of speech synthesis systems continue to increase, these speech synthesis systems lack the ability to produce paragraph-level speech with sufficient naturalness and expressiveness. This is because human speech is contextually coherent in a paragraph, rather than each sentence being independent and not dependent on the context.

[0030] In this embodiment, when performing speech synthesis, a speech synthesis model is used for speech synthesis. The speech synthesis model is a model composed of replacing the speech encoding with a pre-trained semantic model (PhonemeBERT) based on FastPitch and combining it with a multi-head attention mechanism. The model first needs to receive the target text.

[0031] For example, the target text may be a text in any language, such as Chinese, English, etc., which includes characters, words, sentences, etc.

[0032] S12: Acquire corresponding text features from the target text, wherein the text features include phoneme information, word boundary information, and sentence information.

[0033] In this embodiment, phoneme information, a phoneme is the smallest unit that makes up a pronunciation, a word boundary is the number of phonemes contained in a word, and the position of the first and last phonemes of a word can be determined based on the number of phonemes, and sentence information is the number of sentences contained in a text and other information.

[0034] In this embodiment, the target text is analyzed by a front-end analysis module to obtain corresponding text features.

[0035] In this embodiment, the specific steps of obtaining corresponding text features from the target text, wherein the text features include phoneme information, word boundary information, and sentence information, include: S12-1: Recognize the phonemes of each word in the target text to obtain the phoneme information.

[0036] In this embodiment, the phoneme information includes information such as all the phonemes appearing in the target text and the number of each phoneme.

[0037] In this embodiment, the phonemes of each word in the target text are firstly recognized, and then the corresponding phoneme information is obtained.

[0038] S12-2: Identify the number of phonemes included in each word in the target text to obtain the word boundary information.

[0039] In this embodiment, the number of phonemes included in each word in the text is identified by the front-end analysis module. By determining the number of phonemes included in each word, the first phoneme and the last phoneme of each word can be determined, and then the word boundary information can be obtained.

[0040] For example, the target text contains the word "I am", which includes the phonemes "w, o, s, h, i".

[0041] S12-3: Recognize each sentence contained in the target text to obtain the sentence information.

[0042] In this embodiment, each sentence contained in the target text is identified by the front-end analysis module to obtain sentence information.

[0043] S13: Inputting the text feature into a pre-trained semantic model to obtain a phoneme-level semantic feature corresponding to the text feature.

[0044] In this embodiment, the pre-trained semantic model is the PhonemeBERT model. PhonemeBERT is a pre-trained model based on the BERT architecture, which aims to capture rich contextual information through phoneme-level modeling and generate fine-grained phoneme representations to improve the sound quality and expressiveness of the speech synthesis system.

[0045] In this embodiment, pre-training is performed based on the BERT model to model semantic representations at the word and phoneme levels and enhance the generalization ability of the system. This method ensures that the model can achieve good synthesis effects even under low-resource data conditions and improves the accuracy of speech generation.

[0046] In this embodiment, the phoneme encoder in FastPitch is replaced with the pre-trained PhonemeBERT model to utilize the fine-grained phoneme representation it captures. Combined with word boundary information, word-level semantic representation can be obtained, thereby enhancing the speech synthesis model's ability to model speech details and making the synthesized speech more expressive.

[0047] In this embodiment, text features are input into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features. The semantic model extracts phoneme-level semantic features from the text features in combination with word boundary information.

[0048] In this embodiment, phoneme-level and word-level semantic representations are introduced, and the phoneme encoder in FastPitch is replaced by PhonemeBERT to improve the fine-grained feature capture capability of speech synthesis.

[0049] S14: Perform multi-level representation fusion on the semantic features to obtain multi-level semantic features.

[0050] In this embodiment, the multi-level semantic features include phoneme-level features, word-level features, and sentence-level features.

[0051] In this embodiment, multi-level contexts are taken into account simultaneously, the factors and words of the previous sentence and the current sentence are connected, and the coarse-grained sentence-level semantic representation of adjacent sentences is extracted through a pre-trained cross-utterance encoder, thereby capturing the semantic correlation between context sentences.

[0052] In this embodiment, the specific steps of performing multi-level representation fusion on semantic features to obtain multi-level semantic features include: S14-1: Extracting corresponding sentence features from the semantic features.

[0053] In this embodiment, firstly, the corresponding sentence features are extracted from the semantic features. The specific steps include: S14-1-1: Array the semantic features according to the sentence sequence, and treat adjacent sentences as a sentence pair to obtain at least one sentence pair.

[0054] In this embodiment, each semantic feature has a corresponding sentence feature, and the received semantic features are arranged in a sentence sequence, and adjacent sentences are taken as a sentence pair to obtain at least one sentence pair.

[0055] For example, the adjacent sentence sequence Divide into 2L pairs of sentences, divide the adjacent sentence sequence into 2L pairs of sentences, where represents the current sentence, and L is the number of sentences considered in the previous and next contexts.

[0056] S14-1-2: Extract the semantic features corresponding to the sentence pair through a pre-trained cross-coding model to obtain corresponding sentence features.

[0057] In this embodiment, the cross-coding model is a BERT model, and the semantic features corresponding to the sentence pairs are extracted through the pre-trained cross-coding model to obtain corresponding sentence features.

[0058] S14-1-3: Use a multi-head attention mechanism to process the sentence features to obtain processed sentence features.

[0059] In this embodiment, the head attention mechanism is an improved attention mechanism that aims to improve the expressiveness and learning ability of the model by splitting the input features into multiple "heads" and processing each head independently. In order to introduce sentence-level information into the phoneme-level representation, sentence-level feature extraction is performed through BERT. At this stage, the model aligns the global representation at the sentence level with the local representation at the phoneme level. Specifically: the phoneme-level representation is used as the query; the sentence-level representation is used as the key. The multi-head attention mechanism will calculate the correlation between each phoneme and the sentence-level information, and combine the global context information to enhance the characteristics of the phoneme, so that the processed sentence features contain more factor features.

[0060] In this embodiment, after obtaining the sentence features, the sentence features are input into the multi-head attention module of the speech synthesis model, and the sentence features are processed through the multi-head attention mechanism to obtain the processed sentence features.

[0061] S14-2: Fusing the sentence feature with the phoneme feature and word feature included in the semantic feature to obtain the multi-level semantic feature.

[0062] In this embodiment, after the processed sentence features are obtained, the obtained sentence features are fused with the phoneme features and the word features to obtain multi-level semantic features.

[0063] In this embodiment, pre-training is performed based on the BERT model to model semantic representations at the word and phoneme levels and enhance the generalization ability of the system. This method ensures that the model can achieve good synthesis effects even under low-resource data conditions and improves the accuracy of speech generation.

[0064] In this embodiment, based on the BERT pre-training model, the masked language model (MLM) strategy is used to enhance the generalization ability of the system under limited resource conditions and ensure the stability of the synthesis effect. The cross-utterance encoder extracts sentence-level coarse-grained semantic information, enabling the system to effectively capture contextual semantics and improve the naturalness and expressiveness of the synthesized speech.

[0065] S15: aligning the words and duration of the multi-level semantic features, and aligning the phonemes and duration of the multi-level semantic features through a multi-head attention mechanism to obtain the aligned multi-level semantic features.

[0066] In this embodiment, when synthesizing speech, it is necessary to align each word and each phoneme with the duration, so as to obtain the multi-level semantic features after the alignment process.

[0067] In this embodiment, the multi-level semantic features are processed by a multi-head attention mechanism, and the words and phonemes therein are aligned with their respective durations, so as to obtain the multi-level semantic features after alignment. The specific steps include: S15-1: Hard aligning the words and durations in the multi-level semantic features.

[0068] In this embodiment, hard alignment is performed on the words and durations in the multi-level semantic features. Hard alignment is to set the duration of each word in the speech to the same time.

[0069] For example, the duration of the word "eat" included in the multi-level semantic features is hard aligned to 1 second, and the duration of this word appears in the synthesized speech for 1 second each time.

[0070] S15-2: Through a multi-head attention mechanism, the phonemes and durations in the multi-level semantic features are soft-aligned.

[0071] In this embodiment, the phonemes and durations in the multi-level semantic features are soft-aligned through a multi-head attention mechanism. Soft alignment is to flexibly set the pronunciation time of each phoneme according to the position and frequency of each phoneme in different words.

[0072] In this embodiment, for example, a phoneme "w" in the word "we" has its duration aligned to 0.1 seconds, and in the word "look at me", has its duration aligned to 0.2 seconds.

[0073] S15-3: When the word and the phoneme are aligned with the duration, the speech feature is obtained.

[0074] In this embodiment, when the words, phonemes and durations are aligned, the multi-level semantic features after the alignment process are obtained.

[0075] In this embodiment, in view of the uncertainty of phoneme boundaries, traditional hard alignment methods may introduce errors. Therefore, this model adopts word-level hard alignment and combines multi-head attention mechanism to achieve soft alignment of phonemes to duration. This flexible alignment strategy can effectively reduce alignment errors and ensure the accuracy of phoneme duration processing, thereby optimizing the quality of decoder input features. Word-level hard alignment combined with multi-head attention mechanism is used to achieve soft alignment of phonemes to duration, improve alignment accuracy, and avoid noise caused by uncertainty of phoneme boundaries.

[0076] S16: Obtain the speech corresponding to the target text according to the multi-level semantic features after the alignment process.

[0077] In this embodiment, after the speech features are obtained, the speech corresponding to the target text is obtained according to the processed multi-level semantic features. The specific steps are: S16-1: inputting the aligned multi-level semantic features into the decoder.

[0078] In this embodiment, the decoder is used to convert semantics, converting semantic features into speech features.

[0079] In this embodiment, the aligned multi-level semantic features are firstly input into the decoder.

[0080] S16-2: Processing the multi-level semantic features through the decoder to obtain corresponding speech features.

[0081] In this embodiment, the processed multi-level semantic features are decoded by a decoder to obtain corresponding speech features.

[0082] S16-3: Input the speech features into a vocoder to obtain the speech.

[0083] In this embodiment, the obtained speech features are input into a corresponding vocoder for speech synthesis to obtain corresponding speech.

[0084] In this embodiment, after obtaining the multi-level fusion features, it is necessary to predict the pitch of each phoneme through a pitch predictor, and predict the duration of each word and each phoneme through a duration predictor, and align the durations in combination with a multi-head attention mechanism.

[0085] In this embodiment, through multi-level semantic representation, the synthesized speech is significantly enhanced in emotional expression and naturalness, and adapts to more complex language scenarios. The soft alignment strategy effectively reduces the alignment error, ensuring that each word is clearly distinguishable, especially when dealing with repeated words and complex sentences. The optimized system has a higher processing speed, meets the low latency requirements of the streaming TTS system, and is suitable for real-time applications. The system can be flexibly adjusted according to different text types, improves the application effect in a variety of scenarios, and enhances the practicality of speech synthesis technology.

[0086] In this embodiment, after obtaining the multi-level fusion features, it is necessary to predict the pitch of each phoneme through a pitch predictor, and predict the duration of each word and each phoneme through a duration predictor, and align the durations in combination with a multi-head attention mechanism.

[0087] In another embodiment of the present application, the step of training the semantic model includes: S21: Collect training data, where the training data includes text data in at least one field.

[0088] In this embodiment, training data is first collected, and the collected training data includes text data in at least one field.

[0089] For example, 10G of open source text data is collected from multiple fields such as dialogue, popular science, and news, and then cleaned and deduplicated to ensure the quality of training data.

[0090] S22: Classifying the training data according to the length of the sentences to obtain classified training data.

[0091] In this embodiment, batch sampling optimization is performed on the data, and the collected training data is classified according to the length of the sentences to obtain classified training data.

[0092] In this embodiment, since the training data comes from a wide range of sources and the text lengths vary greatly, a batch sampling strategy is used to classify data of different lengths, thereby improving the model training efficiency and avoiding unnecessary computational overhead.

[0093] S23: performing mask processing on the classified training data to obtain processed training data.

[0094] In this embodiment, masking is to hide some data, so as to achieve the purpose of training the model.

[0095] In this embodiment, mask processing is performed on the classified training data to obtain processed training data. Generally, mask processing is performed on a part of the data.

[0096] For example, a masked language model (MLM) is used to enhance the generalization ability of the model in low-resource scenarios. The specific strategy is: 30% of the input will be masked, 80% of the phonemes in the masked part will be masked, 10% will be replaced with other phonemes, and the remaining 10% will remain unchanged.

[0097] S24: Design corresponding symbol ids for the training data.

[0098] In this embodiment, the symbol id is a pre-set id corresponding to each phoneme, word, and sentence.

[0099] For example, corresponding symbol ids are designed for the training data, for example, the symbol id of "a" is 1, the symbol id of "o" is 2, and so on.

[0100] S25: Training the semantic model using the processed training data and the symbol ID to obtain the semantic model.

[0101] In this embodiment, the semantic model is iteratively trained using the processed training data and symbol ID. In each round of training, the training data and symbol ID are input into the semantic model, and the semantic model automatically extracts the features included in the training data to obtain the corresponding semantic features. The parameters of the model are then adjusted according to the actual data until the model training is completed to obtain a trained semantic model.

[0102] In this embodiment, in order to adapt to streaming application scenarios, the model structure of PhonemeBERT can be streamlined and lightweight, reducing computing resource usage, ensuring high efficiency and scalability in real-time applications, and being suitable for streaming TTS scenarios. For example, the number of layers of the model is reduced from 6 to 3, and the model is streamlined without affecting model performance.

[0103] In another embodiment of the present application, the speech synthesis model is also obtained through pre-training. During the training, the collected training materials are used to train the speech synthesis model to obtain a trained speech synthesis model. In the final optimization effect evaluation, different types of texts are tested. The results show that the system not only has significant improvements in expressiveness and naturalness, but also significantly speeds up the processing speed, meeting the low latency requirements of the streaming TTS system. Especially in texts containing repeated words and complex sentence structures, the system can clearly read each word, avoiding the problem of ambiguity in repeated words.

[0104] refer to Figure 2 , Figure 2 is a flowchart of an end-to-end text-to-speech synthesis system with multi-level language and speech representation proposed in an embodiment of the present application, such as Figure 2 As shown in the figure, the synthesized text should first be processed by the front-end, that is, text regularization to convert it from non-standardized spoken text to standardized written language, and then converted into pinyin through g2p and processed into phonemes to obtain the features required by the model, and then the audio content corresponding to the input text is obtained through the back-end model.

[0105] refer to Figure 3 , Figure 3 is a diagram of the speech synthesis model structure proposed in an embodiment of the present application, such as Figure 3 As shown, it includes a text encoder and a sentence-level encoder. The encoder is the PhonemeBERT model, which generates corresponding semantic features. It also includes a multi-head attention mechanism module. After being processed by the multi-head attention module, multi-level fusion features are obtained. The pitch predictor is used to predict the pitch of each phoneme, and the duration predictor is used to predict the duration of each phoneme. The decoder is used to decode the speech features to obtain the corresponding voice code, and the voice coder is used to synthesize the corresponding audio according to the voice code.

[0106] refer to Figure 4, Figure 4 : is a schematic diagram of a speech synthesis model processing flow proposed in an embodiment of the present application. Figure 4 As shown in the figure, based on the FastPitch model, the text encoder is replaced with PhonemeBERT to extract phoneme-level semantic representation. According to the word boundary information, the word-level semantic representation is obtained by upsampling, and the phoneme-level and word-level information are combined. The phoneme-level and word-level information of two adjacent sentences are aligned through the attention mechanism to extract coarse-grained sentence-level semantic information. For paired sentences, sentence features are extracted by the pre-trained cross encoder, and then processed by the multi-head attention mechanism to obtain the processed sentence features, which are then fused with the phoneme features and word features. Finally, the multi-head attention mechanism is used for time alignment, and pitch prediction is performed to obtain the corresponding Mel spectrum. The phoneme-level and word-level information of two adjacent sentences are aligned through the attention mechanism to extract coarse-grained sentence-level semantic information. This multi-level representation method makes the system more natural when processing long sentences, repeated words, and sentences with obvious emotional changes, and the expression of speech synthesis is also stronger. Especially when processing repeated words, the system can effectively avoid ambiguity, and each word is clearly distinguishable. In order to ensure a more accurate alignment process from phonemes to speech features, this embodiment introduces a word-level hard alignment method and combines the multi-head attention mechanism to perform soft alignment on the phoneme duration. In practical applications, this alignment strategy effectively avoids noise and errors caused by uncertainty in phoneme boundaries, thereby improving the clarity and fluency of synthesized speech.

[0107] Based on the same inventive concept, an embodiment of the present application provides a speech synthesis device. Figure 5 , Figure 5 FIG. 5 is a schematic diagram of a speech synthesis device 500 proposed in an embodiment of the present application. Figure 5 As shown, the device comprises: A text receiving module 501 is used to receive a target text; A text feature extraction module 502 is used to obtain corresponding text features from the target text, wherein the text features include phoneme information, word boundary information and sentence information; A semantic feature extraction module 503 is used to input the text feature into a pre-trained semantic model to obtain a phoneme-level semantic feature corresponding to the text feature; A multi-level semantic feature acquisition module 504 is used to perform multi-level representation fusion on the semantic features to obtain multi-level semantic features; The speech feature acquisition module 505 is used to align the multi-level semantic features by words and duration, and to align the multi-level semantic features by phonemes and duration through a multi-head attention mechanism to obtain the multi-level semantic features after alignment; The speech acquisition module 506 is used to obtain the speech corresponding to the target text according to the multi-level semantic features after the alignment process.

[0108] Optionally, the text feature extraction module includes: A phoneme information acquisition submodule, used to recognize the phonemes of each word in the target text to obtain the phoneme information; A word boundary information acquisition submodule is used to identify the number of phonemes included in each word in the target text to obtain the word boundary information; The sentence recognition submodule is used to recognize each sentence contained in the target text to obtain the sentence information.

[0109] Optionally, the step of training the semantic model includes: Collecting training data, wherein the training data includes text data in at least one field; Classifying the training data according to the length of the sentences to obtain classified training data; Performing mask processing on the classified training data to obtain processed training data; Designing a corresponding symbol id for the training data; The semantic model is trained using the processed training data and the symbol ID to obtain the semantic model.

[0110] Optionally, the multi-level semantic feature acquisition module includes: A sentence feature acquisition submodule, used to extract corresponding sentence features from the semantic features; The feature fusion submodule is used to fuse the sentence feature with the phoneme feature and word feature included in the semantic feature to obtain the multi-level semantic feature.

[0111] Optionally, the sentence feature acquisition submodule includes: A sentence pair acquisition submodule, used to obtain at least one sentence pair by taking the semantic features according to the sentence sequence and treating adjacent sentences as a sentence pair; A sentence feature extraction submodule, used to extract semantic features corresponding to the sentence pair through a pre-trained cross-coding model to obtain corresponding sentence features; The sentence feature processing submodule is used to process the sentence features using a multi-head attention mechanism to obtain processed sentence features.

[0112] Optionally, the speech feature acquisition module includes: An alignment submodule is used for hard aligning words and durations in the multi-level semantic features; A soft alignment submodule, used for softly aligning the phonemes and durations in the multi-level semantic features through a multi-head attention mechanism; The speech feature acquisition submodule is used to obtain the speech feature when the word and the phoneme are aligned with the duration.

[0113] Optionally, the voice acquisition module includes: A decoding submodule, used for inputting the aligned multi-level semantic features into a decoder; A voice code acquisition submodule, used for processing the multi-level semantic features through the decoder to obtain corresponding speech features; The audio acquisition submodule is used to input the speech features into a vocoder to obtain the speech.

[0114] Based on the same inventive concept, another embodiment of the present application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method described in any of the above embodiments of the present application.

[0115] Based on the same inventive concept, another embodiment of the present application provides an electronic device, Figure 6 It is a schematic diagram of an electronic device 600 proposed in one embodiment of the present application, including a memory 602, a processor 601 and a computer program stored in the memory and executable on the processor, wherein when the processor is executed, the steps in the dual-site data storage method described in any of the above embodiments of the present application are implemented.

[0116] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0117] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0118] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0119] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0122] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0123] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0124] The speech synthesis method, apparatus, device and storage medium provided by the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A speech synthesis method, characterized in that: The method comprises: receiving a target text; Acquire corresponding text features from the target text, wherein the text features include phoneme information, word boundary information, and sentence information; Inputting the text features into a pre-trained semantic model to obtain phoneme-level semantic features corresponding to the text features; Performing multi-level representation fusion on the semantic features to obtain multi-level semantic features; Aligning the multi-level semantic features by words and duration, and aligning the multi-level semantic features by phonemes and duration through a multi-head attention mechanism to obtain the aligned multi-level semantic features; According to the aligned multi-level semantic features, the speech corresponding to the target text is obtained.

2. The speech synthesis method according to claim 1, characterized in that: The step of obtaining corresponding text features from the target text, wherein the text features include phoneme information, word boundary information, and sentence information, includes: Recognize the phonemes of each word in the target text to obtain the phoneme information; Identifying the number of phonemes included in each word in the target text to obtain the word boundary information; Each sentence contained in the target text is identified to obtain the sentence information.

3. The speech synthesis method according to claim 1, characterized in that: The training steps of the semantic model include: Collecting training data, wherein the training data includes text data in at least one field; Classifying the training data according to the length of the sentences to obtain classified training data; Performing mask processing on the classified training data to obtain processed training data; Designing a corresponding symbol id for the training data; The semantic model is trained using the processed training data and the symbol ID to obtain the semantic model.

4. The speech synthesis method according to claim 1, characterized in that: The multi-level representation fusion of the semantic features to obtain the multi-level semantic features includes: Extracting corresponding sentence features from the semantic features; The sentence feature is fused with the phoneme feature and word feature included in the semantic feature to obtain the multi-level semantic feature.

5. The speech synthesis method according to claim 4, characterized in that: The extracting corresponding sentence features from the semantic features includes: The semantic features are arranged in a sentence sequence, and adjacent sentences are regarded as a sentence pair to obtain at least one sentence pair; Extracting semantic features corresponding to the sentence pairs through a pre-trained cross-coding model to obtain corresponding sentence features; The sentence features are processed using a multi-head attention mechanism to obtain processed sentence features.

6. The speech synthesis method according to claim 1, characterized in that: The step of aligning the words and duration of the multi-level semantic features, and aligning the phonemes and duration of the multi-level semantic features through a multi-head attention mechanism to obtain the aligned multi-level semantic features includes: Hard aligning words and durations in the multi-level semantic features; Softly aligning the phonemes and durations in the multi-level semantic features through a multi-head attention mechanism; When the word and the phoneme are aligned with the duration, the speech feature is obtained.

7. The speech synthesis method according to claim 1, characterized in that: The step of obtaining the speech corresponding to the target text according to the aligned multi-level semantic features includes: Inputting the aligned multi-level semantic features into a decoder; Processing the multi-level semantic features through the decoder to obtain corresponding speech features; The speech features are input into a vocoder to obtain the speech.

8. A speech synthesis device, characterized in that: The device comprises: A text receiving module, used for receiving a target text; A text feature extraction module, used to obtain corresponding text features from the target text, wherein the text features include phoneme information, word boundary information and sentence information; A semantic feature extraction module, used for inputting the text feature into a pre-trained semantic model to obtain a phoneme-level semantic feature corresponding to the text feature; A multi-level semantic feature acquisition module, used for performing multi-level representation fusion on the semantic features to obtain multi-level semantic features; A speech feature acquisition module, used to align the multi-level semantic features by words and duration, and to align the multi-level semantic features by phonemes and duration through a multi-head attention mechanism to obtain the multi-level semantic features after alignment; The speech acquisition module is used to obtain the speech corresponding to the target text according to the multi-level semantic features after the alignment process.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images