Speech synthesis methods, devices, readable media and electronic devices

Abstract

This disclosure relates to a speech synthesis method, apparatus, readable medium, and electronic device. The method includes: determining tone marking information, prosodic marking information, and focal stress marking information of a target text to be processed, wherein the focal stress marking information includes syllable stress information of the target text, the syllable stress information indicating whether each syllable in the target text should be stressed; determining the phoneme sequence of the target text; and generating synthesized audio corresponding to the target text based on the tone marking information, prosodic marking information, focal stress marking information, and phoneme sequence of the target text. Therefore, the position and performance of focal stress can be directly controlled during speech synthesis, improving the controllability of stress in speech synthesis, thereby improving the problems of indistinct and uncontrollable stress performance, and enhancing the intonation variation and naturalness of the synthesized result.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a speech synthesis method, apparatus, readable medium, and electronic device. Background Technology

[0002] Speech synthesis technology can convert any text into corresponding audio. It typically involves two parts: analyzing the text to obtain linguistic information and generating a sound waveform based on the analysis results. The quality of the synthesized speech is usually reflected in the intonation and naturalness of the synthesized speech, with the performance of stress significantly affecting these aspects. Current technologies often rely on neural networks to learn stress acoustic features, but this approach still suffers from uncontrollable stress representation and unclear stress acoustics. Furthermore, these problems are difficult to solve through model optimization alone. Therefore, current speech synthesis technology still suffers from synthesized speech that lacks naturalness and expressiveness. Summary of the Invention

[0003] This section is provided to briefly introduce the concepts, which will be described in detail in the Detailed Description section later. This section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, this disclosure provides a speech synthesis method, the method comprising:

[0005] The tone marking information, prosody marking information, and focal stress marking information of the target text to be processed are determined, wherein the focal stress marking information includes the syllable stress information of the target text, and the syllable stress information is used to indicate whether each syllable in the target text should be stressed;

[0006] Determine the phoneme sequence of the target text;

[0007] Based on the tone annotation information, prosody annotation information, focal stress annotation information, and phoneme sequence of the target text, a synthesized audio corresponding to the target text is generated.

[0008] Secondly, this disclosure provides a speech synthesis apparatus, the apparatus comprising:

[0009] The first determining module is used to determine the tone marking information, prosody marking information and focal stress marking information of the target text to be processed, wherein the focal stress marking information includes the syllable stress information of the target text, and the syllable stress information is used to indicate whether each syllable in the target text should be stressed;

[0010] The second determining module is used to determine the phoneme sequence of the target text;

[0011] The generation module is used to generate a synthesized audio corresponding to the target text based on the tone annotation information, prosody annotation information, focal stress annotation information and phoneme sequence of the target text.

[0012] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect of this disclosure.

[0013] Fourthly, this disclosure provides an electronic device, comprising:

[0014] A storage device on which one or more computer programs are stored;

[0015] One or more processing means are configured to execute the one or more computer programs in the storage device to implement the steps of the method described in the first aspect of this disclosure.

[0016] The above technical solution determines the tone, prosody, and focal stress annotation information of the target text, as well as its phoneme sequence. Based on these annotations, a synthesized audio corresponding to the target text is generated. The focal stress annotation information includes syllable stress information, indicating whether each syllable should be stressed. Therefore, in speech synthesis of target text, in addition to using tone and prosody features, focal stress features are incorporated. This allows for direct control over the position and expression of focal stress during speech synthesis, improving the controllability of stress and addressing issues such as indistinct or uncontrollable stress, thus enhancing the intonation variation and naturalness of the synthesized result. Furthermore, since the focal stress features indicate whether each syllable in the target text should be stressed (i.e., there are only two types, yes and no), the complexity of focal stress annotation is effectively reduced, facilitating the expansion of annotation scale.

[0017] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0019] Figure 1 This is a flowchart of a speech synthesis method provided according to one embodiment of the present disclosure;

[0020] Figure 2 A partial schematic diagram of an exemplary annotation interface provided in this disclosure is shown;

[0021] Figure 3 This is an exemplary flowchart of the step of generating synthesized audio corresponding to target text in the speech synthesis method provided in this disclosure;

[0022] Figure 4 This is a block diagram of a speech synthesis apparatus provided according to one embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0024] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0025] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0026] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0027] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0028] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0029] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0030] As described in the background section, related technologies suffer from synthesized speech that lacks naturalness and expressiveness. Taking Mandarin Chinese as an example, the current C-ToBI (ToBI stands for Tones and Break Indices) annotation method divides stress into four levels, 1 to 4, corresponding to stress in four prosodic units: prosodic words, prosodic phrases, intonation phrases, and sentences, with each prosodic unit labeled with one stress. This method has two main problems: firstly, the annotation content is complex and cumbersome, unsuitable for large-scale application; secondly, its stress annotation relies primarily on the prosodic level division rather than the intensity of the stress itself (for example, although some parts of different prosodic words are labeled with stress, the actual stress levels of these stresses are not necessarily equal), introducing interference information and making statistical modeling of stress difficult. Therefore, current speech synthesis technology still has problems with stress controllability and stress representation.

[0031] To address the aforementioned technical problems, this disclosure provides a speech synthesis method, apparatus, readable medium, and electronic device, which improves the issues of indistinct and uncontrollable stress expression, and enhances the intonation variation and naturalness of the synthesized results.

[0032] Figure 1 This is a flowchart of a speech synthesis method provided according to one embodiment of the present disclosure, such as... Figure 1 As shown, the method provided in this disclosure may include steps 11 to 13.

[0033] In step 11, the tone annotation information, prosody annotation information, and focus accent annotation information of the target text to be processed are determined.

[0034] In this disclosure, tone annotation information is used to reflect content related to tone. The tone annotation information of the target text can be the tone type of each character in the target text. Among them, tone type can include, but is not limited to, the first tone (also known as high level tone, level tone), the second tone (also known as rising tone, rising tone), the third tone (also known as falling tone, rising tone), the fourth tone (also known as falling tone, falling tone), the neutral tone, and the third tone that needs to be tone-changed.

[0035] Prosodic annotation information is used to reflect content related to prosody, and may include, but is not limited to, prosodic boundary information. The prosodic boundary (break index, abbreviated as BRK), also known as the break index, is used to describe the organization and sentence segmentation of information in the speech flow. Optionally, prosodic boundary information may include, but is not limited to, sentence boundaries, intonation phrase boundaries, prosodic phrase boundaries, and prosodic word boundaries.

[0036] Focused stress is used to identify content in a text that needs to be emphasized or highlighted. It is usually characterized by acoustic prominence, such as higher pitch, longer duration, or stronger intensity. For example, if a text contains multiple prosodic words, and the highest pitch of one prosodic word is higher than the highest pitches of the preceding and following prosodic words, or if the duration of a prosodic word (especially the duration of the first or last syllable) is prolonged, then that prosodic word can be identified as the focused stress of the text.

[0037] Focus stress annotation information is used to reflect content related to focus stress, and it can include syllable stress information of the target text. Syllable stress information is used to indicate whether each syllable (i.e., the syllable corresponding to each word) in the target text should be stressed. Syllables that should be stressed can be marked with a first identifier, and syllables that do not need to be stressed can be marked with a second identifier. For example, if the target text consists of the five characters {d1d2d3d4d5}, and the syllables of characters d3 and d4 should be stressed, and the first identifier is 1 and the second identifier is 0, then the syllable stress information of the target text can be marked as {00110}.

[0038] In step 12, the phoneme sequence of the target text is determined.

[0039] In this disclosure, the phoneme sequence corresponding to the text to be synthesized can be obtained through a grapheme-to-phoneme (G2P) model.

[0040] For example, the G2P model can use recurrent neural networks (RNNs) and long short-term memory (LSTMs) to achieve the conversion from glyphs to phonemes.

[0041] In step 13, a synthesized audio corresponding to the target text is generated based on the tone annotation information, prosody annotation information, focal stress annotation information and phoneme sequence of the target text.

[0042] The above technical solution determines the tone, prosody, and focal stress annotation information of the target text, as well as its phoneme sequence. Based on these annotations, a synthesized audio corresponding to the target text is generated. The focal stress annotation information includes syllable stress information, indicating whether each syllable should be stressed. Therefore, in speech synthesis of target text, in addition to using tone and prosody features, focal stress features are incorporated. This allows for direct control over the position and expression of focal stress during speech synthesis, improving the controllability of stress and addressing issues such as indistinct or uncontrollable stress, thus enhancing the intonation variation and naturalness of the synthesized result. Furthermore, since the focal stress features indicate whether each syllable in the target text should be stressed (i.e., there are only two types, yes and no), the complexity of focal stress annotation is effectively reduced, facilitating the expansion of annotation scale.

[0043] To enable those skilled in the art to better understand the speech synthesis method provided in this disclosure, the above steps are illustrated in detail below.

[0044] First, the relevant content of the focus accent marking information used in this disclosure will be explained.

[0045] As mentioned above, the focus stress annotation information of the target text can include syllable stress information used to indicate whether each word in the target text should be stressed. The following is a detailed explanation of how the focus stress annotation information of the target text is determined.

[0046] In one possible implementation, the focus accent information of the target text can be determined through manual annotation. That is, the focus accent annotation operation for the target text can be received, and the focus accent annotation information of the target text can be generated based on the focus accent annotation operation.

[0047] In other words, annotators can directly annotate the focal stresses in the target text, transferring the focal stress features they expect to hear from the synthesized audio to the target text. This annotation process for focal stresses can be performed in three steps.

[0048] The first step can be to annotate each prosodic word in the target text, that is, to mark whether each prosodic word in the target text should be stressed. The information annotated in this step is the prosodic word stress information of the target text. For example, whether a prosodic word is stressed or not can be indicated by 0 or 1.

[0049] The second step involves marking the prosodic words that were marked as needing emphasis in the first step (hereinafter referred to as target prosodic words) as the target words. Specifically, it involves marking whether each segment of the target prosodic word should be emphasized. The information marked in this step is the segmentation emphasis information of the target text. For example, whether a segment in the target prosodic word is emphasized can be indicated by 0 or 1. This segmentation of the target text can be achieved using existing automatic word segmentation technology.

[0050] The third step involves marking the words that were marked as needing stress in the second step (hereinafter referred to as target words) as the target words. That is, marking whether each syllable (i.e., each character) in the target words should be stressed. The information marked in this step is the syllable stress information of the target text.

[0051] Furthermore, to improve the consistency of annotation for focal stress, the third step of annotating syllable stress information can also be based on preset intra-word stress rules. These preset intra-word stress rules indicate the distribution pattern of stressed syllables within a stressed word segment, and can be determined based on perspectives on word stress in the relevant field. For example, most opinions in the relevant field suggest that within a stressed word, there are two patterns: either all syllables are stressed, or only the first syllable is stressed. Accordingly, intra-word stress rules can be preset to either stress all syllables within a stressed word or stress only the first syllable.

[0052] In another possible implementation, the focus accent information of the target text can be determined using a pre-trained focus accent annotation model. Accordingly, the focus accent information of the target text can be obtained in the following way:

[0053] Input the target text into the focus accent annotation model and obtain the focus accent annotation information output by the focus accent annotation model.

[0054] The focus accent annotation model is trained on a second training text with focus accent annotation information.

[0055] The second training text can be text extracted from real-world speech. For such speech, annotators can mark appropriate locations in the text by listening to the audio to obtain the focus stress annotation information of the second training text. This annotation mainly relies on the annotator's auditory perception, and the annotation process can refer to the following aspects (taking stressed punctuation as 1 and unstressed punctuation as 0 as an example):

[0056] First, the prosodic word stress information of the second training text is labeled. The audio is played, and the prosodic words in the corresponding text that are emphasized (e.g., higher pitch, longer duration, or stronger intensity) are identified. The emphasized prosodic words are marked with 1, and the rest are marked with 0. It should be noted that in a piece of text corresponding to an audio, multiple prosodic words can be marked with 1 (indicating that there are multiple emphasized parts in the text that need to be stressed), or all prosodic words can be marked with 0 (indicating that there are no parts in the text that need to be stressed and do not need to be stressed).

[0057] Next, the word segmentation emphasis information of the second training text is labeled, and the audio of the prosodic word (composed of at least one word) that was marked as 1 in the previous step is played. The word segment that is emphasized in the audio of the prosodic word is identified, and the emphasized word segment is marked as 1, and the rest are marked as 0.

[0058] Finally, the syllable stress information of the second training text is labeled, and the audio of the segment corresponding to the segment marked as 1 in the previous step (consisting of at least one character, each character corresponding to one syllable) is played. The syllables that are emphasized in the segment audio are identified, and the emphasized syllables are marked as 1, and the rest are marked as 0.

[0059] Furthermore, to improve the consistency of annotation for focal stress, when annotating syllable stress information in the second training text, annotation can also be performed according to preset intra-word stress rules. Examples of annotation based on preset intra-word stress rules have been disclosed above and will not be repeated here.

[0060] For example, annotators can annotate text using a designated annotation page. Annotation pages can be like... Figure 2 As shown, Figure 2 A partial schematic diagram of an exemplary annotation interface is shown, where the target text is B1B2B3B4B5B6B7B8B9B10B11B12B13 containing 13 characters. Figure 1 The first layer is the sentence layer, used to display the target text. The second layer is used to display the boundaries of prosodic words. The third to fifth layers need to be annotated. The third layer is used to annotate the stress information of prosodic words, the fourth layer is used to annotate the stress information of word segments, and the fifth layer is used to annotate the stress information of syllables. The annotation method for the third to fifth layers can refer to the annotation process above, and will not be repeated here.

[0061] By using the above method, the focus stress annotation information corresponding to the real speech of the second training text can be obtained. Therefore, by using the second training text as input to the neural network model and the focus stress annotation information corresponding to the second training text as the model's target output, the neural network model can be trained. After training, a focus stress annotation model that can automatically generate focus stress annotation information for text can be obtained. Thus, by inputting a piece of text into this focus stress annotation model, the focus stress annotation information corresponding to that piece of text can be automatically obtained from the model's output, eliminating the need for manual annotation and improving the efficiency of determining focus stress annotation information.

[0062] Optionally, as mentioned above, when annotating syllable stress information, word segmentation stress information and prosodic word stress information are also annotated simultaneously. Therefore, the focus stress annotation information of the target text may include, in addition to syllable stress information, at least one of the word segmentation stress information and prosodic word stress information. The explanations and annotation methods for word segmentation stress information and prosodic word stress information have been given above and will not be repeated here.

[0063] In this way, the focus stress annotation information not only provides syllable stress information, but also prosodic word stress information and word segmentation stress information. It can not only reflect the stressed syllable, but also the position of the stressed syllable in the sentence. The focus stress information is richer and is conducive to the subsequent generation of synthesized audio that is more in line with human hearing.

[0064] Optionally, the word segmentation repetition information of the target text can also be used to indicate word boundaries in the target text. Accordingly, the method provided in this disclosure may further include the following steps:

[0065] Based on the syllable stress information of the target text and the preset intra-word stress rules, determine whether there are target word boundaries that violate the intra-word stress rules in the word boundaries indicated by the word segmentation stress information of the target text;

[0066] If target word boundaries exist, modify the word boundaries of the target text so that the modified word segmentation repetition information conforms to the intra-word stress rules.

[0067] Among them, the intra-word stress rules are used to indicate the distribution pattern of stressed syllables in the word segment that should be stressed.

[0068] That is to say, when the syllable stress information corresponding to a certain word segment does not conform to the predefined in-word stress rules, the word boundary can be corrected so that the corrected content conforms to the in-word stress rules. For example, if the in-word stress rules require that either only the first syllable or all syllables in a word segment should be stressed, then if "Department of Chinese" is a word segment to be stressed and its syllable annotation information indicates that only "zhong" and "wen" should be stressed, it can be known that this word segment violates the in-word stress rules and needs to be corrected. The word boundary of "Department of Chinese" needs to be re-planned and corrected to "zhongwen|xi", where "|" is the newly added word boundary.

[0069] In addition, the word boundary indicated by the word segment stress information can also be modified according to the actual semantics of the target text. For example, automatic word segmentation may divide "yikou|qi" into "yi|kouqi", which needs to be corrected.

[0070] The determination of the tone annotation information of the target text can refer to the determination method of the focus stress annotation information and can be carried out by manual annotation or automatic prediction. Among them, the manual annotation method can be implemented by referring to the definition of the tone annotation information in the previous text, and the automatic prediction method can refer to the automatic prediction method of the focus stress annotation model. First, manual annotation is performed on the training text and its audio in advance, and then a neural network model is used for training to obtain a tone annotation model. Then, the tone annotation information is automatically predicted using the tone annotation model. The determination of the prosody annotation information is the same, and no detailed description will be given here.

[0071] Returning to Figure 1 , in step 13, a synthesized audio corresponding to the target text is generated according to the tone annotation information, prosody annotation information, focus stress annotation information, and phoneme sequence of the target text.

[0072] In a possible implementation manner, step 13 may include the following steps 31 to step 35, as Figure 3 shown.

[0073] In step 31, according to the tone annotation information of the target text, tone labels at the phoneme level are determined.

[0074] As mentioned above, the tone types indicated by the tone annotation information may include, but are not limited to, one of the first tone, second tone, third tone, fourth tone, light tone, and the third tone that needs tone sandhi. Correspondingly, the tone labels may include, but are not limited to, one of the first tone, second tone, third tone, fourth tone, light tone, and the third tone that needs tone sandhi. Among them, the idea of determining the tone labels is that different phonemes of the same syllable share the same tone label, that is, the tone labels of the phonemes constituting a syllable are the same as the tone label of that syllable.

[0075] In step 32, phoneme-level prosodic labels are determined based on the prosodic annotation information of the target text.

[0076] As mentioned above, prosodic annotation information may include prosodic boundary information, and correspondingly, prosodic labels may include prosodic boundary labels.

[0077] Prosodic annotation information typically marks a specific location in the text, such as a prosodic phrase boundary. To facilitate subsequent speech synthesis and ensure that the prosodic annotation information corresponds one-to-one with the phonemes of the text to be synthesized, phoneme-level prosodic tags can be further determined based on the prosodic annotation information.

[0078] The approach to determining prosodic labels is as follows: for phoneme positions with prosodic annotation information, label content is generated according to the annotation information; for phoneme positions without prosodic annotation information, a specified alternative content is used. For example, for the phoneme sequence {A1, A2, A3, A4, A5, A6}, assuming that the prosodic annotation information includes prosodic boundary information, and the annotation content is that there is a prosodic phrase boundary at A2 and an intonation phrase boundary at A5, and that prosodic phrase boundaries are represented by 3, intonation phrase boundaries by 4, and no label is represented by N2, then the determined prosodic boundary labels are {N2, 3, N2, N2, 4, N2}.

[0079] In step 33, the phoneme-level focus accent labels are determined based on the focus accent annotation information of the target text.

[0080] As mentioned above, focus stress annotation information may include syllable stress information, and may also include at least one of prosodic word stress information and word segmentation stress information. Accordingly, focus stress labels may include syllable stress labels, and may also include at least one of prosodic word stress labels and word segmentation stress labels.

[0081] For each type of focus stress annotation information, a corresponding focus stress label can be generated. For example, assuming the prosodic annotation information of the target text includes syllable stress information, prosodic word stress information, and word segmentation stress information, then syllable stress labels, prosodic word stress labels, and word segmentation stress labels will be generated respectively, and each label corresponds one-to-one with each phoneme in the phoneme sequence. For instance, if we follow the method of labeling stressed words with 1 and unstressed words with 0, then the syllable stress labels, prosodic word stress labels, and word segmentation stress labels can all be sequences composed of 0s and 1s, corresponding one-to-one with each phoneme.

[0082] In step 34, acoustic feature information corresponding to the target text is generated using a pre-trained speech synthesis model based on tone labels, prosody labels, focus accent labels, and phoneme sequences.

[0083] In other words, tone labels, prosody labels, focus stress labels, and phoneme sequences can be input into a pre-trained speech synthesis model to obtain acoustic feature information corresponding to the target text. For example, acoustic feature information can be Mel spectrum, linear spectrum, etc.

[0084] The aforementioned speech synthesis model may include an encoding module, an attention module, and a decoding module. The encoding module generates a text representation sequence based on the concatenated vectors corresponding to tone labels, prosody labels, focus stress labels, and phoneme sequences. The attention module generates a semantic representation based on the text representation sequence. The decoding module outputs acoustic feature information corresponding to the target text based on the semantic representation.

[0085] The input to the encoder module of the speech synthesis model is a vector representation of the target text. This vector can include a first vector obtained by embedding the phoneme sequence, a second vector obtained by embedding the tone labels, a third vector obtained by embedding the prosody labels, and a fourth vector obtained by embedding the focus accent labels. These four vectors are concatenated to form a concatenated vector, which serves as the input to the encoder module. The encoder module then outputs a text embedding sequence (TE) of the target text. This text embedding sequence is then processed by an attention module to generate a context vector C, which serves as the semantic representation of the target text. The speech representation generated by the attention module enters the decoding module, which outputs the acoustic feature information corresponding to the target text.

[0086] For example, the speech synthesis model is trained in the following way:

[0087] Acquire training samples, wherein each training sample includes the training phoneme sequence, tone annotation information, prosody annotation information and focal stress annotation information corresponding to the first training text, as well as the training acoustic feature information corresponding to the first training text;

[0088] Based on the tone annotation information, prosody annotation information, and focal stress annotation information of the first training text, determine the phoneme-level training tone label, training prosody label, and training focal stress label;

[0089] The model is trained by using the concatenated vectors corresponding to the training phoneme sequence, training tone label, training prosody label, and training focus accent label as inputs to the model, and the training acoustic feature information as the target output of the model, so as to obtain the trained speech synthesis model.

[0090] For example, the speech synthesis model described above can use the Tacotron model.

[0091] The first training text corresponds to an audio file. The acoustic feature information of the audio file is determined and used as the training acoustic feature information.

[0092] In this disclosure, the training factor sequence of the first training text can be determined in a manner similar to that used in step 12 to determine the factor sequence of the target text. Furthermore, the training tone label, training prosody label, and training focus accent label of the first training text can be determined in a manner similar to that used in determining the tone label, prosody label, and focus accent label of the target text. The above will not be repeated here.

[0093] The goal of training a speech synthesis model is to make the synthesized audio output by the model infinitely close to the actual audio of the first training sample; that is, to make the acoustic feature information output by the model infinitely close to the acoustic feature information trained on the training sample. Therefore, based on the acoustic feature information trained on the training sample and the acoustic feature information output by the model during training, the model's loss value can be calculated, and this loss value can be used to adjust the internal parameters of the current model. Then, the adjusted model is used in the next training iteration, and this process is repeated until the conditions for stopping training are met, thus obtaining a fully trained speech synthesis model.

[0094] The trained speech synthesis model obtained after the above training steps can be used in speech synthesis scenarios. For example, during the application of the speech synthesis model, step 34 may include the following steps:

[0095] The concatenated vectors corresponding to the tone labels, prosody labels, focal stress labels, and phoneme sequences of the target text are input into the speech synthesis model to obtain the acoustic feature information of the speech synthesis model corresponding to the target text.

[0096] In step 35, synthesized audio is obtained based on the acoustic feature information corresponding to the target text.

[0097] After obtaining the acoustic feature information of the target text through the speech synthesis model in step 34, the acoustic feature information can be input into a vocoder (e.g., Wavenet vocoder, Griffin-Lim vocoder) to perform speech synthesis, thereby obtaining the synthesized audio corresponding to the text to be synthesized.

[0098] In the above embodiments, the tone information, rhythm information, and focus accent information are accurate to the phoneme level, resulting in higher control precision. This ensures the controllability and naturalness of the accent while achieving accurate tone and rhythm performance.

[0099] Figure 4 This is a block diagram of a speech synthesis apparatus provided according to one embodiment of the present disclosure. Figure 4 As shown, the device 40 may include:

[0100] The first determining module 41 is used to determine the tone marking information, prosody marking information and focal stress marking information of the target text to be processed, wherein the focal stress marking information includes the syllable stress information of the target text, and the syllable stress information is used to indicate whether each syllable in the target text should be stressed.

[0101] The second determining module 42 is used to determine the phoneme sequence of the target text;

[0102] The generation module 43 is used to generate a synthesized audio corresponding to the target text based on the tone marking information, the prosody marking information, the focus accent marking information and the phoneme sequence of the target text.

[0103] Optionally, the generation module 43 includes:

[0104] The first determining submodule is used to determine the tone label at the phoneme level based on the tone annotation information of the target text;

[0105] The second determining submodule is used to determine phoneme-level prosodic labels based on the prosodic annotation information of the target text;

[0106] The third determining submodule is used to determine the phoneme-level focus accent label based on the focus accent annotation information of the target text;

[0107] The generation submodule is used to generate acoustic feature information corresponding to the target text based on the tone label, the prosody label, the focus stress label and the phoneme sequence using a pre-trained speech synthesis model;

[0108] The audio synthesis submodule is used to obtain synthesized audio based on the acoustic feature information corresponding to the target text.

[0109] Optionally, the speech synthesis model includes an encoding module, an attention module, and a decoding module; wherein, the encoding module is used to generate a text representation sequence based on the concatenation vector corresponding to the tone label, the prosody label, the focal stress label, and the phoneme sequence; the attention module is used to generate a semantic representation based on the text representation sequence; and the decoding module is used to output acoustic feature information corresponding to the target text based on the semantic representation.

[0110] Optionally, the speech synthesis model is trained using the following modules:

[0111] The acquisition module is used to acquire training samples, wherein each training sample includes a training phoneme sequence, tone annotation information, prosody annotation information and focal stress annotation information corresponding to the first training text, and training acoustic feature information corresponding to the first training text.

[0112] The third determining module is used to determine the phoneme-level training tone label, training prosody label, and training focus accent label based on the tone labeling information, prosody labeling information, and focus accenting information of the first training text.

[0113] The training module is used to train the model by taking the concatenated vectors corresponding to the training phoneme sequence, the training tone label, the training prosody label and the training focus accent label as the model input and the training acoustic feature information as the model target output, so as to obtain the trained speech synthesis model.

[0114] Optionally, the focus stress annotation information of the target text also includes the word segmentation stress information and / or the prosodic word stress information of the target text. The word segmentation stress information of the target text is used to indicate whether each word in the target text should be stressed, and the prosodic word stress information of the target text is used to indicate whether each prosodic word in the target text should be stressed.

[0115] Optionally, the word segmentation repetition information of the target text is also used to indicate the word boundaries of the target text;

[0116] The device 40 further includes:

[0117] The fourth determining module is used to determine whether there are target word boundaries that violate the intra-word stress rules in the word boundaries indicated by the word segmentation stress information of the target text, based on the syllable stress information of the target text and the preset intra-word stress rules;

[0118] The modification module is used to modify the word boundaries of the target text if the target word boundaries exist, so that the modified word segmentation repetition information conforms to the intra-word stress rules;

[0119] The intra-word stress rules are used to indicate the distribution pattern of stressed syllables in word segments that should be stressed.

[0120] Optionally, the focus accent information of the target text is obtained through the following modules:

[0121] The focus accent annotation module is used to input the target text into the focus accent annotation model and obtain the focus accent annotation information output by the focus accent annotation model. The focus accent annotation model is trained based on a second training text with focus accent annotation information.

[0122] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0123] The following is for reference. Figure 5 This diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0124] like Figure 5 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0125] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0126] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0127] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0128] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0129] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0130] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: determine tone marking information, prosody marking information, and focal stress marking information of a target text to be processed, wherein the focal stress marking information includes syllable stress information of the target text, the syllable stress information being used to indicate whether each syllable in the target text should be stressed; determine the phoneme sequence of the target text; and generate a synthesized audio corresponding to the target text based on the tone marking information, the prosody marking information, the focal stress marking information, and the phoneme sequence of the target text.

[0131] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the modules do not necessarily limit the module itself; for example, the first determining module can also be described as "a module for determining the tone marking information, prosody marking information, and focus stress marking information of the target text to be processed."

[0134] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0135] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0136] According to one or more embodiments of this disclosure, Example 1 provides a speech synthesis method, the method comprising:

[0137] The tone marking information, prosody marking information, and focal stress marking information of the target text to be processed are determined, wherein the focal stress marking information includes the syllable stress information of the target text, and the syllable stress information is used to indicate whether each syllable in the target text should be stressed;

[0138] Determine the phoneme sequence of the target text;

[0139] Based on the tone annotation information, prosody annotation information, focal stress annotation information, and phoneme sequence of the target text, a synthesized audio corresponding to the target text is generated.

[0140] According to one or more embodiments of this disclosure, Example 2 provides the method of Example 1, wherein generating a synthesized audio corresponding to the target text based on the tone marking information, prosodic marking information, focal stress marking information, and phoneme sequence of the target text includes:

[0141] Based on the tone annotation information of the target text, determine the tone labels at the phoneme level;

[0142] Based on the prosodic annotation information of the target text, determine the phoneme-level prosodic labels;

[0143] Based on the focus accent annotation information of the target text, determine the phoneme-level focus accent labels;

[0144] Based on the tone label, the prosody label, the focal stress label, and the phoneme sequence, an acoustic feature information corresponding to the target text is generated using a pre-trained speech synthesis model.

[0145] Synthesized audio is obtained based on the acoustic feature information corresponding to the target text.

[0146] According to one or more embodiments of this disclosure, Example 3 provides the method of Example 2, wherein the speech synthesis model includes an encoding module, an attention module, and a decoding module; wherein the encoding module is used to generate a text representation sequence based on a concatenation vector corresponding to the tone label, the prosody label, the focus accent label, and the phoneme sequence; the attention module is used to generate a semantic representation based on the text representation sequence; and the decoding module is used to output acoustic feature information corresponding to the target text based on the semantic representation.

[0147] According to one or more embodiments of this disclosure, Example 4 provides the method of Example 2, wherein the speech synthesis model is trained in the following manner:

[0148] Acquire training samples, wherein each training sample includes a training phoneme sequence, tone annotation information, prosody annotation information and focal stress annotation information corresponding to the first training text, and training acoustic feature information corresponding to the first training text;

[0149] Based on the tone annotation information, prosody annotation information, and focal stress annotation information of the first training text, determine the phoneme-level training tone label, training prosody label, and training focal stress label;

[0150] The speech synthesis model is trained by using the concatenated vectors corresponding to the training phoneme sequence, the training tone label, the training prosody label, and the training focus accent label as inputs to the model, and using the training acoustic feature information as the target output of the model.

[0151] According to one or more embodiments of this disclosure, Example 5 provides a method as described in any one of Examples 1-4, wherein the focus stress annotation information of the target text further includes word segmentation stress information and / or prosodic word stress information of the target text, wherein the word segmentation stress information of the target text is used to indicate whether each word in the target text should be stressed, and the prosodic word stress information of the target text is used to indicate whether each prosodic word in the target text should be stressed.

[0152] According to one or more embodiments of this disclosure, Example 6 provides the method of Example 5, wherein the word segmentation repetition information of the target text is further used to indicate the word boundaries of the target text;

[0153] The method further includes:

[0154] Based on the syllable stress information of the target text and the preset intra-word stress rules, determine whether there are target word boundaries that violate the intra-word stress rules in the word boundaries indicated by the word segmentation stress information of the target text;

[0155] If the target word boundary exists, the word boundary of the target text is modified so that the modified word segmentation repetition information conforms to the intra-word stress rules;

[0156] The intra-word stress rules are used to indicate the distribution pattern of stressed syllables in word segments that should be stressed.

[0157] According to one or more embodiments of this disclosure, Example 7 provides the method of any one of Examples 1-4, wherein the focus accent information of the target text is obtained in the following manner:

[0158] The target text is input into the focus accent annotation model to obtain the focus accent annotation information output by the focus accent annotation model. The focus accent annotation model is trained based on a second training text with focus accent annotation information.

[0159] According to one or more embodiments of this disclosure, Example 8 provides a speech synthesis apparatus, the apparatus comprising:

[0160] The first determining module is used to determine the tone marking information, prosody marking information and focal stress marking information of the target text to be processed, wherein the focal stress marking information includes the syllable stress information of the target text, and the syllable stress information is used to indicate whether each syllable in the target text should be stressed;

[0161] The second determining module is used to determine the phoneme sequence of the target text;

[0162] The generation module is used to generate a synthesized audio corresponding to the target text based on the tone annotation information, prosody annotation information, focal stress annotation information and phoneme sequence of the target text.

[0163] According to one or more embodiments of the present disclosure, Example 9 provides a computer-readable medium having a computer program stored thereon that, when executed by a processing device, implements the steps of the method described in any one of Examples 1-7.

[0164] According to one or more embodiments of this disclosure, Example 10 provides an electronic device, including:

[0165] A storage device on which one or more computer programs are stored;

[0166] One or more processing means are configured to execute the one or more computer programs in the storage device to implement the steps of any one of the methods described in Examples 1-7.

[0167] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0168] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0169] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A speech synthesis method characterized by, The method comprises: determining tone annotation information, prosody annotation information and focus accent annotation information of a target text to be processed, wherein the focus accent annotation information comprises prosodic word stress information, word stress information and syllable stress information of the target text; the prosodic word stress information is used to indicate whether each prosodic word in the target text should be stressed; the word stress information is used to indicate whether each word in a target prosodic word should be stressed, the target prosodic word being annotated as a prosodic word that should be stressed in the prosodic word stress information; the syllable stress information is used to indicate whether each syllable in a target word should be stressed, the target word being a word annotated as a word that should be stressed in the word stress information; determining a phoneme sequence of the target text; generating synthesized audio corresponding to the target text according to the tone annotation information, the prosody annotation information, the focus accent annotation information and the phoneme sequence of the target text; The word stress information of the target text is also used to indicate the word boundaries of the target text; the method further comprises: determining whether there is a target word boundary that violates the intra-word accent rule in the word boundary indicated by the word stress information of the target text according to the syllable stress information of the target text and a preset intra-word accent rule; if there is the target word boundary, modifying the word boundary of the target text so that the modified word stress information conforms to the intra-word accent rule; wherein the intra-word accent rule is used to indicate the distribution rule of the stressed syllables in the word that should be stressed.

2. The method of claim 1, wherein, The generating of the synthesized audio corresponding to the target text according to the tone annotation information, the prosody annotation information, the focus accent annotation information and the phoneme sequence of the target text comprises: determining tone labels at the phoneme level according to the tone annotation information of the target text; determining prosody labels at the phoneme level according to the prosody annotation information of the target text; determining focus accent labels at the phoneme level according to the focus accent annotation information of the target text; generating acoustic feature information corresponding to the target text by using a pre-trained speech synthesis model according to the tone labels, the prosody labels, the focus accent labels and the phoneme sequence; obtaining synthesized audio according to the acoustic feature information corresponding to the target text.

3. The method of claim 2, wherein, The speech synthesis model comprises an encoding module, an attention module and a decoding module; wherein the encoding module is used to generate a text representation sequence according to a concatenation vector corresponding to the tone labels, the prosody labels, the focus accent labels and the phoneme sequence; the attention module is used to generate a semantic representation according to the text representation sequence; and the decoding module is used to output acoustic feature information corresponding to the target text according to the semantic representation.

4. The method of claim 2, wherein, The speech synthesis model is trained by the following method: obtaining training samples, wherein each training sample comprises a training phoneme sequence, tone annotation information, prosody annotation information and focus accent annotation information corresponding to a first training text, and training acoustic feature information corresponding to the first training text; determine, according to the tone labeling information, the prosody labeling information and the focus accent labeling information of the first training text, training tone labels, training prosody labels and training focus accent labels at a phoneme level; perform model training by taking the concatenation vectors corresponding to the training phoneme sequence, the training tone labels, the training prosody labels and the training focus accent labels as input of the model and taking the training acoustic feature information as target output of the model, to obtain the trained speech synthesis model.

5. The method according to any one of claims 1-4, characterized in that, The focus accent labeling information of the target text is obtained in the following manner: input the target text into a focus accent labeling model to obtain focus accent labeling information output by the focus accent labeling model, the focus accent labeling model being trained based on a second training text with focus accent labeling information.

6. A speech synthesis apparatus characterized by comprising: The apparatus comprises: a first determining module configured to determine tone labeling information, prosody labeling information and focus accent labeling information of a target text to be processed, wherein the focus accent labeling information comprises prosodic word stress information, word stress information and syllable stress information of the target text; the prosodic word stress information is used to indicate whether each prosodic word in the target text should be stressed; the word stress information is used to indicate whether each word in a target prosodic word should be stressed, the target prosodic word being labeled as a prosodic word that should be stressed in the prosodic word stress information; the syllable stress information is used to indicate whether each syllable in a target word should be stressed, the target word being labeled as a word that should be stressed in the word stress information; a second determining module configured to determine a phoneme sequence of the target text; a generating module configured to generate a synthesized audio corresponding to the target text according to the tone labeling information, the prosody labeling information, the focus accent labeling information and the phoneme sequence of the target text. The word stress information of the target text is also used to indicate a word boundary of the target text; the apparatus is further configured to: determine, according to the syllable stress information of the target text and a preset intra-word accent rule, whether there is a target word boundary that violates the intra-word accent rule in the word boundary indicated by the word stress information of the target text; if there is the target word boundary, modify the word boundary of the target text so that the modified word stress information conforms to the intra-word accent rule; wherein the intra-word accent rule is used to indicate a distribution rule of stressed syllables in a word that should be stressed.

7. A computer readable medium having stored thereon a computer program, characterized in that The program, when executed by a processing apparatus, implements the steps of the method of any one of claims 1-5.

8. An electronic device, comprising: comprise: a storage device having one or more computer programs stored thereon; one or more processing devices configured to execute the one or more computer programs in the storage device to implement the steps of the method of any one of claims 1-5.

Images