Speech synthesis method and device

Abstract

The present disclosure relates to a speech synthesis method and device, wherein the method includes obtaining a text to be synthesized; performing phoneme conversion on the text to be synthesized to obtain a first phoneme sequence, wherein the first phoneme sequence includes at least one phoneme and the tone of the syllable in which the at least one phoneme is located; decoupling the phonemes and tones in the first phoneme sequence, and extracting the phoneme features of the text to be synthesized based on the decoupled second phoneme sequence; wherein the second phoneme sequence includes the at least one phoneme; performing text encoding on the text to be synthesized in the syllable dimension to extract the semantic features of the text to be synthesized; fusing the semantic features and the phoneme features, and generating speech based on the fused features. Through the present disclosure, the synthesized speech is not only accurately pronounced, but also has a natural rhythm, ensuring that the synthesized speech achieves the expected effect in terms of naturalness, rhythm, and semantic consistency.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech synthesis method and device. Background Art

[0002] Text-to-Speech (TTS) is a technology that converts text information into speech information. In related technologies, speech synthesis often suffers from problems such as pronunciation errors and lack of rhythm. Summary of the Invention

[0003] In view of this, the present disclosure proposes a speech synthesis method, apparatus, electronic device, storage medium, and computer program product.

[0004] According to one aspect of the present disclosure, a speech synthesis method is provided, the method comprising:

[0005] Get the text to be synthesized;

[0006] Performing phoneme conversion on the text to be synthesized to obtain a first phoneme sequence, wherein the first phoneme sequence includes at least one phoneme and a tone of a syllable in which the at least one phoneme is located;

[0007] Decoupling the phonemes and tones in the first phoneme sequence, and extracting phoneme features of the text to be synthesized based on the decoupled second phoneme sequence; wherein the second phoneme sequence includes the at least one phoneme;

[0008] Performing text encoding on the text to be synthesized in the syllable dimension to extract semantic features of the text to be synthesized;

[0009] The semantic features and the phoneme features are fused, and speech is generated based on the fused features.

[0010] In a possible implementation, performing phoneme conversion on the text to be synthesized to obtain a first phoneme sequence includes:

[0011] Based on the target language corresponding to the text to be synthesized, filtering characters in a non-target language in the text to be synthesized to obtain a filtered text;

[0012] Segmenting the filtered text according to preset pause symbols corresponding to the target language to obtain multiple clauses; wherein the preset pause symbols are used to indicate pauses in sentences;

[0013] Performing phoneme conversion processing on each of the multiple clauses to obtain a phoneme subsequence corresponding to each clause;

[0014] The phoneme subsequences corresponding to each of the clauses are merged to obtain the first phoneme sequence.

[0015] In a possible implementation, the method further includes:

[0016] Based on the first phoneme sequence and the second phoneme sequence, a tone sequence is obtained; the tone sequence includes a tone corresponding to the at least one phoneme; wherein the tone corresponding to any phoneme is determined by the tone of the syllable in which the phoneme is located;

[0017] In the process of generating speech based on the fused features, the pitch of the speech is controlled by the pitch sequence.

[0018] In a possible implementation, encoding the text to be synthesized in the syllable dimension and extracting semantic features of the text to be synthesized includes:

[0019] Performing word segmentation on the text to be synthesized according to syllables to obtain at least one syllable;

[0020] Determining a code corresponding to each syllable in the at least one syllable in combination with a preset vocabulary, wherein the preset vocabulary includes a correspondence between different syllables and different codes;

[0021] The semantic features of the text to be synthesized are extracted based on the code corresponding to each syllable in the at least one syllable.

[0022] In a possible implementation, performing phoneme conversion on each of the multiple clauses to obtain a phoneme subsequence corresponding to each clause includes:

[0023] For any clause among the multiple clauses, determining whether the clause contains a preset character; and performing normalization processing if the clause contains the preset character;

[0024] The normalized clause is subjected to phoneme conversion to obtain a phoneme subsequence corresponding to the clause.

[0025] In a possible implementation, performing phoneme conversion on each of the multiple clauses to obtain a phoneme subsequence corresponding to each clause includes:

[0026] For any one of the multiple clauses, perform phoneme conversion processing using a phoneme conversion model to obtain a phoneme conversion result of the clause;

[0027] determining the phoneme to be corrected in the phoneme conversion result of the clause;

[0028] Based on a preset phoneme correction dictionary, the phonemes to be corrected in the phoneme conversion result of the clause are converted into standard phonemes to obtain a phoneme subsequence corresponding to the clause; wherein the phoneme correction dictionary includes: corresponding relationships between different phonemes to be corrected and different standard phonemes.

[0029] In a possible implementation, the method further includes:

[0030] In the process of generating speech based on the fused features, the duration of each phoneme in the at least one phoneme is predicted based on a duration model; wherein the duration model is trained based on real alignment labels of phoneme-duration.

[0031] In a possible implementation, the method further includes:

[0032] Obtaining a preset duration corresponding to each of the multiple clauses;

[0033] When the duration of the generated speech exceeds the sum of the preset durations of the multiple clauses, at least two adjacent clauses in the speech are subjected to axis-merging processing. After the axis-merging processing, one or more spaces are added between the at least two adjacent clauses so that the duration of the processed speech is no greater than the sum of the preset durations of the multiple clauses.

[0034] According to another aspect of the present disclosure, a speech synthesis device is provided, the device comprising:

[0035] An acquisition module is used to obtain the text to be synthesized;

[0036] a phoneme conversion module, configured to perform phoneme conversion on the text to be synthesized to obtain a first phoneme sequence, wherein the first phoneme sequence includes at least one phoneme and a tone of a syllable in which the at least one phoneme is located;

[0037] a phoneme feature extraction module, configured to decouple the phonemes and tones in the first phoneme sequence, and extract the phoneme features of the text to be synthesized based on a second phoneme sequence after the decoupling; wherein the second phoneme sequence includes the at least one phoneme;

[0038] A semantic feature extraction module, configured to perform text encoding on the text to be synthesized in a syllable dimension and extract semantic features of the text to be synthesized;

[0039] The speech synthesis module is used to fuse the semantic features and the phoneme features and generate speech based on the fused features.

[0040] According to another aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0041] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0042] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0043] Through the above aspects of the present disclosure, a text to be synthesized is obtained; the text to be synthesized is converted into phonemes to obtain a first phoneme sequence, wherein the first phoneme sequence includes at least one phoneme and the tone of the syllable in which the at least one phoneme is located; the phonemes and the tones in the first phoneme sequence are decoupled, and the phoneme features of the text to be synthesized are extracted based on the decoupled second phoneme sequence; wherein the second phoneme sequence includes the at least one phoneme; text encoding is performed on the text to be synthesized in the syllable dimension, and semantic features of the text to be synthesized are extracted; the semantic features and the phoneme features are fused, and speech is generated based on the fused features. In this way, the phonemes and tones in the first phoneme sequence are decoupled, thereby isolating the spatial domain of the tone, and then the phoneme features of the text to be synthesized are extracted based on the decoupled second phoneme sequence, thereby improving the accuracy of the extracted phoneme features; at the same time, the text to be synthesized is encoded in the syllable dimension, which can obtain more accurate and rich semantic features; then, the phoneme features extracted from the decoupled second phoneme sequence are fused with the semantic features, thereby ensuring the quality of the fused features, and the speech synthesized based on the fused features is not only accurately pronounced, but also has a natural rhythm, ensuring that the synthesized speech achieves the expected effect in terms of naturalness, rhythm and semantic consistency.

[0044] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0046] Figure 1 A flowchart of a speech synthesis method according to an embodiment of the present disclosure is shown;

[0047] Figure 2 A schematic structural diagram of a VITS model according to an embodiment of the present disclosure is shown;

[0048] Figure 3A schematic diagram illustrating a method for extracting semantic features and phoneme features according to an embodiment of the present disclosure is shown;

[0049] Figure 4 A schematic diagram of a process for performing speech synthesis using a speech synthesis model according to an embodiment of the present disclosure is shown;

[0050] Figure 5 A structural diagram of a speech synthesis device according to an embodiment of the present disclosure is shown;

[0051] Figure 6 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0052] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0053] As used herein, the terms "comprises," "comprising," "having," or variations thereof are open ended and include one or more stated features, integers, elements, steps, parts, or functions, but do not preclude the presence or addition of one or more other features, integers, elements, steps, parts, functions, or groups thereof.

[0054] When an element is referred to as being "connected," "coupled," "responsive" or variations thereof to another element, it can be directly connected, coupled or responsive to the other element or intervening elements may be present.

[0055] Although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Therefore, without departing from the teachings of the present invention, the first element / operation in some embodiments may be referred to as the second element / operation in other embodiments.

[0056] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0057] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0058] In scenarios such as dubbing for film and television dramas, speech synthesis is not limited to Chinese but has expanded to various minority languages. For example, Thai subtitles for Chinese films and TV dramas can be converted into Thai audio through speech synthesis. However, speech synthesis for different languages, especially for minority languages, can often lead to problems such as pronunciation errors and lack of rhythm.

[0059] In order to solve the above-mentioned problems, the embodiments of the present disclosure provide a speech synthesis method (detailed description is provided below), which performs specific optimization on the speech synthesis of the target language according to the uniqueness of the target language; thereby further improving the quality of the synthesized speech of the target language, ensuring that the synthesized speech is not only accurately pronounced but also has a natural rhythm, providing the audience with an immersive listening experience.

[0060] Exemplarily, the speech synthesis method can be executed by an electronic device such as a terminal device or a server, wherein the terminal device can be a desktop terminal or a mobile terminal, for example, various types of electronic devices such as a laptop computer, a tablet computer, a desktop computer, a smartphone, a smart speaker, a smart watch, a smart TV, and an in-vehicle terminal. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms.

[0061] It should be noted that the above-mentioned film and television drama dubbing production scenario described in the embodiment of the present disclosure is for the purpose of more clearly illustrating the technical solution of the embodiment of the present disclosure, and does not constitute a limitation on the technical solution provided by the embodiment of the present disclosure. Ordinary technicians in this field can know that the technical solution provided by the embodiment of the present disclosure is also applicable to similar technical problems when other similar or new scenarios emerge.

[0062] Figure 1 A flow chart of a speech synthesis method according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the following steps may be included:

[0063] Step 101: Obtain the text to be synthesized.

[0064] The text to be synthesized can be any form of text. For example, it can be dialogue from a film or TV series, lyrics from a piece of music, news reports, etc. The language used in the text to be synthesized (i.e., the target language) can be Chinese, English, Thai, Vietnamese, etc., without limitation. As an example, the target language of the text to be synthesized is a tonal language; for example, Chinese, Thai, Vietnamese, etc.

[0065] Step 102: Perform phoneme conversion on the text to be synthesized to obtain a first phoneme sequence, where the first phoneme sequence includes at least one phoneme and the tone of the syllable where the at least one phoneme is located.

[0066] Phonemes are the smallest units of speech divided according to the natural properties of speech. Different languages ​​have corresponding phonemes. For example, in Chinese, “b”, “p”, “a”, “o”, etc. are all phonemes, and in Thai, “b”, “p”, “a”, “o”, etc. are all phonemes. etc. are all phonemes.

[0067] Tone, also known as tone or sub-tone, indicates the pitch of a sound. In tonal languages, the same sound, pronounced with different lengths and pitch values, can convey different meanings (i.e., semantics or meanings). Therefore, tone can be used to express semantic differences. Furthermore, the rhythmic changes in speech brought about by tone play an important role in tone and emotional expression. Different tonal languages ​​may have different corresponding tones. For example, Chinese has four tones: 1st tone (yinping), 2nd tone (yangping), 3rd tone (shangsheng), and 4th tone (qusheng). Thai, for another example, has five tones: 1st tone (low flat), 2nd tone (medium rising), 3rd tone (low falling), 4th tone (high falling), and 5th tone (high rising).

[0068] A syllable is a phonetic unit composed of phonemes. It is the smallest phonetic segment that people naturally perceive when speaking. Different languages ​​have different syllable structure rules. For example, in Chinese, "mā" is a syllable, which is composed of the phonemes "m" and "a", and the corresponding tone is "yinping"; for another example, in Thai, is a syllable, which consists of phonemes The corresponding tone is "low flat tone".

[0069] Exemplarily, phoneme conversion can be performed using a phoneme conversion model; wherein, different languages ​​have different corresponding phoneme conversion models, and a pre-trained phoneme conversion model corresponding to the target language can be selected, and the text to be synthesized is input into the phoneme conversion model, and the phoneme conversion model performs phoneme conversion on the text to be synthesized and outputs a first phoneme sequence. The phoneme conversion model can be trained using an existing method, which is not limited to this. As an example, the phoneme conversion model can be a grapheme-to-phoneme (G2P) model, which performs phoneme conversion on the text to be synthesized based on a preset phoneme dictionary of the target language and outputs a first phoneme sequence.

[0070] It can be understood that for the same sentence in the target language, different phoneme dictionaries may generate different first phoneme sequences when performing phoneme conversion. Exemplarily, a more fine-grained phoneme dictionary can be constructed in advance according to the characteristics of the target language. For example, the preset phoneme dictionary of the target language contains the correspondence between different syllables and phonemes of the target language. In this way, when the phoneme conversion model performs phoneme conversion on the text to be synthesized based on this phoneme dictionary, it can first determine the syllables corresponding to the text to be synthesized, and then successively determine the phonemes included in each syllable based on this phoneme dictionary. At the same time, it can determine the tones corresponding to each syllable. Then, according to the order of appearance of each syllable in the text to be synthesized, a first phoneme sequence is generated. In this way, based on the preset phoneme dictionary of the target language, the text to be synthesized in the target language can be efficiently converted into a standardized phoneme sequence including phonemes and tones, realizing refined phoneme expression, and further making the pronunciation in the synthesized speech more natural.

[0071] The first phoneme sequence includes the phonemes included in each syllable corresponding to the text to be synthesized and the tones corresponding to each syllable. Among them, the phonemes of any syllable can be one or more, and the tone corresponding to this syllable is the tone of the syllable where different phonemes are located. As an example, the first phoneme sequence may include digital encodings of phonemes and tones. For example, taking "now" in Chinese as an example, based on the preset Chinese phoneme dictionary, the phoneme conversion model can convert the syllable "xiàn" corresponding to "现" in "现在" into phonemes "x", "i", "an", and digitally encode the corresponding tone "falling tone" as "4". It can convert the syllable "zài" corresponding to "在" into phonemes "z", "ai", and digitally encode the corresponding tone "falling tone" as "4". Finally, the phoneme sequence "xian4zai4" can be obtained. Another example, taking Thai as an example, based on the preset Thai phoneme dictionary, the phoneme conversion model can the corresponding syllable "sawa" into phonemes "sawa", and digitally encode the corresponding tone "mid-rising tone" as "2". It can convert the syllable "t" corresponding to into phonemes and digitally encode the corresponding tone "mid-rising tone" as "2". It can convert the syllable "di" corresponding to into phonemes "di", and digitally encode the corresponding tone "low-level tone" as "1". Finally, the phoneme sequence "sawa2 2di1" can be obtained.

[0072] In one possible implementation, performing phoneme conversion on the text to be synthesized to obtain a first phoneme sequence includes: filtering non-target language characters in the text to be synthesized based on the target language corresponding to the text to be synthesized to obtain a filtered text; segmenting the filtered text according to preset pause symbols corresponding to the target language to obtain a plurality of clauses, wherein the preset pause symbols are used to indicate pauses in sentences; performing phoneme conversion on each of the plurality of clauses to obtain a phoneme subsequence corresponding to each clause; and merging the phoneme subsequences corresponding to each clause to obtain the first phoneme sequence. In this way, the characteristics of the target language are fully considered, first filtering non-target language characters in the text to be synthesized, then segmenting the filtered text into clauses according to preset pause symbols corresponding to the target language, and finally performing phoneme conversion on each clause. This step-by-step processing effectively solves the problem of being unable to perform correct phoneme conversion on the entire text to be synthesized due to non-target language characters and target language-specific pause symbols, thereby ensuring the accuracy and reliability of phoneme conversion.

[0073] Non-target language characters can be pre-defined based on the characteristics of the target language. For example, if the target language is Chinese, Korean characters, Japanese characters, and spaces, which do not appear in standard Chinese expression structures, are all non-target language characters. This allows non-target language characters to be filtered out of the synthesized text, preventing them from interfering with the phoneme conversion process.

[0074] For example, corresponding pause symbols can be pre-set based on the characteristics of different languages. For example, for Chinese, the corresponding pause symbols may include commas, periods, question marks, exclamation marks, semicolons, etc. For Thai, the corresponding pause symbols may include spaces. For Vietnamese, the corresponding pause symbols may include spaces, periods, etc. In this way, based on the preset pause symbols corresponding to the target language, the correct segmentation of the target language text can be achieved. In particular, for languages ​​containing spaces (such as Thai and Vietnamese), the text of the language can be correctly segmented according to pause symbols such as spaces, and the semantic integrity of the segmented independent clauses can be guaranteed. Furthermore, phoneme conversion processing is performed on each clause to ensure that each clause can accurately generate a corresponding phoneme subsequence.

[0075] Exemplarily, a phoneme conversion model can be used to perform phoneme conversion on each of the multiple clauses segmented above. As an example, each of the multiple clauses can be input into a G2P model. The G2P model performs phoneme conversion on each clause based on a preset phoneme dictionary of the target language, thereby obtaining a phoneme subsequence corresponding to each clause. The phoneme subsequence may include the phonemes contained in each syllable in the clause and the tone corresponding to each syllable. After completing the phoneme conversion for each clause, the phoneme subsequences are merged into a complete phoneme sequence, i.e., the first phoneme sequence, based on the order in which the clauses appear in the text to be synthesized.

[0076] Furthermore, taking into account the characteristics of different languages, in the process of phoneme conversion using a phoneme conversion model, potential errors in the phoneme conversion process can be corrected through normalization, phoneme correction dictionary, etc., thereby improving the reliability and accuracy of subsequent generated speech.

[0077] In one possible implementation, performing phoneme conversion on each of the multiple clauses to obtain a phoneme subsequence corresponding to each clause includes: determining, for each clause in the multiple clauses, whether the clause contains a preset character; and performing normalization if the clause contains the preset character; and performing phoneme conversion on the normalized clause to obtain a phoneme subsequence corresponding to the clause. Thus, the normalization process effectively reduces the error rate during the phoneme conversion process and improves the accuracy and stability of the phoneme conversion.

[0078] Among them, the preset characters represent characters or character combinations that can appear in the expression structure of the target language standard but cannot be directly read out through the pronunciation rules of the target language. For example, the preset characters may include Arabic numerals, symbols, units and other characters, and may also include a combination of these characters and text characters. For any language, the preset characters can be pre-set according to the characteristics of the language; for example, "2kg", "5℃", "188XXXX0000", "1999 yuan" and so on in Chinese.

[0079] Normalization refers to converting preset characters into the written characters of the target language, where the written characters of the target language are in a standard and normative representation form, so that they can be pronounced according to the pronunciation rules of the target language. In this way, when a clause contains preset characters, normalization is performed through preset normalization rules to convert the clause containing preset characters into a standard and normative representation form to ensure correct speech generation in the subsequent process; among them, the preset normalization rules are used to indicate the correspondence between preset characters and the written characters of the target language, and the preset normalization rules can be set according to the characteristics of the target language. For example, in Chinese, the combination of numbers and characters "1999 yuan" is normalized to "one thousand nine hundred and ninety-nine yuan", the combination of numbers and characters "1999 year" is normalized to "nineteen ninety-nine year", the symbol "℃" is normalized to "degree Celsius", the combination of characters and numbers "phone number 188XXXX0000" is normalized to "phone number one eight eight XXXX zero zero zero zero", the combination of numbers and units "2 kg" is normalized to "two kilograms", etc.; for another example, in Thai and Vietnamese, for year numbers, a single-syllable morpheme simplified reading method is adopted, and the combination of numbers and characters "1999 year" is normalized to "one thousand and ninety-nine year".

[0080] Considering that when the phoneme conversion model performs phoneme conversion on any clause, it can first determine the syllables corresponding to the clause based on the correspondence between characters and syllables, and then for any syllable, convert it into the phonemes corresponding to the syllable based on the phoneme dictionary; however, due to the existence of polyphonic characters, that is, the same character may correspond to different syllables, therefore, the character may be mapped to the wrong syllable, resulting in incorrect phonemes finally converted. For example, the syllable corresponding to the character "行" in "银行" is "háng", while the syllable corresponding to it in "行走" is "xíng", and the correct phonemes corresponding to the two syllables are different; or, when training the phoneme conversion model, the training samples may have deficiencies such as noise, pronunciation habits, unclear pronunciation, accent differences, annotation errors, etc., so that the phoneme conversion model learns the wrong correspondence between characters and syllables, resulting in errors when the phoneme conversion model determines the syllables corresponding to each clause, and further leading to incorrect phonemes in the phoneme subsequence corresponding to the clause. For example, the syllable corresponding to the character "中" is "zhōng", and due to pronunciation habits, it may be marked as the pronunciation "zōng" in the training samples.

[0081] In a possible implementation, the process of respectively performing phoneme conversion processing on each of the multiple clauses to obtain a phoneme subsequence corresponding to each clause includes: for any one of the multiple clauses, performing phoneme conversion processing using a phoneme conversion model to obtain the phoneme conversion result of this clause; determining the phonemes to be corrected in the phoneme conversion result of this clause; based on a preset phoneme correction dictionary, converting the phonemes to be corrected in the phoneme conversion result of this clause into standard phonemes to obtain the phoneme subsequence corresponding to this clause; wherein, the phoneme correction dictionary includes: the corresponding relationships between different phonemes to be corrected and different standard phonemes. In this way, by determining the phonemes to be corrected in the phoneme conversion result of this clause, and then through the constructed phoneme correction dictionary, the correct mapping from the phonemes to be corrected to the standard phonemes is achieved, thereby correcting potential incorrect phonemes in real time during the phoneme conversion process, significantly improving the accuracy and stability of the phoneme conversion.

[0082] Exemplarily, after performing phoneme conversion processing using a phoneme conversion model to obtain the phoneme conversion result of any one clause, it is possible to detect the phoneme conversion result of the clause based on the pronunciation rules of the target language or common incorrect pronunciation problems to determine whether there are errors in the phoneme conversion result. If there are errors, then correspondingly determine the phonemes or phoneme combinations with errors in the phoneme conversion result as the phonemes to be corrected. If there are no errors, then correspondingly configure the phonemes to be corrected as empty. As an example, based on the pronunciation rules of Chinese, the phonemes 'j', 'q', 'x' do not directly combine with the phoneme 'a'. Therefore, if it is detected that 'ja', 'qa', or 'xa' exists in the phoneme conversion result, it is determined that there is an error in the phoneme conversion result, and the corresponding 'ja', 'qa', or 'xa' can be used as the phonemes to be corrected. As another example, it is possible to pre-determine the characters that are prone to errors based on common incorrect pronunciation problems in Chinese, such as polyphonic characters, easily confused characters, flat and retroflex characters, etc. Then, based on the semantics in this clause, determine the reference phonemes of the syllables corresponding to these preset characters in this clause, and then determine whether the reference phonemes are consistent with the phonemes corresponding to these preset characters in the phoneme conversion result. If they are not consistent, it is determined that there is an error in the phoneme conversion result, and correspondingly use the phonemes or phoneme combinations with errors in the phoneme conversion result as the phonemes to be corrected; for example, if this clause includes the preset character '卡车' (truck), through semantic analysis, the reference phoneme of the corresponding syllable can be determined as 'ka'. If the phoneme corresponding to this clause in the phoneme conversion result is 'qia', it is determined that there is an error, and 'qia' is used as the phoneme to be corrected; for another example, if this clause includes the preset character '料' (material), through semantic analysis, the reference phoneme of the corresponding syllable can be determined as 'liao'. If the phoneme corresponding to this clause in the phoneme conversion result is 'niao', it is determined that there is an error, and the phoneme 'n' is used as the phoneme to be corrected.

[0083] For example, samples can be collected based on the pronunciation rules of the target language and common mispronunciation problems to train the phoneme correction model, and the output results of the phoneme conversion model can be automatically detected by the trained phoneme correction model, wherein the phoneme correction model has the ability to detect whether there are errors in the phoneme conversion results of the clause and to locate the phonemes to be corrected; the training process of the phoneme correction model can be implemented using existing methods; in addition, the phonemes to be corrected in the phoneme conversion results of the clause can also be determined by manual judgment or manual assisted judgment, and there is no limitation on this.

[0084] Exemplarily, for any clause, after determining that there are errors in the phoneme conversion result of the clause and determining the phonemes to be corrected, the phonemes to be corrected can be converted into standard phonemes based on a preset corrected phoneme dictionary, thereby obtaining the final phoneme subsequence of the clause. The phoneme correction dictionary can be pre-set based on the pronunciation rules of the target language, the pronunciation habits of the people who use the target language, etc. As an example, the phoneme correction dictionary can include corresponding relationships related to pronunciation rules, such as "ja"-"jia", "qa"-"qia", "xa"-"xia", etc.; it can also include corresponding relationships related to flat and retroflex tongue, such as "zh"-"z", "sh"-"s", "z"-"zh", etc.; it can also include corresponding relationships related to polyphones, such as "xing"-"hang", "ka"-"qia", etc.; it can also include corresponding relationships related to accents, such as "n"-"l", etc.

[0085] Exemplarily, based on a preset phoneme correction dictionary, the phoneme to be corrected in the phoneme conversion result of the clause is converted into a standard phoneme, and a phoneme subsequence corresponding to the clause is obtained. It may further include: based on the preset phoneme correction dictionary, the phoneme to be corrected in the phoneme conversion result of the clause is converted into a standard phoneme, and when the tone of the syllable where the standard phoneme is located is different from the tone of the syllable where the phoneme to be corrected is located, the tone of the syllable where the phoneme to be corrected is located is replaced with the tone of the syllable where the standard phoneme is located; based on the standard phoneme and the converted tone, a phoneme subsequence corresponding to the clause is obtained. For example, if the clause includes the preset character "长", through semantic analysis, the phoneme of the corresponding syllable can be determined as "zhang", and the tone of this syllable is determined as "rising tone". If the corresponding phoneme in the phoneme conversion result of the clause is "chang", it is determined that there is an error, and "chang" is used as the phoneme to be corrected, and the tone corresponding to the syllable where this phoneme is located is "rising tone". Based on the corresponding relationship "chang"-"zhang" in the phoneme correction dictionary, the phoneme to be corrected "chang" is replaced with the standard phoneme "zhang", and at the same time, the tone "rising tone" of the syllable where the phoneme to be corrected is located is replaced with "rising tone", and a phoneme subsequence corresponding to the clause is obtained. The phoneme subsequence includes the phoneme "zhang" of the syllable corresponding to the character "长" and the tone "rising tone" of this syllable.

[0086] Step 103: Decouple the phonemes and tones in the first phoneme sequence, and extract the phoneme features of the text to be synthesized based on the decoupled second phoneme sequence; wherein, the second phoneme sequence includes the at least one phoneme.

[0087] Exemplarily, in the first phoneme sequence, all phonemes and all tones can be split to achieve the decoupling of phonemes and tones. The split phonemes form the second phoneme sequence, that is, the decoupled second phoneme sequence, and the second phoneme sequence is a phoneme sequence without tones. In this way, the phonemes and tones in the first phoneme sequence are decoupled, and the generated second phoneme sequence includes the phonemes of each syllable corresponding to the text to be synthesized and does not include the tones corresponding to each syllable. Furthermore, when extracting the phoneme features of the text to be synthesized based on the second phoneme sequence, the action space domain of the tone is isolated, and the phoneme features of the text to be synthesized can be extracted more accurately, thereby improving the quality of the fusion features obtained in the subsequent fusion of phoneme features and semantic features.

[0088] Among them, the phoneme feature is the feature of the phoneme and is used to represent pronunciation information. After obtaining the second phoneme sequence above, the phonemes in the second phoneme sequence can be embedded (Embedding) to extract the phoneme features of the text to be synthesized. As an example, models such as G2P can be used to process the second phoneme sequence to extract the phoneme features of the text to be synthesized.

[0089] In a possible implementation, a tone sequence can also be obtained based on the first phoneme sequence and the second phoneme sequence; the tone sequence includes the tone corresponding to the at least one phoneme; wherein the tone corresponding to any phoneme is determined by the tone of the syllable in which the phoneme is located.

[0090] Since the first phoneme sequence includes the phonemes of each syllable corresponding to the text to be synthesized and the tones corresponding to each syllable, and the second phoneme sequence includes the phonemes of each syllable, the first phoneme sequence and the second phoneme sequence contain the same phonemes, and the difference between the two is whether they contain tones. Therefore, based on the first phoneme sequence and the second phoneme sequence, the tones corresponding to each phoneme in the first phoneme sequence can be determined, and then a tone sequence can be formed; thereby achieving the independence of tones from the first phoneme sequence.

[0091] Exemplarily, for any syllable, the tone corresponding to the syllable can be used as the tone corresponding to each phoneme in the syllable, thereby ensuring that the dimension of the second phoneme sequence is the same as that of the tone sequence, that is, any phoneme in the second phoneme sequence corresponds to a tone in the tone sequence, so that the pronunciation of each phoneme can be subsequently controlled. For example, the first phoneme sequence is "xian4zai4", the second phoneme sequence is "xianzai", and the tone sequence is "44444".

[0092] Step 104: perform text encoding on the text to be synthesized in the syllable dimension to extract semantic features of the text to be synthesized.

[0093] Among them, semantic features can provide contextual information of the text to be synthesized, and can reflect the relationship between the corresponding syllables in the text to be synthesized, pause rhythm, etc. This information is crucial for predicting the duration of phonemes. For example, for the text to be synthesized "I go to the station", by extracting semantic features, we can know that "station" as a whole has fewer pauses between "car" and "station", while the pause between "go" and "car" is relatively long, that is, the duration of the syllable "chē" corresponding to "car" and the syllable "zhàn" corresponding to "station" is shorter than the duration of the syllable "qù" corresponding to "go"; in this way, after extracting the semantic features of the text to be synthesized, the duration of each phoneme can be predicted based on the semantic features.

[0094] For example, a semantic feature extraction model can be used to encode the text to be synthesized at the syllable level to extract the semantic features of the text to be synthesized. Semantic feature extraction models can be pre-trained for different languages. These trained semantic feature extraction models can learn to capture the semantic features of text in that language at the syllable level.

[0095] As an example, the semantic feature extraction model can be a BERT (Bidirectional Encoder Representations from Transformers) model, where BERT is a pre-trained model of the Transformer architecture. It exhibits extremely high performance by pre-training on large-scale unlabeled text and then fine-tuning on specific tasks. For example, the BGE-M3 (Bert Embedding with Multi-Linguality, Multi-Functionality, Multi-Granularity) model can be used: the BGE-M3 model is a multi-lingual, multi-functional, and multi-granular text embedding representation model based on BERT. For example, the output of the last layer of the BERT model can be used as the semantic feature of the text to be synthesized, thereby capturing the deep semantic information of the text to be synthesized.

[0096] In one possible implementation, the text to be synthesized is encoded in the syllable dimension to extract the semantic features of the text to be synthesized, including: segmenting the text to be synthesized according to syllables to obtain at least one syllable; determining the encoding corresponding to each syllable in the at least one syllable in combination with a preset vocabulary; wherein the preset vocabulary includes a correspondence between different syllables and different encodings; and extracting the semantic features of the text to be synthesized based on the encoding corresponding to each syllable in the at least one syllable.

[0097] Among them, the preset vocabulary corresponds to the target language, and the syllables in the vocabulary and the codes corresponding to each syllable can be pre-configured according to the characteristics of the target language; for example, based on the syllables in the vocabulary, word segmentation can be performed by matching, and the syllables corresponding to the text to be synthesized can be determined in turn; then, the corresponding codes in the vocabulary can be queried using the syllables obtained after the word segmentation process to determine the codes corresponding to each syllable, thereby realizing text encoding of the text to be synthesized in the syllable dimension; then, the codes corresponding to each syllable can be composed into a coding sequence and input into the semantic feature extraction model, so that the semantic features of the text to be synthesized can be extracted. The coding can be any coding form such as Chinese character coding, custom coding, etc. In this way, by performing word segmentation and coding on the text to be synthesized in the syllable dimension, more accurate and rich semantic features can be obtained, thereby providing a high-quality feature foundation for the subsequent deep fusion of phoneme features and semantic features.

[0098] Step 105: Fusing the semantic features and the phoneme features, and generating speech based on the fused features.

[0099] For example, a speech synthesis model can be used to fuse the semantic features and the phoneme features, and speech synthesis can be performed based on the fused features to generate speech. Speech synthesis models corresponding to different languages ​​can be pre-trained. For the target language corresponding to the text to be synthesized, a pre-trained speech synthesis model corresponding to that language is selected, and the semantic features and phoneme features extracted from the text to be synthesized are fused to output the synthesized speech.

[0100] As an example, a speech synthesis model can be based on the Variational Inference with adversarial learning for end-to-end Text-to-Speech (VITS) architecture. VITS is a speech synthesis architecture that combines a variational autoencoder (VAE) with a generative adversarial network (GAN). The VAE efficiently encodes and decodes audio features, mapping them to a low-dimensional latent space, which can then be restored to a high-quality speech signal through a decoder. This not only preserves the original characteristics of the sound but also introduces randomness, providing rich expressiveness for speech generation, making the synthesized speech more natural and diverse. During the training phase of the VITS model, the discriminator in the generative adversarial network can perform real-time quality assessment and optimization of the generated speech signal. By distinguishing the differences between the speech generated by the decoder in the variational autoencoder and the real speech, it provides feedback to the decoder in the variational autoencoder, thereby continuously optimizing the decoder's ability to generate speech. This process ensures that the synthesized speech achieves the expected results in terms of naturalness, rhythm, and semantic consistency.

[0101] Figure 2 A structural diagram of a VITS model according to an embodiment of the present disclosure is shown as follows: Figure 2As shown in FIG, the trained speech synthesis model may include: an encoder (Text Encoder), a mapping layer (Projection), a stochastic duration predictor (Stochastic Duration Predictor), a flow model (Flow), and a decoder (Decoder); the VITS model can be used to fuse the above-mentioned refined phoneme features and semantic features and convert them into high-quality speech; wherein the encoder is used to fuse the phoneme features and semantic features to generate fused features (also called intermediate features or audio features), for example, the phoneme features corresponding to each phoneme and the semantic features corresponding to each phoneme can be weighted summed, concatenated or multiplied to generate the fused features corresponding to each phoneme; the mapping layer is used to map the fused features to the latent space; the stochastic duration predictor is used to predict the pronunciation duration of each phoneme corresponding to the text to be synthesized based on the fused features, and map it to the latent space, and through the monotonic alignment model (Monotonic Alignment The VITS model performs a latent search (Search) in the latent space, aligning the fused features with the duration of each phoneme's pronunciation. The stream model generates latent variables based on this alignment, and the decoder decodes these latent variables to generate speech corresponding to the text to be synthesized, i.e., the synthesized speech. In this way, the trained VITS model can generate high-quality speech that sounds natural and matches the text to be synthesized. The training process of the speech synthesis model can be implemented using existing technologies and is not limited thereto.

[0102] For example, the decoder in VITS can be a vocoder based on the High-Fidelity GAN (HiFiGAN) model. HiFiGAN is a powerful neural network vocoder that can generate high-fidelity speech waveforms while maintaining a high generation speed. For example, the encoder in VITS can also be configured with the aforementioned semantic feature extraction model and phoneme conversion model. In this way, the text to be synthesized can be input into the VITS encoder, which extracts the semantic and phoneme features of the text to be synthesized and performs feature fusion.

[0103] Exemplarily, when the semantic features and phoneme features of the text to be synthesized are fused, the dimensions of the semantic features and the phoneme features can be aligned first, and then the aligned semantic features and phoneme features can be fused to generate the fused features. As an example, the dimensional alignment of the semantic features and the phoneme features can be performed by greedy repeat, which distributes the semantic features evenly to each phoneme according to the dimension and the order of the phonemes in the second phoneme sequence. For the semantic features of the remaining dimensions, the semantic features of the remaining dimensions are still assigned to the phonemes with the highest order according to the order of the phonemes. For example, if the dimension of the semantic feature is 7 and the number of phonemes is 3, the dimension of the semantic feature corresponding to each phoneme is 2, and then the semantic feature of the remaining dimension is assigned to the first phoneme. In this way, the phoneme features and semantic features of the text to be synthesized are accurately aligned, and the deep fusion of the semantic features and the phoneme features is achieved, thereby ensuring that the pronunciation and rhythm corresponding to each phoneme in the generated speech are more accurate.

[0104] In one possible implementation, the method further includes: in the process of generating speech based on the fused features, predicting the duration of each phoneme in the at least one phoneme based on a duration model; wherein the duration model is trained based on the real alignment label of the phoneme-duration.

[0105] The duration model can be a DurationNet model, i.e. a DurationNet model based on the ground truth. The real alignment label of phoneme-duration is used to indicate the correspondence between each phoneme in the training sentence and the real duration of each phoneme in the training sentence. For example, the above Figure 2 The random duration predictor in the VITS model in

[15] is replaced by the DurationNet model based on GroundTruth, and during the training process, training sentences with real alignment labels including phonemes and phoneme-duration are used for training.

[0106] In this way, the trained DurationNet model can better synthesize the prosody of speech. The DurationNet model is trained using existing methods, which are not limited to this.

[0107] For example, taking Vietnamese as the target language, the DurationNet model can include: a bidirectional gated recurrent unit (GRU), a fully connected layer, and a softplus activation function. Compared with structures such as convolution, GRU has significant advantages in processing sequence data, especially in capturing temporal dependencies. The bidirectional GRU can capture the front-to-back dependencies of the fused features, while using forward and reverse context information to generate high-dimensional temporal features. The fully connected layer performs a nonlinear transformation on the high-dimensional temporal features output by the bidirectional GRU, completing the dimensional mapping from temporal features to duration. The softplus activation function ensures that the predicted duration is a positive number. In this way, the features fused from semantic features and phoneme features are input into the DurationNet model, which can accurately predict the duration of each phoneme in the second phoneme sequence, thereby further improving the quality of speech synthesis.

[0108] Furthermore, considering scenarios such as dubbing for film and television dramas, there may be requirements for the duration of the generated speech. For example, for a single actor's multiple consecutive lines of dialogue, the duration of the generated speech needs to be consistent with the duration of the corresponding screen image to maintain synchronization between the audio and the image. However, due to differences in different languages, the length of the synthesized speech corresponding to the same line may vary. Therefore, it is necessary to perform splitting and merging in the generated speech to ensure that the speech length corresponding to the same line in different languages ​​is the same. Merging means removing pauses between adjacent clauses to merge them into a single clause, while splitting means adding pauses within a clause to split it into multiple clauses.

[0109] In one possible implementation, the method further includes: obtaining a preset duration corresponding to each clause in the multiple clauses; when the duration of the generated speech exceeds the sum of the preset durations of the multiple clauses, performing a merging process on at least two adjacent clauses in the speech, and after the merging process, adding one or more spaces between the at least two adjacent clauses so that the duration of the processed speech is no greater than the sum of the preset durations of the multiple clauses.

[0110] Exemplarily, after the merging process, one or more spaces can be added between at least two adjacent clauses by means of blank coding, wherein the duration of each space can be set as needed, and the duration of the one or more spaces added between two adjacent clauses is less than the pause duration of the two adjacent clauses before the merging; that is, relative to the pause duration between the two adjacent clauses before the merging, the pause between the two adjacent clauses after adding one or more spaces becomes shorter, thereby shortening the overall duration of the two adjacent clauses through the merging, strengthening the sense of pause between the two adjacent clauses participating in the merging, and ensuring that the duration of the final processed speech is no greater than the sum of the preset durations of the multiple clauses.

[0111] As an example, after the axis-matching process, one or more spaces may be added between the at least two adjacent clauses so that the duration of the processed speech is the same as the sum of the preset durations of the multiple clauses.

[0112] In this way, when the duration of the generated synthesized speech exceeds the sum of the preset durations of multiple clauses, at least two adjacent clauses in the speech are subjected to axis processing, thereby shortening the duration of the synthesized speech and avoiding problems such as asynchrony between the sound and the picture caused by the inconsistency between the duration of the synthesized speech and the preset duration in scenes such as dubbing of film and television dramas. The preset duration can be determined by the duration of the pictures corresponding to the multiple clauses. In addition, considering that the pauses between the clauses participating in the axis are removed during the axis processing, resulting in a weakened sense of pause in the synthesized speech, therefore, after the axis processing is performed, one or more spaces are added between the two adjacent clauses participating in the axis, thereby strengthening the sense of pause in the speech synthesis, which can not only add a more distinct sense of rhythm to the film and television plot, but also strengthen the transmission of emotions and the presentation of scenes.

[0113] In a possible implementation, the method further includes: in the process of generating speech based on the fused features, controlling the pitch of the speech through the tone sequence. Since the phoneme sequence includes: the pitch corresponding to each phoneme of the syllable corresponding to the text to be synthesized; in this way, the pitch of the speech is controlled by the tone sequence, so that the pronunciation and rhythm of each phoneme are sufficiently accurate. In addition, when generating the fused features, the phoneme features are extracted based on the second phoneme sequence after the phonemes and tones in the first phoneme sequence are decoupled, and then the phoneme features are fused with the semantic features of the text to be synthesized to obtain the fused features, thereby isolating the spatial domain of the tone; at the same time, the tone of the synthesized speech is controlled by the tone sequence independently separated from the phoneme sequence, thereby achieving independent control of the tone and phonemes, and ensuring the quality of the synthesized speech.

[0114] For example, during speech synthesis, pitch is typically expressed as a fundamental frequency (F0) curve. For example, in Chinese, different pitches correspond to different fundamental frequency curves. Thus, a pitch sequence can be used to determine the fundamental frequency of the audio segment corresponding to each phoneme during speech synthesis, thereby controlling the pitch of the generated speech. For example, when using the VITS model for speech synthesis, the pitch sequence can be used to control the fundamental frequency of the audio segment corresponding to each phoneme in the decoder to achieve pitch control of the synthesized speech.

[0115] In an embodiment of the present disclosure, a text to be synthesized is obtained; phoneme conversion is performed on the text to be synthesized to obtain a first phoneme sequence, wherein the first phoneme sequence includes at least one phoneme and the tone of the syllable in which the at least one phoneme is located; the phonemes and the tones in the first phoneme sequence are decoupled, and the phoneme features of the text to be synthesized are extracted based on the decoupled second phoneme sequence; wherein the second phoneme sequence includes the at least one phoneme; text encoding is performed on the text to be synthesized in the syllable dimension, and semantic features of the text to be synthesized are extracted; the semantic features and the phoneme features are fused, and speech is generated based on the fused features. In this way, the phonemes and tones in the first phoneme sequence are decoupled, thereby isolating the spatial domain of the tone, and then the phoneme features of the text to be synthesized are extracted based on the decoupled second phoneme sequence, thereby improving the accuracy of the extracted phoneme features; at the same time, the text to be synthesized is encoded in the syllable dimension, which can obtain more accurate and rich semantic features; then, the phoneme features extracted from the decoupled second phoneme sequence are fused with the semantic features, thereby ensuring the quality of the fused features, and the speech synthesized based on the fused features is not only accurately pronounced, but also has a natural rhythm, ensuring that the synthesized speech achieves the expected effect in terms of naturalness, rhythm and semantic consistency.

[0116] As an example, since the text to be synthesized is normalized and a phoneme correction dictionary is applied during speech synthesis, the content of the text to be synthesized can be standardized, thereby generating a standardized and correct phoneme sequence. This facilitates the speech synthesis model to generate more accurate synthesized speech based on the standardized phoneme sequence, reducing the occurrence of missing words, omissions, and pronunciation errors in the synthesized speech, and improving the accuracy of speech synthesis. In addition, for target languages ​​such as minority languages, the speech generated by the speech synthesis method in the disclosed embodiment is no different from real people in terms of pronunciation accuracy and rhythm, as measured by people whose native language is the target speech; it can be effectively applied to scenarios such as dubbing for international film and television dramas.

[0117] For example, the above speech synthesis method can be implemented by using a trained speech synthesis model. For example, taking the speech synthesis model as a VITS-based model, the above Figure 2The VITS encoder shown in the figure is configured with the trained BERT model and G2P model.

[0118] Figure 3 A schematic diagram of extracting semantic features and phoneme features according to an embodiment of the present disclosure is shown as follows: Figure 3 As shown in the figure, the text to be synthesized can be input into the BERT model, and the BERT model can extract the semantic features of the text to be synthesized; the text to be synthesized can be input into the G2P model, and the G2P model can extract the phoneme features of the text to be synthesized; and then the semantic features are fused with the phoneme features to obtain audio features, that is, the fused features.

[0119] Figure 4 A flow chart of speech synthesis using a speech synthesis model according to an embodiment of the present disclosure is shown as follows: Figure 4 As shown, first input the text to be synthesized into the VITS model configured with the BERT model and the G2P model (corresponding to Figure 1 In step 101), the G2P model in the VITS model performs phoneme conversion on the synthesized text to obtain a phoneme sequence (corresponding to Figure 1 Step 102), then decouple the phonemes and tones in the phoneme sequence, and extract the phoneme features (corresponding to Figure 1 Step 103); BERT model encodes the synthesized text and extracts semantic features (corresponding to Figure 1 Finally, the phoneme features and semantic features are fused and then speech synthesis is performed, so that high-quality, natural and fluent speech can be synthesized to achieve a highly expressive speech synthesis effect (corresponding to Figure 1 (Step 105).

[0120] Based on the same inventive concept of the above method embodiment, an embodiment of the present disclosure further provides a speech synthesis device, which can be used to execute the technical solution described in the above method embodiment.

[0121] Figure 5 A structural diagram of a speech synthesis device according to an embodiment of the present disclosure is shown as follows: Figure 5As shown, the device includes: an acquisition module 501, used to acquire a text to be synthesized; a phoneme conversion module 502, used to perform phoneme conversion on the text to be synthesized to obtain a first phoneme sequence, wherein the first phoneme sequence includes at least one phoneme and the tone of the syllable in which the at least one phoneme is located; a phoneme feature extraction module 503, used to decouple the phonemes and tones in the first phoneme sequence, and extract the phoneme features of the text to be synthesized based on the decoupled second phoneme sequence; wherein the second phoneme sequence includes the at least one phoneme; a semantic feature extraction module 504, used to perform text encoding on the text to be synthesized in the syllable dimension, and extract the semantic features of the text to be synthesized; a speech synthesis module 505, used to fuse the semantic features and the phoneme features, and generate speech based on the fused features.

[0122] In an embodiment of the present disclosure, a text to be synthesized is obtained; phoneme conversion is performed on the text to be synthesized to obtain a first phoneme sequence, wherein the first phoneme sequence includes at least one phoneme and the tone of the syllable in which the at least one phoneme is located; the phonemes and the tones in the first phoneme sequence are decoupled, and the phoneme features of the text to be synthesized are extracted based on the decoupled second phoneme sequence; wherein the second phoneme sequence includes the at least one phoneme; text encoding is performed on the text to be synthesized in the syllable dimension, and semantic features of the text to be synthesized are extracted; the semantic features and the phoneme features are fused, and speech is generated based on the fused features. In this way, the phonemes and tones in the first phoneme sequence are decoupled, thereby isolating the spatial domain of the tone, and then extracting the phoneme features of the text to be synthesized based on the decoupled second phoneme sequence, thereby improving the accuracy of the extracted phoneme features; at the same time, the text to be synthesized is encoded in the syllable dimension, which can obtain more accurate and rich semantic features; then, the phoneme features extracted from the decoupled second phoneme sequence are fused with the semantic features, thereby ensuring the quality of the fused features, so that the final synthesized speech achieves the expected effect in terms of naturalness, rhythm and semantic consistency.

[0123] In one possible implementation, the phoneme conversion module 502 is further configured to: filter characters in a non-target language in the text to be synthesized based on the target language corresponding to the text to be synthesized to obtain a filtered text; segment the filtered text according to preset pause symbols corresponding to the target language to obtain a plurality of clauses; wherein the preset pause symbols are used to indicate pauses in sentences; perform phoneme conversion on each of the plurality of clauses to obtain a phoneme subsequence corresponding to each clause; and merge the phoneme subsequences corresponding to each clause to obtain the first phoneme sequence.

[0124] In one possible implementation, the speech synthesis module 505 is further used to: obtain a tone sequence based on the first phoneme sequence and the second phoneme sequence; the tone sequence includes the tone corresponding to the at least one phoneme; wherein the tone corresponding to any phoneme is determined by the tone of the syllable in which the phoneme is located; and in the process of generating speech based on the fused features, the tone of the speech is controlled by the tone sequence.

[0125] In one possible implementation, the semantic feature extraction module 504 is further used to: segment the text to be synthesized according to syllables to obtain at least one syllable; determine the encoding corresponding to each syllable in the at least one syllable in combination with a preset vocabulary; wherein the preset vocabulary includes a correspondence between different syllables and different encodings; and extract the semantic features of the text to be synthesized based on the encoding corresponding to each syllable in the at least one syllable.

[0126] In one possible implementation, the phoneme conversion module 502 is further used to: determine, for any clause among the multiple clauses, whether the clause contains preset characters; and perform normalization processing if the clause contains preset characters; and perform phoneme conversion processing on the normalized clause to obtain a phoneme subsequence corresponding to the clause.

[0127] In one possible implementation, the phoneme conversion module 502 is further used to: perform phoneme conversion processing on any clause among the multiple clauses using a phoneme conversion model to obtain a phoneme conversion result of the clause; determine the phonemes to be corrected in the phoneme conversion result of the clause; and convert the phonemes to be corrected in the phoneme conversion result of the clause into standard phonemes based on a preset phoneme correction dictionary to obtain a phoneme subsequence corresponding to the clause; wherein the phoneme correction dictionary includes: a correspondence between different phonemes to be corrected and different standard phonemes.

[0128] In one possible implementation, the speech synthesis module 505 is further used to: in the process of generating speech based on the fused features, predict the duration of each phoneme in the at least one phoneme based on a duration model; wherein the duration model is trained based on the real alignment label of the phoneme-duration.

[0129] In one possible implementation, the speech synthesis module 505 is further used to: obtain a preset duration corresponding to each clause in the multiple clauses; when the duration of the generated speech exceeds the sum of the preset durations of the multiple clauses, perform a merging process on at least two adjacent clauses in the speech, and after the merging process, add one or more spaces between the at least two adjacent clauses so that the duration of the processed speech is no greater than the sum of the preset durations of the multiple clauses.

[0130] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0131] An embodiment of the present disclosure further provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0132] An embodiment of the present disclosure further provides a non-volatile computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0133] An embodiment of the present disclosure further provides a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0134] Figure 6 FIG1 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Figure 6 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0135] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0136] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0137] A computer-readable storage medium can be a tangible device that can hold and store programs / instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0138] The computer programs (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0139] The computer program (or computer program instructions) for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, by utilizing state information of computer-readable program instructions to personalize and customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0140] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0141] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0142] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0143] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0144] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Get the text to be synthesized; Performing phoneme conversion on the text to be synthesized to obtain a first phoneme sequence, wherein the first phoneme sequence includes at least one phoneme and a tone of a syllable in which the at least one phoneme is located; Decoupling the phonemes and tones in the first phoneme sequence, and extracting phoneme features of the text to be synthesized based on the decoupled second phoneme sequence; wherein the second phoneme sequence includes the at least one phoneme; Performing text encoding on the text to be synthesized in the syllable dimension to extract semantic features of the text to be synthesized; The semantic features and the phoneme features are fused, and speech is generated based on the fused features.

2. The method according to claim 1, characterized in that The performing phoneme conversion on the text to be synthesized to obtain a first phoneme sequence includes: Based on the target language corresponding to the text to be synthesized, filtering characters in a non-target language in the text to be synthesized to obtain a filtered text; Segmenting the filtered text according to preset pause symbols corresponding to the target language to obtain multiple clauses; wherein the preset pause symbols are used to indicate pauses in sentences; Performing phoneme conversion processing on each of the multiple clauses to obtain a phoneme subsequence corresponding to each clause; The phoneme subsequences corresponding to each of the clauses are merged to obtain the first phoneme sequence.

3. The method according to claim 1, characterized in that The method further comprises: Based on the first phoneme sequence and the second phoneme sequence, a tone sequence is obtained; the tone sequence includes a tone corresponding to the at least one phoneme; wherein the tone corresponding to any phoneme is determined by the tone of the syllable in which the phoneme is located; In the process of generating speech based on the fused features, the pitch of the speech is controlled by the pitch sequence.

4. The method according to claim 1, wherein The step of encoding the text to be synthesized in the syllable dimension and extracting semantic features of the text to be synthesized includes: Performing word segmentation on the text to be synthesized according to syllables to obtain at least one syllable; Determining a code corresponding to each syllable in the at least one syllable in combination with a preset vocabulary, wherein the preset vocabulary includes a correspondence between different syllables and different codes; The semantic features of the text to be synthesized are extracted based on the code corresponding to each syllable in the at least one syllable.

5. The method according to claim 2, characterized in that The performing phoneme conversion processing on each of the plurality of clauses to obtain a phoneme subsequence corresponding to each clause includes: For any clause among the multiple clauses, determining whether the clause contains a preset character; and performing normalization processing if the clause contains the preset character; The normalized clause is subjected to phoneme conversion to obtain a phoneme subsequence corresponding to the clause.

6. The method according to claim 2, characterized in that The performing phoneme conversion processing on each of the plurality of clauses to obtain a phoneme subsequence corresponding to each clause includes: For any one of the multiple clauses, perform phoneme conversion processing using a phoneme conversion model to obtain a phoneme conversion result of the clause; determining the phoneme to be corrected in the phoneme conversion result of the clause; Based on a preset phoneme correction dictionary, the phonemes to be corrected in the phoneme conversion result of the clause are converted into standard phonemes to obtain a phoneme subsequence corresponding to the clause; wherein the phoneme correction dictionary includes: corresponding relationships between different phonemes to be corrected and different standard phonemes.

7. The method according to claim 1, characterized in that The method further comprises: In the process of generating speech based on the fused features, the duration of each phoneme in the at least one phoneme is predicted based on a duration model; wherein the duration model is trained based on real alignment labels of phoneme-duration.

8. The method according to claim 2, characterized in that The method further comprises: Obtaining a preset duration corresponding to each of the multiple clauses; When the duration of the generated speech exceeds the sum of the preset durations of the multiple clauses, at least two adjacent clauses in the speech are subjected to axis merging processing. After the axis merging processing, one or more spaces are added between the at least two adjacent clauses so that the duration of the processed speech is no greater than the sum of the preset durations of the multiple clauses; wherein axis merging means removing the pauses between adjacent clauses to merge them into one clause.

9. A speech synthesis device, characterized in that: The device comprises: An acquisition module is used to obtain the text to be synthesized; a phoneme conversion module, configured to perform phoneme conversion on the text to be synthesized to obtain a first phoneme sequence, wherein the first phoneme sequence includes at least one phoneme and a tone of a syllable in which the at least one phoneme is located; a phoneme feature extraction module, configured to decouple the phonemes and tones in the first phoneme sequence, and extract the phoneme features of the text to be synthesized based on a second phoneme sequence after the decoupling; wherein the second phoneme sequence includes the at least one phoneme; A semantic feature extraction module, configured to perform text encoding on the text to be synthesized in a syllable dimension and extract semantic features of the text to be synthesized; The speech synthesis module is used to fuse the semantic features and the phoneme features and generate speech based on the fused features.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

11. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images