Speech synthesis method and device, electronic equipment and storage medium

Abstract

This invention provides a speech synthesis method, apparatus, electronic device, and storage medium. The method includes: acquiring target text; performing prosodic level prediction on the target text to obtain prosodic boundary information of the target text; determining each prosodic word in the target text based on the prosodic word boundaries in the prosodic boundary information; acquiring the dialectal phonology corresponding to the target text, and determining the dialectal tone sandhi pattern corresponding to each prosodic word based on the dialectal phonology, wherein the dialectal tone sandhi pattern is used to indicate the tone sandhi boundary in each prosodic word; and synthesizing the dialectal speech corresponding to the target text based on the prosodic boundary information and the dialectal tone sandhi pattern. The speech synthesis method, apparatus, electronic device, and storage medium provided by this invention can improve the accuracy and intelligibility of speech synthesis.

Description

Technical Field

[0001] The present invention relates to the technical field of speech processing, and in particular, to a speech synthesis method, device, electronic device and storage medium. Background Art

[0002] Speech synthesis, also known as text-to-speech (TTS) technology, can convert any text information into standard and fluent speech in real time, which is equivalent to installing an artificial mouth on a machine. Among them, dialect speech synthesis belongs to a special speech synthesis technology.

[0003] Currently, in a dialect speech synthesis system, dialect text is mainly used as input for speech synthesis. However, since most southern dialects, such as Shanghainese and Minnan dialect, lack a unified writing system, it is difficult to popularize the dialect speech synthesis system. And there are certain differences between Mandarin text and dialect text, which greatly reduces the intelligibility and naturalness of the dialect speech synthesized using Mandarin text. Therefore, in a dialect speech synthesis system with Mandarin text as input, prosodic naturalness and pronunciation accuracy have become two major bottlenecks. Based on this, in order to improve the accuracy of dialect speech synthesis, in the existing dialect speech synthesis system, Mandarin prosodic words are usually used to predict the boundaries of dialect connected tone sandhi, and a boundary prediction model of its connected tone sandhi is constructed through a word segmentation algorithm, so as to obtain more accurate dialect speech.

[0004] However, defining the Mandarin prosodic word boundary and the dialect connected tone sandhi boundary as the same semantic unit will cause incorrect judgment of the connected tone sandhi boundary of some special words. For example, for the common Mandarin prosodic word "我是" (I am), the connected tone sandhi boundary of its corresponding Shanghainese text "吾是" is "吾 / 是", while in the current technical solution, the two characters "我是" will be divided into the same connected tone sandhi unit, resulting in incorrect tones in the front-end text, and further reducing the accuracy and intelligibility of the dialect speech synthesis. Summary of the Invention

[0005] The present invention provides a speech synthesis method, device, electronic device and storage medium, which are used to solve the defect of low accuracy of dialect speech synthesis in the prior art, and achieve the purpose of improving the accuracy and intelligibility of dialect speech synthesis.

[0006] The present invention provides a speech synthesis method, including:

[0007] Obtain a target text;

[0008] Perform prosodic level prediction on the target text to obtain prosodic boundary information of the target text;

[0009] Based on the prosodic word boundaries in the prosodic boundary information, each prosodic word in the target text is determined;

[0010] Obtain the dialect phonology and tone corresponding to the target text, and determine the dialect tone sandhi pattern corresponding to each prosodic word based on the dialect phonology and tone. The dialect tone sandhi pattern is used to indicate the tone sandhi boundary in each prosodic word.

[0011] Based on the prosodic boundary information and the dialectal tone sandhi pattern, the dialectal speech corresponding to the target text is synthesized.

[0012] According to a speech synthesis method provided by the present invention, the step of performing prosodic level prediction on the target text to obtain the prosodic boundary information of the target text includes:

[0013] The target text is input into the encoding layer of the prosody prediction model to obtain the first feature information of each character in the target text;

[0014] The first feature information of each character is input into the decoding layer of the prosody prediction model to obtain the first probability that the boundary of each character is a prosodic boundary;

[0015] Based on a first probability that the boundary of each character is a prosodic boundary, the prosodic boundary information of the target text is determined.

[0016] According to a speech synthesis method provided by the present invention, determining the dialect tone sandhi pattern corresponding to each of the prosodic words based on the dialect phonology includes:

[0017] Based on the dialect sounds, rhymes, and tones, determine the number of dialect syllables corresponding to each of the prosodic words;

[0018] Based on the dialect pronunciation and tone, the first feature information of each character in the target text, the number of dialect syllables corresponding to each prosodic word, and the number of Mandarin syllables corresponding to each prosodic word, the dialect tone sandhi pattern of each prosodic word is determined.

[0019] According to a speech synthesis method provided by the present invention, the step of determining the dialect tone sandhi pattern of each prosodic word based on the dialect initials and finals, the first feature information of each character in the target text, the number of dialect syllables corresponding to each prosodic word, and the number of Mandarin syllables corresponding to each prosodic word includes:

[0020] The first feature information of each character and the number of Mandarin syllables corresponding to each prosodic word are input into the encoding layer of the tone sandhi prediction model to obtain the semantic feature information of each prosodic word.

[0021] The dialect phonology and tone are input into the coding layer to obtain the second feature information corresponding to each character in the dialect. Based on each second feature information and the number of dialect syllables corresponding to each prosodic word, the dialect phonology and tone feature information of each prosodic word is determined.

[0022] Based on the semantic feature information of each prosodic word and the corresponding dialectal phonological and tonal feature information, the target feature information is determined;

[0023] The target feature information is input into the decoding layer of the tone sandhi prediction model to obtain the second probability of each prosodic word as a tone sandhi pattern.

[0024] Based on the second probability of each prosodic word as a tone sandhi pattern, the dialect tone sandhi pattern of each prosodic word is determined.

[0025] According to a speech synthesis method provided by the present invention, the prosody prediction model is trained based on the following method:

[0026] Obtain at least two text samples, each of which includes prosodic level annotation information;

[0027] Based on the prosodic level annotation information, determine the prosodic boundary label;

[0028] Each of the text samples is input into the initial prosodic prediction model to obtain the predicted prosodic boundary corresponding to each of the text samples;

[0029] Based on the predicted prosodic boundary and the prosodic boundary label, the initial prosodic prediction model is iteratively trained to obtain the prosodic prediction model.

[0030] According to a speech synthesis method provided by the present invention, the tone sandhi prediction model is trained based on the following method:

[0031] Obtain the dialect audio corresponding to each of the text samples, wherein the dialect audio includes the pronunciation, rhyme and tone annotation information of the dialect;

[0032] For the phonological and tone annotation information corresponding to the audio of each dialect, the tone sandhi pattern label of each prosodic word sample is determined based on the phonological and tone annotation information. The tone sandhi pattern label is used to indicate the tone sandhi boundary in each prosodic word sample.

[0033] For each text sample, obtain the third feature information of each character in the text sample;

[0034] The third feature information of each character, the number of dialect syllables corresponding to each prosodic word sample in the text sample, the number of Mandarin syllables corresponding to each prosodic word sample in the text sample, and the initial tone and rhyme annotation information of the dialect corresponding to the text sample are input into the initial tone sandhi prediction model to obtain the predicted tone sandhi pattern of each prosodic word sample in the text sample.

[0035] Based on the predicted tone shift pattern and the tone shift pattern label, the initial tone shift pattern prediction model is iteratively trained to obtain the tone shift pattern prediction model.

[0036] According to a speech synthesis method provided by the present invention, determining the tone sandhi pattern label of each prosodic word sample based on the tone annotation information includes:

[0037] Based on the phonological and tone annotation information, the tone sandhi boundaries of the corresponding dialect are determined, as well as the number of dialect syllables corresponding to each prosodic word sample in the corresponding text sample.

[0038] Based on the tone sandhi boundaries and the number of dialect syllables corresponding to each prosodic word sample, the tone sandhi pattern label of each prosodic word sample is determined.

[0039] The present invention also provides a speech synthesis device, comprising:

[0040] The acquisition module is used to acquire the target text;

[0041] The prediction module is used to perform prosodic level prediction on the target text to obtain the prosodic boundary information of the target text;

[0042] The determination module is used to determine each prosodic word in the target text based on the prosodic word boundaries in the prosodic boundary information;

[0043] The acquisition module is also used to acquire the dialectal sounds, rhymes, and tones corresponding to the target text;

[0044] The determining module is further configured to determine the dialect tone sandhi pattern corresponding to each of the prosodic words based on the dialect phonology and tone, wherein the dialect tone sandhi pattern is used to indicate the tone sandhi boundary in each of the prosodic words.

[0045] The synthesis module is used to synthesize the dialect speech corresponding to the target text based on the prosodic boundary information and the dialect tone sandhi pattern.

[0046] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech synthesis methods.

[0047] The present invention also provides an electronic device, including a speaker, a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to: acquire target text; perform prosodic level prediction on the target text to obtain prosodic boundary information of the target text; determine each prosodic word in the target text based on the prosodic word boundaries in the prosodic boundary information; acquire the dialectal phonology corresponding to the target text, and determine the dialectal tone sandhi pattern corresponding to each prosodic word based on the dialectal phonology, wherein the dialectal tone sandhi pattern is used to indicate the tone sandhi boundary in each prosodic word; and synthesize the dialectal speech corresponding to the target text based on the prosodic boundary information and the dialectal tone sandhi pattern.

[0048] The speaker is used to play the dialect speech.

[0049] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech synthesis method as described above.

[0050] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described speech synthesis methods.

[0051] The speech synthesis method, apparatus, electronic device, and storage medium provided by this invention predict the prosodic level of the acquired target text to obtain the prosodic boundary information of the target text, and determine each prosodic word in the target text based on the prosodic word boundaries in the prosodic boundary information. Then, the dialectal phonology corresponding to the target text is obtained, and the dialectal tone sandhi pattern corresponding to each prosodic word is determined based on the dialectal phonology, and the dialectal tone sandhi pattern is synthesized based on the prosodic boundary information and the dialectal tone sandhi pattern. The dialectal tone sandhi pattern is used to indicate the tone sandhi boundary in each prosodic word. Therefore, this embodiment of the invention uses a single prosodic word in the target text as a unit, and determines the dialectal tone sandhi pattern corresponding to each prosodic word through the dialectal phonology, thereby avoiding the problem of incorrect tone sandhi boundaries for some special words caused by defining the Mandarin prosodic word boundary and the dialectal tone sandhi boundary as the same semantic unit. This improves the accuracy of dialectal tone sandhi corresponding to the target text, and thus improves the accuracy and intelligibility of dialect speech synthesis. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is one of the flowcharts of the speech synthesis method provided in the embodiments of the present invention;

[0054] Figure 2 This is a second schematic flowchart of the speech synthesis method provided in this embodiment of the invention;

[0055] Figure 3 This is a schematic diagram of the structure of the prosody prediction model provided in an embodiment of the present invention;

[0056] Figure 4 This is the third flowchart illustrating the speech synthesis method provided in this embodiment of the invention;

[0057] Figure 5 This is a schematic diagram of the structure of the continuous reading tone shifting pattern prediction model provided in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the speech synthesis device provided in an embodiment of the present invention;

[0059] Figure 7 This is one of the structural schematic diagrams of the electronic device provided by the present invention;

[0060] Figure 8 This is the second schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0062] With the continuous development of technology, users have increasingly higher requirements for the quality of speech synthesis. Among these requirements, tone sandhi is one of the important factors affecting the quality of speech synthesis. Tone sandhi refers to the change in tone when two or more syllables are influenced by the preceding and following syllables in a dialectal speech. This phenomenon is more common in southern Chinese dialects, such as Shanghainese and Minnan, and it has a very important impact on the naturalness and intelligibility of the synthesized dialect speech.

[0063] Specifically, when predicting the boundary of connected speech tone sandhi for Mandarin text, it is necessary to first construct a prosodic word segmentation model. That is, first collect a large number of Mandarin texts, obtain the prosodic word boundaries of the Mandarin texts through manual annotation, and use the manually annotated prosodic word boundaries as the annotation data of the Mandarin texts. When it is determined that the annotated prosodic word has no ambiguity, include the prosodic word in the frequency of the word in the pronunciation dictionary. Then, through the single-character dictionary of the dialect, find out the single-character pronunciation of the prosodic word, and perform tone transformation according to the known tone sandhi rules, so as to include the prosodic word in the pronunciation dictionary. Based on this, using the entries and word frequency information in the constructed pronunciation dictionary, construct a directed acyclic graph with word frequency as the edge weight, that is, the prosodic word segmentation model of the simple algorithm model.

[0064] Thus, on the basis of constructing the prosodic word segmentation model, use greedy algorithms such as the shortest path algorithm to predict the prosodic word boundaries of the input Mandarin text, and mark the intonation phrase labels at special characters such as punctuation marks in the input Mandarin text according to the rules, so as to obtain the prosodic word boundaries (which are also the boundaries of connected speech tone sandhi) and intonation phrase boundaries of the Mandarin text. It can be seen that in this prior art, it is necessary to manually annotate a large number of Mandarin texts to extract word frequency information from the annotation data to construct a prosodic word segmentation model, which takes a long time and is costly. In addition, the word segmentation algorithm used in this prior art is a shallow learning model, and its learning ability is relatively limited, and it is impossible to obtain a prosodic word segmentation model with high accuracy through the training of a large number of text samples.

[0065] In addition, the method of predicting the boundary of dialect connected speech tone sandhi by using the boundary of Mandarin prosodic words in this prior art cannot cover some special case words that do not conform to the tone sandhi rules. For example, the Shanghai dialect corresponding to the Mandarin prosodic word "很好" is "交关好", and its boundary of connected speech tone sandhi is "交关 / 好". Here, " / " represents the boundary of connected speech tone sandhi, so the connected speech tone sandhi pattern corresponding to the Mandarin prosodic word "很好" is "2+1", and this connected speech tone sandhi pattern indicates the number of dialect syllables included in each connected speech tone sandhi unit. However, in the current technical solution, the two characters of "很好" will be divided into the same connected speech tone sandhi unit, that is, the boundary of dialect connected speech tone sandhi is "交关好 / ", resulting in the connected speech tone sandhi pattern becoming "3", that is, misjudging its boundary of connected speech tone sandhi, thus greatly reducing the accuracy of the synthesized dialect speech, making the intelligibility not high, and the synthesized dialect speech is not natural enough.

[0066] Based on this, this invention proposes a speech synthesis method. In this method, each prosodic word in the target text is used as a unit. By using the dialectal phonology and tone corresponding to the target text, the dialectal tone sandhi pattern corresponding to each prosodic word is determined. In other words, the boundaries of prosodic words and dialectal tone sandhi are distinguished. This avoids the problem of incorrect tone sandhi boundaries for some special words caused by defining the boundaries of Mandarin prosodic words and dialectal tone sandhi as the same semantic unit. This improves the accuracy of dialectal tone sandhi corresponding to the target text, thereby improving the accuracy of dialect speech synthesis and enhancing the intelligibility and naturalness of dialect speech.

[0067] The following is combined Figures 1-5 The speech synthesis method provided in this embodiment of the invention is described below. This method can be applied to speech synthesis scenarios, especially dialect speech synthesis scenarios. Furthermore, the execution subject of this method can be a speech synthesis device, such as a mobile phone, computer, or any other electronic device capable of speech synthesis.

[0068] Figure 1 This is one of the flowcharts illustrating the speech synthesis method provided in the embodiments of the present invention, such as... Figure 1 As shown, the method includes:

[0069] Step 101: Obtain the target text.

[0070] The target text is in Mandarin Chinese.

[0071] Specifically, the target text can be obtained through methods such as web crawling, scanning paper documents, or taking photos.

[0072] Step 102: Perform prosodic level prediction on the target text to obtain the prosodic boundary information of the target text.

[0073] The prosodic hierarchy comprises three distinct levels: prosodic words, prosodic phrases, and intonation phrases, each corresponding to pauses of varying lengths in natural language communication. In speech synthesis, accurate prosodic hierarchy prediction of the target text can improve the naturalness and intelligibility of the synthesized speech.

[0074] Specifically, the prediction of prosodic word boundaries, prosodic phrase boundaries, and intonation phrase boundaries can be performed as different prediction tasks. For example, the target text can be input into different prediction modules in a prosodic hierarchical prediction model to obtain different prosodic boundaries of the target text, such as prosodic word boundaries, prosodic phrase boundaries, and intonation phrase boundaries. Then, based on the fact that the boundary of a larger level is also the boundary of a smaller level, these different prosodic boundaries can be integrated. Taking the prosodic word boundary as the smallest level and the intonation phrase boundary as the largest level as an example, suppose the target text "If you didn't hear clearly, it's better to speak into the microphone." corresponds to the prosodic level "If you didn't hear #1 clearly #3, it's better to speak #2 into the microphone #1 #3.", then "#1" is a prosodic word boundary, "#2" is a prosodic phrase boundary and also a prosodic word boundary, and "#3" is an intonation phrase boundary and also a prosodic word and prosodic phrase boundary.

[0075] Step 103: Based on the prosodic word boundaries in the prosodic boundary information, determine each prosodic word in the target text.

[0076] Specifically, using the prosodic boundary information obtained above, each prosodic word in the target text is determined. Continuing with the example of the target text "It's better to speak into the microphone than to hear clearly," whose corresponding prosodic level is "It's better to speak into the microphone than to hear clearly #1 #3 #3 #1 #2 #3 #1 #2 #3 #3 #1 #2 #3 #1 #3," where "#1," "#2," and "#3" all represent the prosodic word boundaries of the target text. By segmenting the text using these prosodic word boundaries, all the prosodic words in the target text can be obtained, namely "It's better to hear clearly," "It's better to speak into the microphone than to hear clearly #3 #1 #2 #3 #1 #1 #2 #3 #1 #2 #3 #1 #2 #3 #1 #2 #3 #2 #1 #2 #3 #2 #1 #2 #2 #3 #2 #1 #2 #2 #3 #2 #1 #2 #2 #2 #3 #2 #2 #1 #2 #2 #2 #2 #2 #2 #3 ...3 #2 #3 #3 #4 #3 #4 #3 #4 #4 #3 #4 #3 #4 #3 #4 #4 #3 #4 #3 #4 #4 #3 #4 #4 #5 #3 #4 #4 #5 #3 #4 #4 #5 #3 #4 #4 #5 #3 #4 #4 #5 #3 #4 #4 #5 #4 #5 #3 #4 #5 #4 #5 #4 #5 #4 #5 #4

[0077] Step 104: Obtain the dialect sounds, rhymes and tones corresponding to the target text, and determine the dialect tone sandhi patterns corresponding to each prosodic word based on the dialect sounds, rhymes and tones. The dialect tone sandhi patterns are used to indicate the tone sandhi boundaries in each prosodic word.

[0078] Among them, dialect phonology includes the dialect pronunciation and the tone corresponding to the pronunciation. For example, the target text "It's better to say it into the microphone if you didn't hear clearly." corresponds to the Shanghainese phonology "[m a6][t in0]...[j ioa9][h oa9]", where "[]" is a syllable. The first character of the syllable is the initial consonant corresponding to the Shanghainese pronunciation; the second part of the characters is the final vowel corresponding to the Shanghainese pronunciation; and the number part is the tone corresponding to the Shanghainese pronunciation.

[0079] Specifically, the prosody tones of the target text in the corresponding dialect can be first queried through a pronunciation dictionary, and then the obtained prosody tones of the dialect are input into a sandhi pattern prediction model to obtain the sandhi pattern of the corresponding dialect for each prosodic word in the target text. Here, the sandhi pattern of the dialect refers to the segmentation pattern of the sandhi boundary of the corresponding prosodic word in Mandarin, and can also be understood as the tone change pattern of the pronunciation controlled by a linguistic unit composed of several characters in a continuous dialect stream.

[0080] Taking Shanghainese as an example, the prosody tones of the target text "Didn't hear clearly. It's better to speak对着 the microphone." in the corresponding dialect are "[m a6][t in0][ts in5][s ang7]…[j ioa9][h oa9]". Combining with the sandhi rules of Shanghainese, the sandhi boundary corresponding to this target text is "[m a6] / [t in0] / [ts in5][s ang7] / …[b i3][jioa4] / [h oa9]". Among them, the prosody tones of the prosodic word "Didn't" in the dialect are "[m a6] / "; the prosody tones of "hear clearly" in the dialect are "[t in0] / [ts in5][s ang7] / "; the prosody tones of "better" in the dialect are "[bi3][jioa4] / [h oa9] / ". Thus, the sandhi patterns of the prosodic words "Didn't", "hear clearly", and "better" in the dialect are "1", "1+2", and "2+1" respectively. This sandhi pattern of the dialect indicates the sandhi boundary in each prosodic word.

[0081] Step 105: Synthesize the dialect voice corresponding to the target text based on the prosody boundary information and the sandhi pattern of the dialect.

[0082] Specifically, after obtaining the prosody boundary information and the sandhi pattern of the dialect corresponding to the target text, first, based on the prosodic word boundaries in the prosody boundary information, the prosodic word boundaries, prosodic phrase boundaries, and intonation phrase boundaries in the target text are obtained, and based on the sandhi pattern of the dialect corresponding to each prosodic word, the dialect voice corresponding to the target text that conforms to the dialect tone change rules is generated, thereby improving the intelligibility and naturalness of the dialect voice.

[0083] The voice synthesis method provided by the embodiment of the present invention takes a single prosodic word in the target text as a unit, and determines the sandhi pattern of the corresponding dialect for each prosodic word through the prosody tones of the dialect corresponding to the target text, so as to avoid the problem of incorrect sandhi boundaries of some special case words caused by defining the prosodic word boundary in Mandarin and the sandhi boundary in the dialect as the same semantic unit, improve the accuracy of the sandhi of the dialect corresponding to the target text, and further improve the accuracy of the dialect voice synthesis, and improve the intelligibility and naturalness of the dialect voice.

[0084] Figure 2 This is the second flowchart illustrating the speech synthesis method provided in this embodiment of the invention. Based on the above embodiments, this embodiment details the specific implementation process of performing prosodic level prediction on the target text to obtain the prosodic boundary information of the target text. For example... Figure 2 As shown, the method includes:

[0085] Step 201: Input the target text into the encoding layer of the prosody prediction model to obtain the first feature information of each character in the target text.

[0086] Figure 3 This is a schematic diagram of the structure of the prosody prediction model provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the prosodic prediction model consists of three parts: an encoding layer, a decoding layer, and an output layer. The encoding layer extracts the first feature information from the target text; the decoding layer uses the first feature information output by the encoding layer to decode and transform the text according to different tasks; and the output layer uses the different prosodic level boundaries output by the decoding layer to predict prosodic boundary labels.

[0087] Specifically, after obtaining the target text, it is input into the encoding layer of the prosody prediction model so that the encoding layer can encode the target text to obtain the first feature information of each character in the target text. For example... Figure 3 As shown, after the target text is input into the encoding layer of the prosody prediction model, it is segmented using BERT (Bidirectional Encoder Representation from Transformers) to obtain each Mandarin character in the target text. Each Mandarin character is then input into the encoder to obtain the code corresponding to each character. This code is then input into the character feature extractor to obtain the first feature information corresponding to that character. For example, suppose the length of the target text is L. t After encoding, the result will be of length L. t A numerical encoding string, where each digit represents the encoding of the corresponding Mandarin character; further, in this string of length L... t After the numerical encoded string is input into the character feature extractor, each Mandarin character will obtain a feature of length D. t The first feature information, from which the dimension corresponding to the target text can be obtained is (D t L t The first feature information F t Among them, the first feature information F t It incorporates semantic feature information of the target text.

[0088] Step 202: Input the first feature information of each character into the decoding layer of the prosody prediction model to obtain the first probability that the boundary of each character is the prosodic boundary.

[0089] Specifically, such as Figure 3 As shown, the decoding layer treats the prosodic word boundaries, prosodic phrase boundaries, and intonation phrase boundaries corresponding to the target text as separate tasks for prediction. Each prediction task performs a separate feature transformation on the first feature information output by the encoding layer, using the traditional Seq2Seq+Attention method. Through the decoding processing of the decoding layer, the prosodic prediction model can learn the features of different prosodic level labels by combining character feature information and contextual semantic environment.

[0090] Taking the current prediction task as predicting prosodic word boundaries as an example, assume that the first feature information F input to the decoding layer... t The dimension is (D) t L t If the decoding layer has a length of D, then the decoding layer will process each dimension. t The first feature information is learned, and the first probability P of the prosodic word boundary is taken as the boundary of each character. i Among them, P i A larger numerical value indicates a higher probability that the current position is a prosodic word boundary. Based on this, the decoding layer will eventually output a value of length L. t The prosodic level probability vector P t That is, the first probability P of each character i The set. For example, suppose the target text is "If you didn't hear me clearly, it's better to speak into the microphone." The encoding layer yields a set with dimension (D). t After obtaining the first feature information of (16), a prosodic level probability vector of length 16 is obtained through the decoding layer [0.1, 0.8, 0.2, 0.1, ..., 0.9, 0.05]. Each position represents the first probability P that the character at that position is a prosodic word boundary. i .

[0091] Step 203: Determine the prosodic boundary information of the target text based on the first probability that the boundary of each character is a prosodic boundary.

[0092] Specifically, such as Figure 3 As shown, the output layer will generate a probability vector P for each prosodic level output by the decoding layer. t Prosodic boundary transformation is performed by comparing the first probability corresponding to each character with a probability threshold to obtain a length of L. t Rhythmic boundary label I t [0,1,…,1,0]. Where 0 represents that the position is not a prosodic boundary, and 1 represents that the position is a prosodic boundary.

[0093] Furthermore, the prosodic boundary labels generated from all prosodic levels are integrated. The integration logic is to use the highest-level prosodic boundary label. For example, if a position is predicted as both a prosodic word and a prosodic phrase boundary, that position is determined as a prosodic phrase boundary. Finally, the merged prosodic boundary labels are integrated with the target text to obtain the final target text with prosodic boundary labels. Taking the prosodic word boundary as the smallest level and the intonation phrase boundary as the largest level as an example, assuming the prosodic word boundary label of the target text "It's better to say it into the microphone." is [0,1,0,0,1,1,0,0,1,0]; the prosodic phrase boundary label is [0,0,0,0,0,1,0,0,1,0]; and the intonation phrase boundary label is [0,0,0,0,0,0,0,0,1,0], then the integrated target text is "It's better to say #2 into #1 microphone #1 #3." Among them, "#1" is the boundary of the prosodic word, "#2" is the boundary of the prosodic phrase, and "#3" is the boundary of the intonation phrase.

[0094] In this embodiment, the target text is input into the encoding layer of the prosody prediction model to obtain the first feature information of each character in the target text. The first feature information of each character is then input into the decoding layer of the prosody prediction model to obtain the first probability that the boundary of each character is a prosodic boundary. Based on the first probability that the boundary of each character is a prosodic boundary, more prosodic boundary information in the target text can be determined. This provides a foundation for obtaining a more accurate dialectal tone sandhi pattern after further determining each prosodic word in the target text.

[0095] In one possible implementation, the prosodic prediction model described above can be trained as follows: acquire at least two text samples, each of which includes prosodic level annotation information; determine prosodic boundary labels based on the prosodic level annotation information; input each text sample into the initial prosodic prediction model to obtain the predicted prosodic boundary corresponding to each text sample; and iteratively train the initial prosodic prediction model based on the predicted prosodic boundary and the prosodic boundary label to obtain the prosodic prediction model.

[0096] Specifically, at least two text samples are first manually annotated to a prosodic level. For example, the Mandarin text sample "I didn't hear clearly, it's better to speak into the microphone." corresponds to the prosodic level annotation "I didn't hear #1 clearly #3, it's better to speak into the microphone #1 #2 #1 #3." Here, "#1" is a prosodic word marker, "#2" is a prosodic phrase marker, and "#3" is an intonation phrase marker.

[0097] Secondly, based on the prosodic hierarchy annotation information of the annotation, the prosodic boundary labels of each text sample are determined. For example, for the prosodic hierarchy annotation of the Mandarin text "Didn't#1 hear clearly#3. It's better#3 to speak#2对着#1 microphone#1.", the corresponding prosodic boundary labels are: "0 1 0 0 1 0 0 1 0 0 1 1 0 0 1 0". Here, 0 represents that the position of the character is a non-prosodic boundary, and 1 represents that the position of the character is a prosodic boundary. In addition, the special characters such as punctuation marks in the text sample are treated as separate characters in the labels here.

[0098] Further, after each text sample is input into the initial prosody prediction model and the prosodic boundaries corresponding to each text sample are obtained, according to the corresponding prosodic boundary labels, the loss value Loss between the predicted prosodic boundary and the true prosodic boundary is calculated through the loss function calculation method shown in the following formula (1). t .

[0099] Loss t = CrossEntropy(P t , G t ) (1)

[0100] Where P t is the predicted value of the prosodic boundary, and G t is the true value of the prosodic boundary.

[0101] Based on this, according to the calculated loss value between the predicted prosodic boundary and the true prosodic boundary, the initial prosody prediction model is iteratively trained, that is, the backpropagation algorithm is used to update the relevant parameters in the initial prosody prediction model, so that the loss value after the model is reconstructed becomes smaller and smaller until the convergence condition or the loss threshold is reached, thereby obtaining the prosody prediction model.

[0102] Further, for the three different prosodic hierarchy boundary labels included in each text sample, namely the prosodic word boundary label, the prosodic phrase boundary label, and the intonation phrase boundary label, combined with the prosodic boundary predicted by the initial prosody prediction model, through the loss function calculation method shown in the above formula (1), the loss values of different prosodic hierarchy boundaries are calculated, and then the total loss value Loss is calculated through the following formula (2). [[ID=​​​​​​​​​​​​​​​​​​

[0104] Among them, loss L1 is the loss value of the prosodic word boundary, and loss L2 is the loss value of the prosodic phrase boundary, and loss L3 is the loss value of the intonation phrase boundary; alpha L1 is the loss weight of the prosodic word boundary, and alpha L2 is the loss weight of the prosodic phrase boundary, and alpha L3 is the loss weight of the intonation phrase boundary.

[0105] Furthermore, based on the calculated total loss value, the initial prosody prediction model is iteratively trained until the convergence condition or the loss threshold is reached, so as to obtain a more accurate prosody prediction model.

[0106] In this embodiment, by obtaining the prosodic hierarchy annotation information included in at least two text samples, the prosodic boundary labels are determined, and the text samples are input into the initial prosody prediction model to predict the prosodic boundaries corresponding to each text sample. Furthermore, based on the predicted prosodic boundaries and the prosodic boundary labels, the initial prosody prediction model is iteratively trained to obtain a more accurate prosody prediction model. Thus, in the case of using the more accurate prosody prediction model as the prosody boundary prediction model for the target text, the accuracy of the prosody boundary prediction of the target text can be improved.

[0107] Based on any of the above embodiments, when determining the dialect connected speech tone sandhi patterns corresponding to each prosodic word based on the dialect phonology, the following method can be adopted: based on the dialect phonology, determine the number of dialect syllables corresponding to each prosodic word; based on the dialect phonology, the first feature information of each character in the target text, the number of dialect syllables corresponding to each prosodic word, and the number of Mandarin syllables corresponding to each prosodic word, determine the dialect connected speech tone sandhi patterns corresponding to each prosodic word.

[0108] Specifically, it is necessary to first predict the prosodic boundaries of the target text through the prosody prediction model; then, count the number of Mandarin syllables corresponding to each prosodic word in the target text. At the same time, obtain the dialect phonology corresponding to each prosodic word in the target text through the pronunciation dictionary, so as to count the number of dialect syllables corresponding to each prosodic word in the target text. Taking the dialect as Shanghai dialect and the target text as "méi yǒu tīng qīng chu" as an example, assuming that the prosodic boundary label output by the prosody prediction model is "méi yǒu#1 tīng qīng chu#3", the corresponding dialect phonologies are [m a6] and [t in0][ts in5][s ang7] respectively. Thus, the number of Mandarin syllables corresponding to each prosodic word in the target text can be obtained as 2 and 3 respectively, and the number of dialect syllables corresponding to each prosodic word in the target text can be obtained as 1 and 3 respectively.

[0109] Based on this, on the basis of obtaining the dialect phonology tones, the first feature information of each character in the target text, the number of dialect syllables corresponding to each prosodic word, and the number of Mandarin syllables corresponding to each prosodic word, and then according to the corresponding dialect sandhi rules, the dialect phonology tones corresponding to each prosodic word in the target text are divided into sandhi units to obtain the dialect sandhi patterns of each prosodic word. Continuing with the example where the dialect is Shanghainese and the target text is "didn't hear clearly", combining with the Shanghainese sandhi rules, the dialect phonology tones corresponding to "didn't" are "[m a6] / "; the phonology tones corresponding to "hear clearly" are "[t in0] / [ts in5] [s ang7] / ". Among them, the dialect corresponding to "didn't" has one sandhi unit; the dialect corresponding to "hear clearly" has two sandhi units. Thus, it can be further obtained that the dialect sandhi patterns corresponding to the prosodic words "didn't" and "hear clearly" are "1" and "1 + 2" respectively.

[0110] In this embodiment, through the dialect phonology tones, the number of dialect syllables corresponding to each prosodic word is determined, so that the sandhi units of the dialect phonology tones corresponding to each prosodic word in the target text can be determined through the dialect phonology tones, the first feature information of each character in the target text, the number of dialect syllables corresponding to each prosodic word, and the number of Mandarin syllables corresponding to each prosodic word, as well as the number of dialect syllables included in the corresponding sandhi unit, thereby the dialect sandhi patterns of each prosodic word can be determined more accurately.

[0111] Figure 4 This is the third flowchart of the speech synthesis method provided by the embodiments of the present invention. On the basis of the above embodiments, the specific implementation process of determining the dialect sandhi patterns of each prosodic word based on the dialect phonology tones, the first feature information of each character in the target text, the number of dialect syllables corresponding to each prosodic word, and the number of Mandarin syllables corresponding to each prosodic word is described in detail. As Figure 4 shown, the method includes:

[0112] Step 401: Input the first feature information of each character and the number of Mandarin syllables corresponding to each prosodic word into the encoding layer of the sandhi pattern prediction model to obtain the semantic feature information of each prosodic word.

[0113] Figure 5 This is the structural diagram of the sandhi pattern prediction model provided by the embodiments of the present invention. As Figure 5As shown in the figure, the connected tone sandhi pattern prediction model includes three parts: an encoding layer, a decoding layer, and an output layer. Among them, the encoding layer is used to extract target feature information from the target text; the decoding layer uses the target feature information output by the encoding layer for decoding to generate a second probability representing each connected tone sandhi pattern; the output layer uses the second probability output by the decoding layer to predict the connected tone sandhi pattern labels of each prosodic word.

[0114] Specifically, after obtaining the first feature information of each character and the number of Mandarin syllables corresponding to each prosodic word through the foregoing embodiments, they are input into the encoding layer of the connected tone sandhi pattern prediction model, so as to perform average pooling on the first feature information of each character based on the number of Mandarin syllables corresponding to each prosodic word, and obtain the semantic feature information of each prosodic word.

[0115] Taking the dialect as Shanghainese and the target text as "méiyǒu tīng qīngchu" as an example, among them, the number of Mandarin syllables corresponding to the prosodic words "méiyǒu" and "tīng qīngchu" in the target text is [2, 3]. Further, after inputting the first feature information of each character and the number of Mandarin syllables corresponding to each prosodic word into the encoding layer of the connected tone sandhi pattern prediction model, the encoding layer will perform average pooling on the first feature information with a dimension of (D t , 5) in the form of [2, 3] to obtain semantic feature information F with a dimension of (D t , 2). semantic .

[0116] Step 402: Input the dialect initials, finals, and tones into the encoding layer to obtain the second feature information corresponding to each character in the dialect, and determine the dialect initials, finals, and tones feature information of each prosodic word based on the second feature information corresponding to each character and the number of dialect syllables corresponding to each prosodic word.

[0117] Specifically, as Figure 5 shown, after inputting the dialect initials, finals, and tones obtained through the pronunciation dictionary into the encoding layer of the connected tone sandhi pattern prediction model, the second feature information corresponding to each character in the dialect is obtained through the embedding layer; then, based on the number of dialect syllables corresponding to each prosodic word, average pooling is performed on the second feature information corresponding to each character in the dialect to obtain the dialect initials, finals, and tones feature information of each prosodic word.

[0118] Continuing with the dialect as Shanghainese and the target text as "méiyǒu tīng qīngchu" as an example, among them, the number of dialect syllables corresponding to the dialect initials, finals, and tones "[m a6]" and "[t in0][ts in5][s ang7]" of the prosodic words "méiyǒu" and "tīng qīngchu" is [1, 3]. Further, after inputting the dialect initials, finals, and tones into the encoding layer of the connected tone sandhi pattern prediction model, the encoding layer will pass the dialect initials, finals, and tones through the embedding layer to obtain a dimension of (D dialectThe second feature information of (4) is average pooled in the form of [1,3] to obtain a dimension of (D) dialect ,2) Dialectal phonological and tonal features F dialect .

[0119] Step 403: Determine the target feature information based on the semantic feature information of each prosodic word and the corresponding dialectal phonological and tonal feature information.

[0120] Specifically, such as Figure 5 As shown, after obtaining the semantic feature information of each prosodic word in the target text and the corresponding dialectal phonological and tonal feature information, the semantic feature information and the dialectal phonological and tonal feature information are concatenated to obtain the target feature information of the target text.

[0121] Continuing with the example of Shanghainese as the dialect and "didn't hear clearly" as the target text, the semantic feature information F obtained above will be used... semantic With dialectal phonological and tonal features F dialect By concatenating the components, we obtain a dimension of (D) t +D dialect ,2) Target feature information F sandhi Among them, the target feature information F sandhi It integrates the semantic features of the prosodic words "no" and "listen clearly" with the phonological features of the dialect.

[0122] Step 404: Input the target feature information into the decoding layer of the tone sandhi prediction model to obtain the second probability of each prosodic word for each tone sandhi pattern.

[0123] Specifically, such as Figure 5 As shown, the decoding layer of the liaison tone sandhi prediction model also uses the Seq2Seq+Attention method. Its only difference from the prosody prediction model is that the output prediction probability is no longer the first probability that the boundary of the current character is a prosodic boundary, but rather the second probability that the prosodic word at the current position is each liaison tone sandhi pattern. For example, suppose the target text has L... sandhi There are C rhyming words, and the tone sandhi patterns are C. sandhi If the target text obtains target feature information through the encoding layer, and then passes through the decoding layer, it will obtain a dimension (C). sandhi ,L sandhi The second probability vector is denoted by . Each row represents a tone sandhi pattern, each column represents a prosodic word, and the numerical value represents the second probability that the prosodic word at the corresponding position belongs to each tone sandhi pattern.

[0124] Step 405: Determine the dialectal tone sandhi pattern of each prosodic word based on the second probability of each prosodic word as a tone sandhi pattern.

[0125] Specifically, for the second probability vector with the dimension of (C sandhi , L sandhi ) output by the decoding layer, the output layer will select the dialect sandhi pattern corresponding to the maximum second probability in each column as the dialect sandhi pattern label for each prosodic word, and output the position of this dialect sandhi pattern among C sandhi types of dialect sandhi patterns. Taking the two prosodic words "méiyǒu" (没有) and "tīng qīngchu" (听清楚) and six dialect sandhi patterns "1", "2", "3", "1+1", "1+2", "2+1" as an example, the output layer will take the maximum second probability in each column of the second probability vector with the dimension of (6, 2) output by the decoding layer as the dialect sandhi pattern label for the prosodic word at the corresponding position, and output the position of this dialect sandhi pattern among the 6 types of dialect sandhi patterns, that is, the output dialect sandhi pattern label is [1, 5]. Among them, 1 represents that "méiyǒu" is the first sandhi pattern, that is, "1"; 5 represents that "tīng qīngchu" is the fifth sandhi pattern, that is, "1+2".

[0126] In this embodiment, by inputting the first feature information of each character and the number of Mandarin syllables corresponding to each prosodic word into the encoding layer of the sandhi pattern prediction model, the semantic feature information of each prosodic word is obtained; by inputting the dialect initials, finals and tones into the encoding layer, the second feature information corresponding to each character in the dialect is obtained, and based on each second feature information and the number of dialect syllables corresponding to each prosodic word, the dialect initials, finals and tones feature information of each prosodic word is determined, and then based on the semantic feature information of each prosodic word and the corresponding dialect initials, finals and tones feature information, the target feature information is determined; then the target feature information is input into the decoding layer of the sandhi pattern prediction model to obtain the second probability of each prosodic word for each sandhi pattern, and based on the second probability of each prosodic word for each sandhi pattern, the more matching sandhi pattern of each prosodic word in the target text is determined, so as to further determine the more accurate dialect sandhi pattern of each prosodic word. Thereby, the accuracy of dialect speech synthesis is improved.

[0127] Furthermore, the aforementioned tone sandhi prediction model can be trained as follows: Obtain the dialect audio corresponding to each text sample, including the dialect's initial, final, and tone annotation information; for each dialect audio, determine the tone sandhi pattern label for each prosodic word sample based on the initial, final, and tone annotation information, which indicates the tone sandhi boundary in each prosodic word sample; for each text sample, obtain the third feature information of each character; input the third feature information of each character, the number of dialect syllables corresponding to each prosodic word sample in the text sample, the number of Mandarin syllables corresponding to each prosodic word sample in the text sample, and the initial, final, and tone annotation information of the dialect corresponding to the text sample into the initial tone sandhi prediction model to obtain the predicted tone sandhi pattern for each prosodic word sample in the text sample; based on the predicted tone sandhi pattern and the tone sandhi pattern label, iteratively train the initial tone sandhi prediction model to obtain the tone sandhi prediction model.

[0128] Specifically, the dialect audio corresponding to each text sample is first annotated with the corresponding dialect sounds, rhymes and tones. For example, the Shanghai dialect audio corresponding to the text sample "I didn't hear clearly, it's better to say it into the microphone." is annotated as "[m a6][t in0]...[j ioa9][h oa9]".

[0129] Secondly, for the initial consonant, vowel, and tone annotation information marked on the dialect audio, the corresponding sandhi boundaries of the dialect are determined through the corresponding sandhi rules of the dialect, and the number of dialect syllables corresponding to each prosodic word sample in the corresponding text sample is counted. Based on the sandhi boundaries and the number of dialect syllables corresponding to each prosodic word sample, the sandhi pattern label of each prosodic word sample is determined. Taking the dialect audio as Shanghainese and the text sample as "Didn't hear clearly. It's better to speak对着麦克风说比较好。" as an example, the initial consonant, vowel, and tone annotation information corresponding to this dialect audio is "[m a6][t in0][ts in5][sang7]…[j ioa9][h oa9]", and combined with the sandhi rules of Shanghainese, the corresponding sandhi boundaries of this dialect are obtained as "[m a6] / [t in0] / [ts in5][sang7] / …[b i3][j ioa4] / [h oa9]". Among them, the initial consonant, vowel, and tone of the prosodic word sample "没有" corresponding to the dialect are "[m a6] / "; the initial consonant, vowel, and tone of the prosodic word sample "听清楚" are "[tin0] / [tsin5][s ang7] / "; the initial consonant, vowel, and tone of the prosodic word sample "比较好" are "[b i3][jioa4] / [h oa9] / ". It can be seen from this that the number of dialect syllables corresponding to the prosodic word samples "没有", "听清楚", and "比较好" are 1, 3, and 3 respectively. At the same time, combined with the sandhi boundaries of the syllables within each prosodic word sample, it can be seen that the dialect corresponding to the prosodic word sample "没有" has one sandhi unit, and the corresponding number of syllables is 1; the dialect corresponding to the prosodic word sample "听清楚" has two sandhi units, and the corresponding number of syllables is 1 and 2 respectively; the dialect corresponding to the prosodic word sample "比较好" has two sandhi units, and the corresponding number of syllables is 2 and 1 respectively. Based on this, the sandhi pattern labels corresponding to the prosodic word samples "没有", "听清楚", and "比较好" are "1", "***", and "***" respectively. In the above method, since the sandhi pattern label corresponding to each prosodic word sample is determined, more targeted label information can be obtained, making the determined sandhi pattern label more accurate.

[0130] Then, each text sample is input into the encoding layer of the trained prosody prediction model to obtain each Mandarin character in each text sample using BERT tokenization, and the encoding corresponding to each character is obtained by inputting the Mandarin character into the encoder, and then the third feature information of each character is obtained by inputting the encoding into the character feature extractor.

[0131] It should be noted that there are some inaccuracies in the original text. For example, the Chinese text "对着麦克风说比较好。" seems to be incomplete or incorrect. And in the translation of the relevant part in , there are some "***" in the original text which need to be corrected according to the correct content. The above translation is based on the existing text as accurately as possible.Furthermore, after inputting the third feature information of each character, the number of dialect syllables corresponding to each prosodic word sample in the text sample, the number of Mandarin syllables corresponding to each prosodic word sample in the text sample, and the tone and phonological annotation information of the dialect corresponding to the text sample into the initial tone sandhi prediction model, the predicted tone sandhi patterns of each prosodic word sample in the text sample are obtained. Then, combined with the tone sandhi pattern labels of the prosodic word samples, the loss value Loss between the predicted tone sandhi pattern and the actual tone sandhi pattern is calculated using the loss function calculation method shown in the following formula (3). sandhi .

[0132] Loss sandhi = CrossEntropy(P sandhi G sandhi (3)

[0133] Among them, P sandhi G is the predicted value for the liaison and tone sandhi pattern. sandhi This is the actual value for the tone sandhi mode.

[0134] Based on this, the initial tone shift prediction model is iteratively trained according to the calculated loss value between the predicted tone shift pattern and the actual tone shift pattern. That is, the backpropagation algorithm is used to update the relevant parameters in the initial tone shift prediction model so that the loss value after the model is reconstructed becomes smaller and smaller until the convergence condition or loss threshold is reached, thereby obtaining the tone shift prediction model.

[0135] In this embodiment, by acquiring the dialectal tone and rhyme annotation information included in the dialect audio corresponding to each text sample, the tone sandhi pattern label of each prosodic word sample is determined. The tone sandhi pattern label is used to indicate the tone sandhi boundary in each prosodic word sample. Based on this, for each text sample, the third feature information of each character in the text sample is obtained. This information, along with the third feature information of each character, the number of dialect syllables corresponding to each prosodic word sample in the text sample, the number of Mandarin syllables corresponding to each prosodic word sample in the text sample, and the tone and rhyme of the corresponding dialect in the text sample, are used to determine the tone sandhi pattern label of each prosodic word sample. The labeled information is input into the initial tone sandhi prediction model to obtain the predicted tone sandhi patterns of each prosodic word sample in the text sample. Based on the predicted tone sandhi patterns and tone sandhi pattern labels, the initial tone sandhi prediction model is iteratively trained to obtain a more accurate tone sandhi prediction model. Thus, by using the more accurate tone sandhi prediction model as the prediction model for the tone sandhi patterns of the target text, the accuracy of the prediction of dialect tone sandhi patterns of the target text can be improved, thereby providing a foundation for the subsequent synthesis of more accurate dialect speech.

[0136] The speech synthesis apparatus provided in the embodiments of the present invention is described below. The speech synthesis apparatus described below can be referred to in correspondence with the speech synthesis method described above.

[0137] Figure 6 This is a schematic diagram of the structure of the speech synthesis device provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the device includes:

[0138] Module 610 is used to acquire the target text;

[0139] The prediction module 620 is used to predict the prosodic level of the target text and obtain the prosodic boundary information of the target text.

[0140] The determination module 630 is used to determine each prosodic word in the target text based on the prosodic word boundaries in the prosodic boundary information.

[0141] The acquisition module 610 is also used to acquire the dialect sounds and tones corresponding to the target text;

[0142] The determining module 630 is further used to determine the dialect tone sandhi pattern corresponding to each prosodic word based on the dialect sounds and tones. The dialect tone sandhi pattern is used to indicate the tone sandhi boundary in each prosodic word.

[0143] The synthesis module 640 is used to synthesize the dialect speech corresponding to the target text based on prosodic boundary information and dialectal tone sandhi patterns.

[0144] The speech synthesis device provided in this invention takes a single prosodic word in the target text as a unit and determines the dialect tone sandhi pattern corresponding to each prosodic word through the dialect sounds and tones corresponding to the target text. This avoids the problem of incorrect tone sandhi boundaries for some special words caused by defining the boundaries of Mandarin prosodic words and dialect tone sandhi boundaries as the same semantic unit, thereby improving the accuracy of dialect tone sandhi corresponding to the target text, and thus improving the accuracy of dialect speech synthesis, and improving the intelligibility and naturalness of dialect speech.

[0145] Optionally, the prediction module 620 is specifically used for:

[0146] The target text is input into the encoding layer of the prosody prediction model to obtain the first feature information of each character in the target text;

[0147] The first feature information of each character is input into the decoding layer of the prosody prediction model to obtain the first probability that the boundary of each character is the prosodic boundary;

[0148] Based on the first probability that the boundary of each character is a prosodic boundary, the prosodic boundary information of the target text is determined.

[0149] Optionally, the determining module 630 is specifically used for:

[0150] Based on the sounds, rhymes, and tones of the dialect, determine the number of dialect syllables corresponding to each prosodic word;

[0151] Based on the dialect initials, finals, and tones, the first feature information of each character in the target text, the number of dialect syllables corresponding to each prosodic word, and the number of Mandarin syllables corresponding to each prosodic word, the dialect tone sandhi pattern of each prosodic word is determined.

[0152] Optionally, the determining module 630 is specifically used for:

[0153] The first feature information of each character and the number of Mandarin syllables corresponding to each prosodic word are input into the encoding layer of the tone sandhi prediction model to obtain the semantic feature information of each prosodic word.

[0154] The dialect phonology and tone are input into the coding layer to obtain the second feature information corresponding to each character in the dialect. Based on the second feature information and the number of dialect syllables corresponding to each prosodic word, the dialect phonology and tone feature information of each prosodic word is determined.

[0155] Based on the semantic features of each prosodic word and the corresponding dialectal phonological features, the target feature information is determined.

[0156] The target feature information is input into the decoding layer of the tone sandhi prediction model to obtain the second probability of each prosodic word for each tone sandhi pattern.

[0157] Based on the second probability of each prosodic word as a tone sandhi pattern, the dialect tone sandhi pattern of each prosodic word is determined.

[0158] Optionally, the prosody prediction model is trained in the following manner;

[0159] The acquisition module 610 is also used to acquire at least two text samples, each text sample including prosodic level annotation information;

[0160] The determining module 630 is further configured to determine the prosodic boundary label based on the prosodic level annotation information;

[0161] The input module is used to input each text sample into the initial prosodic prediction model to obtain the predicted prosodic boundary corresponding to each text sample.

[0162] The training module is used to iteratively train the initial prosodic prediction model based on the predicted prosodic boundaries and prosodic boundary labels to obtain the prosodic prediction model.

[0163] Optionally, the tone sandhi prediction model is trained in the following manner;

[0164] The acquisition module 610 is also used to acquire the dialect audio corresponding to each text sample, the dialect audio including the pronunciation, rhyme and tone annotation information of the dialect;

[0165] The determining module 630 is further configured to determine the tone sandhi pattern label of each prosodic word sample based on the tone and rhyme annotation information corresponding to the audio of each dialect, and the tone sandhi pattern label is used to indicate the tone sandhi boundary in each prosodic word sample.

[0166] The acquisition module 610 is also used to acquire the third feature information of each character in the text sample for each text sample;

[0167] The input module is also used to input the third feature information of each character, the number of dialect syllables corresponding to each prosodic word sample in the text sample, the number of Mandarin syllables corresponding to each prosodic word sample in the text sample, and the tone and rhyme annotation information of the dialect corresponding to the text sample into the initial tone sandhi prediction model to obtain the predicted tone sandhi pattern of each prosodic word sample in the text sample.

[0168] The training module is also used to iteratively train the initial tone shift prediction model based on the predicted tone shift pattern and tone shift pattern label to obtain the tone shift prediction model.

[0169] Optionally, the determining module 630 is specifically used for:

[0170] Based on the initial, final, and tone annotation information, the tone sandhi boundaries of the corresponding dialect and the number of dialect syllables corresponding to each prosodic word sample in the corresponding text sample are determined.

[0171] Based on the tone sandhi boundaries and the number of dialect syllables corresponding to each prosodic word sample, the tone sandhi pattern label of each prosodic word sample is determined.

[0172] The apparatus of this embodiment can be used to execute the method of any of the above-described speech synthesis device-side method embodiments. Its specific implementation process and technical effects are similar to those of the above-described speech synthesis device-side method embodiments. For details, please refer to the detailed description of the above-described speech synthesis device-side method embodiments, which will not be repeated here.

[0173] Figure 7 This example illustrates one of the physical structural diagrams of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a speech synthesis method, including: acquiring target text; performing prosodic level prediction on the target text to obtain prosodic boundary information; determining each prosodic word in the target text based on the prosodic word boundaries in the prosodic boundary information; acquiring the dialectal phonology corresponding to the target text, and determining the dialectal tone sandhi pattern corresponding to each prosodic word based on the dialectal phonology, the dialectal tone sandhi pattern indicating the tone sandhi boundary in each prosodic word; and synthesizing the dialectal speech corresponding to the target text based on the prosodic boundary information and the dialectal tone sandhi pattern.

[0174] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] Figure 8 Example 2: A schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, and also includes a speaker 850. The processor 810, communications interface 820, memory 830, and speaker 850 communicate with each other via the communication bus 840. The speaker 850 is used to play dialect speech. The processor 810 can call logical instructions in the memory 830 to execute a speech synthesis method, including: acquiring target text; performing prosodic level prediction on the target text to obtain prosodic boundary information of the target text; determining each prosodic word in the target text based on the prosodic word boundaries in the prosodic boundary information; acquiring the dialect phonology corresponding to the target text, and determining the dialect tone sandhi pattern corresponding to each prosodic word based on the dialect phonology, the dialect tone sandhi pattern being used to indicate the tone sandhi boundaries in each prosodic word; and synthesizing the dialect speech corresponding to the target text based on the prosodic boundary information and the dialect tone sandhi pattern.

[0176] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0177] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the speech synthesis methods provided by the above methods, including: acquiring target text; performing prosodic level prediction on the target text to obtain prosodic boundary information of the target text; determining each prosodic word in the target text based on the prosodic word boundaries in the prosodic boundary information; acquiring the dialectal phonology corresponding to the target text, and determining the dialectal tone sandhi pattern corresponding to each prosodic word based on the dialectal phonology, wherein the dialectal tone sandhi pattern is used to indicate the tone sandhi boundary in each prosodic word; and synthesizing the dialectal speech corresponding to the target text based on the prosodic boundary information and the dialectal tone sandhi pattern.

[0178] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech synthesis method provided by the above methods, including: acquiring target text; performing prosodic level prediction on the target text to obtain prosodic boundary information of the target text; determining each prosodic word in the target text based on the prosodic word boundaries in the prosodic boundary information; acquiring the dialectal phonology corresponding to the target text, and determining the dialectal tone sandhi pattern corresponding to each prosodic word based on the dialectal phonology, wherein the dialectal tone sandhi pattern is used to indicate the tone sandhi boundary in each prosodic word; and synthesizing the dialectal speech corresponding to the target text based on the prosodic boundary information and the dialectal tone sandhi pattern.

[0179] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that, include: Obtain the target text; Prosodic level prediction is performed on the target text to obtain the prosodic boundary information of the target text; Based on the prosodic word boundaries in the prosodic boundary information, each prosodic word in the target text is determined; Obtain the dialect phonology and tone corresponding to the target text, and determine the dialect tone sandhi pattern corresponding to each prosodic word based on the dialect phonology and tone. The dialect tone sandhi pattern is used to indicate the tone sandhi boundary in each prosodic word. Based on the prosodic boundary information and the dialectal tone sandhi pattern, the dialectal speech corresponding to the target text is synthesized.

2. The speech synthesis method according to claim 1, characterized in that, The step of performing prosodic level prediction on the target text to obtain the prosodic boundary information of the target text includes: The target text is input into the encoding layer of the prosody prediction model to obtain the first feature information of each character in the target text; The first feature information of each character is input into the decoding layer of the prosody prediction model to obtain the first probability that the boundary of each character is a prosodic boundary; Based on a first probability that the boundary of each character is a prosodic boundary, the prosodic boundary information of the target text is determined.

3. The speech synthesis method according to claim 1 or 2, characterized in that, The determination of the dialect tone sandhi pattern corresponding to each of the prosodic words based on the dialect phonology includes: Based on the dialect sounds, rhymes, and tones, determine the number of dialect syllables corresponding to each of the prosodic words; Based on the dialect phonology, the first feature information of each character in the target text, the number of dialect syllables corresponding to each prosodic word, and the number of Mandarin syllables corresponding to each prosodic word, the dialect tone sandhi pattern of each prosodic word is determined.

4. The speech synthesis method according to claim 3, characterized in that, The step of determining the dialect tone sandhi pattern of each prosodic word based on the dialect phonology, the first feature information of each character in the target text, the number of dialect syllables corresponding to each prosodic word, and the number of Mandarin syllables corresponding to each prosodic word includes: The first feature information of each character and the number of Mandarin syllables corresponding to each prosodic word are input into the encoding layer of the tone sandhi prediction model to obtain the semantic feature information of each prosodic word. The dialect phonology and tone are input into the coding layer to obtain the second feature information corresponding to each character in the dialect. Based on each second feature information and the number of dialect syllables corresponding to each prosodic word, the dialect phonology and tone feature information of each prosodic word is determined. Based on the semantic feature information of each prosodic word and the corresponding dialectal phonological and tonal feature information, the target feature information is determined; The target feature information is input into the decoding layer of the tone sandhi prediction model to obtain the second probability of each prosodic word as a tone sandhi pattern. Based on the second probability of each prosodic word as a tone sandhi pattern, the dialect tone sandhi pattern of each prosodic word is determined.

5. The speech synthesis method according to claim 2, characterized in that, The prosody prediction model was trained in the following manner: Obtain at least two text samples, each of which includes prosodic level annotation information; Based on the prosodic level annotation information, determine the prosodic boundary label; Each of the text samples is input into the initial prosodic prediction model to obtain the predicted prosodic boundary corresponding to each of the text samples; Based on the predicted prosodic boundary and the prosodic boundary label, the initial prosodic prediction model is iteratively trained to obtain the prosodic prediction model.

6. The speech synthesis method according to claim 5, characterized in that, The prediction model for continuous tone sandhi patterns is trained in the following manner: Obtain the dialect audio corresponding to each of the text samples, wherein the dialect audio includes the pronunciation, rhyme and tone annotation information of the dialect; For the phonological and tone annotation information corresponding to the audio of each dialect, the tone sandhi pattern label of each prosodic word sample is determined based on the phonological and tone annotation information. The tone sandhi pattern label is used to indicate the tone sandhi boundary in each prosodic word sample. For each text sample, obtain the third feature information of each character in the text sample; The third feature information of each character, the number of dialect syllables corresponding to each prosodic word sample in the text sample, the number of Mandarin syllables corresponding to each prosodic word sample in the text sample, and the initial tone and rhyme annotation information of the dialect corresponding to the text sample are input into the initial tone sandhi pattern prediction model to obtain the predicted tone sandhi pattern of each prosodic word sample in the text sample. Based on the predicted tone shift pattern and the tone shift pattern label, the initial tone shift pattern prediction model is iteratively trained to obtain the tone shift pattern prediction model.

7. The speech synthesis method according to claim 6, characterized in that, The step of determining the tone sandhi pattern label for each prosodic word sample based on the phonological and tonal annotation information includes: Based on the phonological and tone annotation information, the tone sandhi boundaries of the corresponding dialect are determined, as well as the number of dialect syllables corresponding to each prosodic word sample in the corresponding text sample. Based on the tone sandhi boundaries and the number of dialect syllables corresponding to each prosodic word sample, the tone sandhi pattern label of each prosodic word sample is determined.

8. A speech synthesis device, characterized in that, include: The acquisition module is used to acquire the target text; The prediction module is used to perform prosodic level prediction on the target text to obtain the prosodic boundary information of the target text; The determination module is used to determine each prosodic word in the target text based on the prosodic word boundaries in the prosodic boundary information; The acquisition module is also used to acquire the dialectal sounds, rhymes, and tones corresponding to the target text; The determining module is further configured to determine the dialect tone sandhi pattern corresponding to each of the prosodic words based on the dialect phonology and tone, wherein the dialect tone sandhi pattern is used to indicate the tone sandhi boundary in each of the prosodic words. The synthesis module is used to synthesize the dialect speech corresponding to the target text based on the prosodic boundary information and the dialect tone sandhi pattern.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the speech synthesis method as described in any one of claims 1 to 7.

10. An electronic device comprising a speaker, a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor is used to acquire target text; perform prosodic level prediction on the target text to obtain prosodic boundary information of the target text; and determine each prosodic word in the target text based on the prosodic word boundaries in the prosodic boundary information. The dialect phonology and tone corresponding to the target text are obtained, and the dialect tone sandhi pattern corresponding to each prosodic word is determined based on the dialect phonology and tone. The dialect tone sandhi pattern is used to indicate the tone sandhi boundary in each prosodic word. Based on the prosodic boundary information and the dialect tone sandhi pattern, the dialect speech corresponding to the target text is synthesized. The speaker is used to play the dialect speech.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the speech synthesis method as described in any one of claims 1 to 7.

Images