Speech synthesis method and device, speech synthesis model training method and device

Abstract

The application provides a speech synthesis method and device, a speech synthesis model training method and device. The method comprises the following steps: determining first position information of a word to be read again in speech text, and performing prosody prediction on the speech text to obtain at least one first prosody prediction result. The first prosody prediction result comprises at least one of a pitch prediction result, a volume prediction result and a pronunciation duration prediction result. The prosody prediction result of the speech text corresponding to the first position information in the at least one first prosody prediction result is adjusted to obtain at least one second prosody prediction result. The readjustment comprises at least one of increasing the pitch, increasing the volume and reducing the pronunciation duration. According to the at least one second prosody prediction result, a speech synthesis result is generated. The application can realize the readjustment of the speech synthesis result at the specified word position, improve the expressiveness of the synthesized speech, and make the synthesized speech more natural in the dialogue application.

Description

Technical Field

[0001] This application relates to the field of speech processing, specifically to a speech synthesis method and apparatus, and a training method and apparatus for a speech synthesis model. Background Technology

[0002] Artificial intelligence (AI) is currently booming, and research on speech synthesis based on deep neural networks is maturing. In natural communication, people's speech has intonation and emphasizes key words and phrases to help listeners grasp the main points. For example, in the sentence "I like cookies," the speaker usually emphasizes the word "cookie" to emphasize what they like, or emphasizes the word "like" to highlight that cookies are something they enjoy. However, synthesized speech, lacking emphasis, often emphasizes the entire sentence or none at all, making it difficult for listeners to grasp the main points and resulting in a lack of engaging delivery. Therefore, for speech synthesis technology to simulate human speech, it needs controllable emphasis synthesis capabilities.

[0003] Therefore, how to achieve word-level stress control to make synthesized speech more expressive and varied has become an urgent technical problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide a speech synthesis method and apparatus, and a speech synthesis model training method and apparatus, which can realize the repetition of synthesized speech at specified character and word positions, thereby improving the expressiveness of synthesized speech.

[0005] In a first aspect, embodiments of this application provide a speech synthesis method, the method comprising: determining first position information of a word to be stressed in a speech text, and performing prosodic prediction on the speech text to obtain at least one first prosodic prediction result; adjusting the prosodic prediction result of the speech text corresponding to the first position information in the at least one first prosodic prediction result to obtain at least one second prosodic prediction result, wherein the adjustment includes at least one of raising the pitch, increasing the volume, and reducing the pronunciation duration; and generating a speech synthesis result based on the at least one second prosodic prediction result.

[0006] Secondly, embodiments of this application provide a training method for a speech synthesis model, comprising: determining the sample pitch spectrum of each speaker's speech sample data based on sample data; calculating the sample mean and sample standard deviation of the sample pitch spectrum; performing regularization processing on the sample pitch spectrum based on the sample mean and sample standard deviation; and training the speech synthesis model based on the regularized sample pitch spectrum.

[0007] Thirdly, embodiments of this application provide a speech synthesis apparatus, comprising: a first determining module, configured to determine first position information of a word to be stressed in a speech text, and perform prosodic prediction on the speech text to obtain at least one first prosodic prediction result; an adjusting module, configured to adjust the prosodic prediction result of the speech text corresponding to the first position information in the at least one first prosodic prediction result to obtain at least one second prosodic prediction result, wherein the adjustment includes at least one of raising the pitch, increasing the volume, and reducing the pronunciation duration; and a generating module, configured to generate a speech synthesis result based on the at least one second prosodic prediction result.

[0008] Fourthly, embodiments of this application provide a computationally readable storage medium storing a computer program for executing the speech synthesis method described in the first aspect above, or the training method for the speech synthesis model described in the second aspect above.

[0009] Fifthly, embodiments of this application provide an electronic device, including: a processor; and a memory for storing processor-executable instructions, wherein the processor is used to execute the speech synthesis method described in the first aspect above, or the training method for the speech synthesis model described in the second aspect above.

[0010] This application provides a speech synthesis method and apparatus, and a speech synthesis model training method and apparatus. By acquiring at least one first phonological prediction result of the speech text and adjusting the phonological prediction result at the corresponding position of the word to be stressed, at least one second phonological prediction result is obtained. Based on the at least one second phonological prediction result, a speech synthesis result with stress effect is obtained, thereby realizing word (or character) level stress control, improving the expressiveness of the speech synthesis result (i.e., synthesized speech), and making the speech synthesis result more natural in dialogue applications. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the implementation environment of the training method for the speech synthesis model provided in an exemplary embodiment of this application.

[0013] Figure 2a This is a schematic diagram of the implementation environment of the speech synthesis method provided in an exemplary embodiment of this application.

[0014] Figure 2b This is a schematic diagram of a speech synthesis model provided in an exemplary embodiment of this application.

[0015] Figure 3 This is a flowchart illustrating a speech synthesis method provided in an exemplary embodiment of this application.

[0016] Figure 4 This is a flowchart illustrating a speech synthesis method provided in another exemplary embodiment of this application.

[0017] Figure 5 This is a flowchart illustrating a speech synthesis method provided in yet another exemplary embodiment of this application.

[0018] Figure 6 This is a flowchart illustrating a speech synthesis method provided in another exemplary embodiment of this application.

[0019] Figure 7 This is a flowchart illustrating a speech synthesis method provided in another exemplary embodiment of this application.

[0020] Figure 8 This is a flowchart illustrating a speech synthesis method provided in another exemplary embodiment of this application.

[0021] Figure 9 This is a flowchart illustrating a speech synthesis method provided in another exemplary embodiment of this application.

[0022] Figure 10 This is a flowchart illustrating a training method for a speech synthesis model provided in an exemplary embodiment of this application.

[0023] Figure 11 This is a schematic diagram of the structure of a speech synthesis device provided in an exemplary embodiment of this application.

[0024] Figure 12 This is a schematic diagram of the structure of a training device for a speech synthesis model provided in an exemplary embodiment of this application.

[0025] Figure 13 This is a block diagram of an electronic device for speech synthesis or training a speech synthesis model, provided in an exemplary embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] Text-to-Speech (TTS) is a technology that generates artificial speech through mechanical and electronic methods, and research on speech synthesis based on deep neural networks is becoming increasingly mature. Several landmark studies have emerged in this field, such as the Tacotron model based on autoregressive learning, the FastSpeech model based on alignment learning, the GlowTTS model based on affine learning, and the DiffGAN model based on a noise reduction diffusion probability model.

[0028] Although the quality and effect of synthesized speech have approached that of real people, a lack of fine-grained emphasis remains a problem in practical industrial applications. In other words, during normal communication, speech has intonation and emphasizes key words and phrases, allowing the listener to grasp the main points. Therefore, for speech synthesis technology to simulate human conversation, it must have controllable emphasis synthesis capabilities. Current technologies typically employ the following methods.

[0029] For example, one method involves using part-of-speech tagging to determine the emphasis positions in the synthesized text. However, this method, which relies on text information processing to determine emphasis positions, can lead to errors in predicting emphasis, resulting in failed emphasis effects. Furthermore, since emphasizing different words highlights different meanings, it's impossible to theoretically determine which words in the text require emphasis.

[0030] For example, a repetitive speech synthesis model can be directly obtained by marking the stress positions during training. However, this method requires a large amount of manual annotation during training, and the characteristics of the synthesized stress positions are consistent with the stress distribution of the labeled data, making it difficult to control flexibly.

[0031] For example, it provides a function to control the stress of the last word in English pronunciation. However, the above method mainly solves the problem of controlling the stress at the end of English sentences. It does not have the function of controlling the stress within the entire sentence. Moreover, simple pitch adjustment cannot achieve the ideal stress effect. In addition, the three methods illustrated above are mainly for speech synthesis models of a single speaker, not for stress solutions in multi-speaker speech synthesis models.

[0032] To address the aforementioned issues, this application provides a speech synthesis method. This method determines the first position information of the word to be stressed in the speech text and performs prosodic prediction on the speech text to obtain at least one first prosodic prediction result. The first prosodic prediction result includes at least one of pitch prediction result, volume prediction result, and pronunciation duration prediction result. The method then adjusts the prosodic prediction result of the speech text corresponding to the first position information in the at least one first prosodic prediction result to obtain at least one second prosodic prediction result. The adjustment includes at least one of raising the pitch, increasing the volume, and decreasing the pronunciation duration. Based on the at least one second prosodic prediction result, a speech synthesis result is generated. This achieves word (or character) level stress control, improves the expressiveness of the speech synthesis result (i.e., synthesized speech), and makes the speech synthesis result more natural in dialogue applications.

[0033] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0034] Figure 1 This is a schematic diagram of the implementation environment for a speech synthesis model training method provided in this application embodiment. The implementation environment includes a computer device 110 and a server 120. The computer device 110 can also be connected to the server 120 via a communication network. Optionally, the communication network can be a wired network or a wireless network.

[0035] The computer device 110 can be a general-purpose computer or a computer device composed of dedicated integrated circuits, etc., and this application embodiment does not limit this. For example, the computer device 110 can be a mobile terminal device such as a tablet computer, or it can be a personal computer (PC), such as a laptop computer and a desktop computer, etc. Those skilled in the art will know that the number of the above-mentioned computer devices 110 can be one or more, and their types can be the same or different. For example, there can be one computer device 110, or there can be dozens or hundreds of computer devices 110, or more. This application embodiment does not limit the number and type of computer devices 110.

[0036] In some alternative embodiments, the computer device 110 may acquire sample data and use the acquired sample data to train a neural network to obtain a network model for multi-speaker speech synthesis.

[0037] Server 120 is a single server, or a combination of several servers, or a virtualization platform, or a cloud computing service center.

[0038] In some alternative embodiments, computer device 110 sends the acquired sample data to server 120, and server 120 then trains the neural network using the sample data to obtain a network model for multi-speaker speech synthesis.

[0039] Figure 2a This is a schematic diagram of the implementation environment of the speech synthesis method provided in an exemplary embodiment of this application. The implementation environment includes speech text 210, speech synthesis model 220, and speech synthesis result 230.

[0040] like Figure 2a As shown, when it is necessary to synthesize speech with emphasis effect, the speech text 210 can be input into the speech synthesis model 220. The speech synthesis model 220 can adjust the emphasis of the input speech text 210 at the word and character level and output the synthesized speech with emphasis effect (i.e., speech synthesis result 230).

[0041] Figure 2b This is a schematic diagram of a speech synthesis model 220 provided in an exemplary embodiment of this application. Figure 2b As shown, the speech synthesis model 220 can be a FastPitch model. The speech synthesis model 220 includes a text encoding module, a prosody predictor (also called an "intermediate layer"), a repeat-on-phonemes module 225, and a speech synthesis module, wherein the text encoding module includes a text encoder 221, and the speech synthesis module includes a Mel decoder 226.

[0042] Specifically, the text encoder 221 includes a phoneme embedding module 2211, a speaker embedding module 2212, and a feed-forward transformer (FFT) module 2213. The prosody predictor includes a pitch predictor 223, an energy predictor 224, and a duration predictor 222. The Mel decoder 226 includes an FFT module 2261 and a fully connected layer 2262, wherein the fully connected layer 2262 outputs Mel spectral features.

[0043] The text encoder encodes the input phoneme sequence into a priori distribution representation. The Mel decoder decodes the target Mel spectrum from the prosody predictor's processing results.

[0044] Figure 3 This is a flowchart illustrating a speech synthesis method provided in an exemplary embodiment of this application. Figure 3 The method can be implemented by a speech synthesis model on a computing device. As shown Figure 3 in the figure, the speech synthesis method includes the following contents.

[0045] 310: Determine first position information of a word to be stressed in the speech text, and perform prosody prediction on the speech text to obtain at least one first prosody prediction result.

[0046] Specifically, the word to be stressed may be an important word or character in a sentence that needs to be highlighted in a stressed manner. For example, in "I love the garden", if the place that is loved is to be emphasized, the word to be stressed is "garden"; if the action of loving the garden is to be emphasized, the word to be stressed is "love".

[0047] The first position information may be the position in the phoneme sequence. That is, the first position information may be the start and end positions of the phoneme sequence. For example, if the speech text is "I love the garden", the corresponding phoneme sequence is "wo3 ai4 hua1 yuan2" (where the numbers 3, 4, 2 or 1 represent tones), and if the word to be stressed is "garden", the position of the word to be stressed is 8 to 18.

[0048] The manner of performing prosody prediction on the speech text may be: after inputting the speech text into the speech synthesis model, obtaining a tone prediction result based on a tone predictor in the speech synthesis model, and / or obtaining a volume prediction result based on a volume predictor in the speech synthesis model, and / or obtaining a pronunciation duration prediction result based on a duration predictor in the speech synthesis model. It should be noted that the prosody prediction may be predicting the tone, pronunciation duration (i.e., speech rate) and volume of the speech text.

[0049] The at least one first prosody prediction result may be one or more. The at least one first prosody prediction result may include at least one of a tone prediction result, a pronunciation duration prediction result, and a volume prediction result, and the embodiments of the present application do not specifically limit the quantity and specific content of the first prosody prediction result.

[0050] It should be noted that there is no fixed sequence requirement for the step of determining the first position information and the step of obtaining the at least one first prosody prediction result. The step of determining the first position information may be performed first, or the step of obtaining the at least one first prosody prediction result may be performed first.

[0051] 320: Perform stress adjustment on the prosody prediction result of the speech text corresponding to the first position information in the at least one first prosody prediction result, so as to obtain at least one second prosody prediction result.

[0052] In one embodiment, emphasis adjustment includes at least one of raising the pitch, increasing the volume, and decreasing the pronunciation duration. The purpose of emphasis adjustment is to give the word to be emphasized a stressed effect. Emphasis adjustment may include one or more of raising the pitch, increasing the volume, and decreasing the pronunciation duration, and this application embodiment does not specifically limit this.

[0053] Specifically, in at least one first prosodic prediction result, the prosodic prediction result of the speech text corresponding to the word to be stressed is determined, and then the prosodic prediction result is adjusted for stress to obtain at least one second prosodic prediction result with a stress effect. The second prosodic prediction result can be understood as the first prosodic prediction result after stress adjustment. For example, when the first prosodic prediction result is a pitch prediction result, the second prosodic prediction result can be the first prosodic prediction result after improving the pitch prediction result corresponding to the position of the word to be stressed.

[0054] In one embodiment, when at least one first prosodic prediction result is a pitch prediction result, the first pitch prediction result corresponding to the first position information is first determined, then a pitch adjustment value is obtained, and the first pitch prediction result is added to the pitch adjustment value to obtain a pitch prediction result with a stressed tone effect. And / or, when at least one first prosodic prediction result is a pronunciation duration prediction result, the first pronunciation duration prediction result corresponding to the first position information is first determined, and the mean of the first pronunciation duration in the first pronunciation duration prediction result is calculated. Then a reduction factor is obtained, and the mean of the first pronunciation duration is divided by the reduction factor to obtain a pronunciation duration prediction result with a stressed tone effect. And / or, when at least one first prosodic prediction result is a volume prediction result, the first volume prediction result corresponding to the first position information is first determined, then a volume enhancement factor is obtained, and the first volume prediction result is multiplied by the volume enhancement factor to obtain a volume prediction result with a stressed tone effect.

[0055] In one embodiment, after the first prosodic prediction result of the first position information of the speech text corresponding to the word to be stressed has been adjusted for stress, the adjusted prosodic prediction result with stress effect can replace the original first prosodic prediction result at the same position to obtain a second prosodic prediction result. For example, after the first tone prediction result corresponding to the first position information of the word to be stressed has been adjusted for stress, the tone prediction result with stress effect can replace the original first tone prediction result to obtain a second tone prediction result.

[0056] It should be noted that for a detailed description of step 320, please refer to the following embodiments.

[0057] 330: Generate a speech synthesis result based on at least one second prosodic prediction result.

[0058] Specifically, the speech synthesis result can be understood as the synthesized speech with emphasis effects output by the speech synthesis model. In other words, the speech synthesis model can support the emphasis function of specific characters and words.

[0059] The speech synthesis model may include a Mel decoder. First, the speech synthesis model can add at least one second prosodic prediction result to the encoded result and perform feature extension according to the duration distribution, aligning the encoded result with the length of the Mel spectrum. Then, the Mel decoder of the speech synthesis model decodes the Mel spectrum from the encoded result, thus obtaining a speech synthesis result with repetition.

[0060] It should be noted that the speech synthesis model in this application embodiment is a multi-speaker model. Therefore, a speaker embedding layer (e.g., Speaker Embedding) is added to the input side of the model. The speaker features obtained by the speaker embedding layer are added to the speech text, so that the text encoder of the speech synthesis model is input with the speaker's timbre information.

[0061] Therefore, the embodiments of this application obtain at least one first phonological prediction result of the speech text, adjust the phonological prediction result at the corresponding position of the word to be stressed to obtain at least one second phonological prediction result, and obtain a speech synthesis result with stress effect based on at least one second phonological prediction result, thereby realizing word (or character) level stress control, improving the expressiveness of the speech synthesis result (i.e., synthesized speech), and making the speech synthesis result more natural in dialogue applications.

[0062] Figure 4 This is a flowchart illustrating a speech synthesis method provided in another exemplary embodiment of this application. Figure 4 The example is Figure 3 Further limitations on step 320 in the embodiments. For example... Figure 4 As shown, step 320 further includes the following.

[0063] 410: In at least one first prosodic prediction result, determine the first pitch prediction result of the speech text corresponding to the first position information.

[0064] Specifically, at least one first prosodic prediction result can be a pitch prediction result based on the pitch predictor output in the speech synthesis model, where the pitch prediction result can be a pixel-level pitch spectrum. It should be noted that the pixel-level pitch spectrum can be obtained by first assigning a fundamental frequency value to each phoneme in the speech text, and then arranging the fundamental frequency values ​​corresponding to each phoneme in the same order according to the order of the phonemes in the speech text.

[0065] In one embodiment, in the pitch prediction results output by the pitch predictor of the speech synthesis model, the first pitch prediction result corresponding to the word to be stressed can be determined based on the first position information. This first pitch prediction result can be understood as the pitch segment corresponding to the word to be stressed.

[0066] 420: Obtain the pitch adjustment value, and based on the first pitch prediction result and the pitch adjustment value, obtain the pitch prediction result with stress effect.

[0067] Specifically, the pitch adjustment value can be obtained iteratively, and the unit of this pitch adjustment value can be Hz. It should be noted that for details on how to obtain the pitch adjustment value, please refer to [link to relevant documentation]. Figure 5 The embodiments described herein will not be repeated here to avoid repetition.

[0068] In one embodiment, the first tone prediction result corresponding to the first position information of the word to be stressed is added with a certain frequency tone adjustment value to obtain an adjusted first tone prediction result. The adjusted first tone prediction result replaces the original first tone prediction result at the same position to obtain a tone prediction result with stress repetition effect.

[0069] Therefore, it can be seen that the embodiments of this application, by raising the pitch of the word to be stressed, make the stress effect of the word to be stressed more obvious, which provides a guarantee for the subsequent generation of speech synthesis results with stress effect and enhances the expressiveness of speech synthesis results.

[0070] Figure 5 This is a flowchart illustrating a speech synthesis method provided in yet another exemplary embodiment of this application. Figure 5 The example is Figure 4 Further limitations on step 420 in the embodiments. For example... Figure 5 As shown, step 420 further includes the following.

[0071] 510: In at least one first prosodic prediction result, determine a second tone prediction result for the speech text corresponding to the second position information, and a third tone prediction result for the speech text corresponding to the third position information.

[0072] In one embodiment, the second position information is the position information of the word preceding the word to be emphasized, and the third position information is the position information of the word following the word.

[0073] Specifically, both the second and third positional information can be the positions of the phoneme sequence. That is, the second positional information can be represented by the positions of the beginning and end of the phoneme sequence, and the third positional information can also be represented by the positions of the beginning and end of the phoneme sequence. Furthermore, the second positional information can be the positional information of the word (or character) preceding the word to be stressed, and the third positional information can be the positional information of the word (or character) following the word to be stressed.

[0074] For example, when the audio text is “I love the Sky Garden” and the word to be stressed is “sky”, the corresponding phoneme sequence is “wo3 ai4 kong1 zhong1 hua1 yuan2”. Then the position of the word “love” adjacent to the word to be stressed (i.e., the second position information) is 4 to 7; the position of the word “garden” adjacent to the word to be stressed (i.e., the third position information) is 21 to 31.

[0075] The second tone prediction result can be the tone prediction result of the word preceding the word to be stressed, determined based on the second position information, from the tone prediction results output by the tone predictor of the speech synthesis model. This second tone prediction result can be understood as the tone segment corresponding to the word (or character) preceding the word to be stressed.

[0076] The third tone prediction result can be the third tone prediction result of the word adjacent to the word to be stressed, determined based on the third position information, from the tone prediction results output by the tone predictor of the speech synthesis model. This third tone prediction result can be understood as the tone segment corresponding to the word (or character) adjacent to the word to be stressed.

[0077] It should be noted that when the word to be emphasized is the first word in the audio text, the preceding word does not need to be considered during the calculation; that is, the second positional information does not need to be calculated. When the word to be emphasized is the last word in the audio text, the following word does not need to be considered during the calculation; that is, the third positional information does not need to be calculated.

[0078] 520: Calculate the mean of the first tone of the first tone prediction result, the mean of the second tone of the second tone prediction result, and the mean of the third tone of the third tone prediction result.

[0079] Specifically, the mean can be calculated based on the non-zero pitch values ​​in the pitch prediction results. The mean calculated for the first pitch prediction results is the first pitch mean, the mean calculated for the second pitch prediction results is the second pitch mean, and the mean calculated for the third pitch prediction results is the third pitch mean. The embodiments of this application do not specifically limit the calculation method of the mean.

[0080] It should be noted that since a zero pitch value represents a voiceless sound, where the throat does not vibrate during pronunciation and there is no fundamental frequency, only non-zero pitch values ​​are calculated when calculating the mean.

[0081] 530: Based on the first pitch mean, the initial pitch adjustment value, and the pre-stored pitch standard deviation of the speaker to be synthesized, obtain the initial stress pitch mean.

[0082] 540: Step a: Based on the first pitch mean, the pitch adjustment value after the Nth update, and the pre-stored pitch standard deviation of the speaker to be synthesized, obtain the mean of the stressed pitch value after the Nth update. In one embodiment, N≥1.

[0083] Specifically, the speech synthesis model can pre-store information on multiple speakers to be synthesized, allowing users to specify which speaker's voice to synthesize during the speech synthesis process. Furthermore, the speech synthesis model can include the pitch standard deviation, maximum pitch value, and pitch mean of multiple speakers to be synthesized, in order to facilitate subsequent calculation of pitch adjustment values.

[0084] The formula for calculating the average initial stressed pitch value is detailed in Formula (1) below:

[0085]

[0086] Where P(0)′ is the initial mean stress value of the word to be stressed; P pred (0) represents the first pitch mean; F represents the initial pitch adjustment value; Pitch std (Speaker X) represents the standard deviation of the pitch of the speaker "X".

[0087] It should be noted that the embodiments of this application can iterate the average value of the stressed pitch value by continuously adjusting the pitch adjustment value (i.e., F). The initial pitch adjustment value can be a preset value, and the subsequent adjustment of the pitch adjustment value can be done by increasing the predetermined value to obtain the updated pitch adjustment value. The embodiments of this application do not specifically limit the method of updating the pitch adjustment value.

[0088] In one embodiment, the mean stressed pitch value after the first update is obtained based on the first pitch mean, the pitch adjustment value after the first update, and the pre-stored pitch standard deviation of the speaker to be synthesized. ... The mean stressed pitch value after the Nth update is obtained based on the first pitch mean, the pitch adjustment value after the Nth update, and the pre-stored pitch standard deviation of the speaker to be synthesized, where N≥2. It should be noted that the formula for calculating the mean stressed pitch value after the Nth update is the same as the formula for calculating the initial mean stressed pitch value; please refer to formula (1) for details.

[0089] It should also be noted that when the average of the repeated pitch values ​​meets the condition in step 540, the iteration ends, and the pitch adjustment value applied in the last calculation is taken as the final pitch adjustment value.

[0090] 550: Repeat step a to obtain the average stressed pitch value after the Nth update, until the average stressed pitch value after the Nth update is greater than the average of the second pitch, the average of the third pitch and the prediction result of the first pitch, and less than the maximum pitch value of the speaker to be synthesized in the pre-stored value. Then, use the pitch adjustment value after the Nth update as the pitch adjustment value.

[0091] Specifically, by using continuously updated pitch adjustment values, step a (i.e., step 540) is repeatedly executed to obtain the average stressed pitch value after the first update, ..., to obtain the average stressed pitch value after the Nth update. This continues until the average stressed pitch value obtained after the Nth update is greater than the first pitch prediction result before adjustment, the average pitch value of the word preceding the word to be stressed (i.e., the second pitch value), and the average pitch value of the word following the word to be stressed (i.e., the third pitch value), and the average stressed pitch value after the Nth update is less than the pre-stored maximum pitch value of the speaker to be synthesized. At this point, the pitch adjustment value applied when calculating the average stressed pitch value after the Nth update is used as the final pitch adjustment value, where N≥2.

[0092] Therefore, it can be seen that the embodiments of this application obtain the tone adjustment value through iterative calculation, making the obtained tone adjustment value more accurate. At the same time, the tone comparison with the preceding and following adjacent words is added to ensure the stress effect of the word to be stressed in the sentence, avoid the problem of the tone of the word to be stressed being too high, and thus provide a guarantee for generating a more natural speech synthesis result.

[0093] Figure 6 This is a flowchart illustrating a speech synthesis method provided in another exemplary embodiment of this application. Figure 6 The example is Figure 3 Further limitations on step 320 in the embodiments. For example... Figure 6 As shown, step 320 further includes the following.

[0094] 610: In at least one first prosodic prediction result, determine the first pronunciation duration prediction result of the speech text corresponding to the first position information.

[0095] Specifically, at least one first prosodic prediction result can be a pronunciation duration prediction result based on the duration predictor output in the speech synthesis model, wherein the pronunciation duration prediction result can be a pixel-level sequence, that is, each pixel in the speech text corresponds to a pronunciation duration value.

[0096] In one embodiment, in the pronunciation duration prediction result output by the duration predictor of the speech synthesis model, the first pronunciation duration prediction result of the speech text corresponding to the word to be stressed can be determined based on the first position information.

[0097] 620: Calculate the mean of the first phonation duration in the first phonation duration prediction results.

[0098] Specifically, the embodiments of this application may use existing methods for calculating the average of the first phonation duration. The embodiments of this application do not impose specific limitations on the calculation of the average.

[0099] 630: Obtain the reduction factor, and based on the mean of the first pronunciation duration and the reduction factor, obtain the pronunciation duration prediction result with stress effect.

[0100] Specifically, the deceleration factor can be calculated iteratively. For details on the specific method of obtaining the deceleration factor, please refer to [link to relevant documentation]. Figure 7 The embodiments described herein will not be repeated here to avoid repetition.

[0101] In one embodiment, the mean of the first pronunciation duration of the speech text corresponding to the first position information of the word to be emphasized is divided by a slowing factor to obtain a slowed-down prediction result of the first pronunciation duration of the word to be emphasized. Then, the slowed-down prediction result of the first pronunciation duration replaces the prediction result of the first pronunciation duration at the same position to obtain a pronunciation duration prediction result with an emphasis effect.

[0102] It should be noted that pronunciation duration can be understood as speech rate, that is, how fast or slow one speaks a sentence or a word. Therefore, the pronunciation duration prediction result can also be understood as the speech rate prediction result.

[0103] Therefore, it can be seen that by reducing the pronunciation duration (i.e., speech rate) of the words to be stressed, the stress effect of the words to be stressed is more obvious, which provides a guarantee for the subsequent generation of speech synthesis results with stress effect and enhances the expressiveness of speech synthesis results.

[0104] Figure 7 This is a flowchart illustrating a speech synthesis method provided in another exemplary embodiment of this application. Figure 7 The example is Figure 6 Further limitations on step 630 in the embodiments. For example... Figure 7 As shown, step 630 further includes the following:

[0105] 710: In at least one first prosodic prediction result, determine a second pronunciation duration prediction result for the speech text corresponding to the second position information, and a third pronunciation duration prediction result for the speech text corresponding to the third position information.

[0106] In one embodiment, the second position information is the position information of the word preceding the word to be emphasized, and the third position information is the position information of the word following the word.

[0107] Specifically, the second pronunciation duration prediction result can be the second pronunciation duration prediction result of the preceding word adjacent to the word to be stressed, determined based on the second position information, from the pronunciation duration prediction results output by the duration predictor of the speech synthesis model. This second pronunciation duration prediction result can be understood as a segment of the pronunciation duration prediction results corresponding to the preceding word (or character) adjacent to the word to be stressed.

[0108] The third pronunciation duration prediction result can be the pronunciation duration prediction result of the word adjacent to the word to be stressed, determined based on the third position information, within the pronunciation duration prediction result output by the duration predictor of the speech synthesis model. This third pronunciation duration prediction result can be understood as a segment of the pronunciation duration prediction result corresponding to the word (or character) adjacent to the word to be stressed.

[0109] It should be noted that the second and third location information are related to... Figure 5 The embodiments are basically the same; please refer to [link / reference] for details. Figure 5 The embodiment describes this. Furthermore, when the word to be emphasized is the first word (or character), there is no need to calculate the second position information; when the word to be emphasized is the last word (or character), there is no need to calculate the third position information.

[0110] 720: Calculate the mean of the second pronunciation duration in the second pronunciation duration prediction results and the mean of the third pronunciation duration in the third pronunciation duration prediction results.

[0111] Specifically, the average of the second pronunciation duration prediction result and the third pronunciation duration prediction result are calculated respectively. The method of calculating the average is not specifically limited in the embodiments of this application.

[0112] 730: Based on the mean of the first pronunciation duration and the initial deceleration factor, obtain the initial mean of the stressed pronunciation duration.

[0113] 740: Step a: Based on the mean of the first pronunciation duration and the reduction factor after the Nth update, obtain the mean of the stressed pronunciation duration after the Nth update. In one embodiment, N≥1.

[0114] Specifically, the formula for obtaining the initial average duration of stressed syllables is detailed in formula (2) below:

[0115]

[0116] Where Dur(0)′ is the initial average duration of stressed syllables in the word to be stressed; Dur pred(0) represents the mean duration of the first phonation; α is the initial deceleration factor.

[0117] It should be noted that the embodiments of this application can iterate the average duration of stressed pronunciation by continuously adjusting the reduction factor (i.e., α). The initial reduction factor can be a preset value, and subsequent adjustments to the reduction factor can be made by increasing the preset value to obtain an updated reduction factor. The embodiments of this application do not specifically limit the method of obtaining the reduction factor.

[0118] In one embodiment, the average duration of stressed utterance after the first update is obtained based on the mean duration of the first utterance and the reduction factor after the first update. ... The average duration of stressed utterance after the Nth update is obtained based on the mean duration of the first utterance and the reduction factor after the Nth update, where N≥2. It should be noted that the formula for calculating the average duration of stressed utterance after the Nth update is the same as the formula for calculating the average duration of stressed utterance initially; please refer to formula (2) for details.

[0119] It should also be noted that the iteration ends when the average duration of stressed pronunciation meets the condition in step 740, and the reduction factor applied in the last calculation is used as the final reduction factor.

[0120] 750: Repeat step a to obtain the average duration of stressed pronunciation after the Nth update, until the average duration of stressed pronunciation after the Nth update is greater than the average duration of the second pronunciation and the average duration of the third pronunciation. Then, use the reduction factor after the Nth update as the reduction factor.

[0121] Specifically, by employing a continuously updated reduction factor, step 730 is repeated to obtain the average duration of stressed syllables after the first update, ..., to obtain the average duration of stressed syllables after the Nth update. This continues until the average duration of stressed syllables after the Nth update is greater than the average durations of the second and third syllables. At this point, the reduction factor applied when calculating the average duration of stressed syllables after the Nth update is used as the final reduction factor, where N≥2.

[0122] It should be noted that the condition for ending the iteration in step 740 can be understood as the constraint condition for obtaining the final deceleration factor.

[0123] Therefore, it can be seen that the embodiments of this application obtain the reduction factor through iterative calculation, making the obtained reduction factor more accurate. At the same time, the comparison with the pronunciation duration of adjacent words before and after the sentence is added to ensure the emphasis effect of the word to be emphasized in the sentence, avoiding the problem of the speech rate of the word to be emphasized being too slow or too fast, thus providing a guarantee for generating more natural speech synthesis results.

[0124] Figure 8This is a flowchart illustrating a speech synthesis method provided in another exemplary embodiment of this application. Figure 8 The example is Figure 3 Further limitations on step 320 in the embodiments. For example... Figure 8 As shown, step 320 further includes the following.

[0125] 810: In at least one first prosodic prediction result, determine the first volume prediction result of the speech text corresponding to the first position information.

[0126] Specifically, at least one first prosodic prediction result can be a volume prediction result based on the output of the volume predictor in the speech synthesis model, wherein the volume prediction result can be a pixel-level volume value sequence, that is, each phoneme in the speech text corresponds to a volume value.

[0127] In one embodiment, in the volume prediction result output by the volume predictor of the speech synthesis model, the first volume prediction result corresponding to the word to be emphasized can be determined based on the first position information.

[0128] 820: Calculate the first volume mean of the first volume prediction result.

[0129] Specifically, the mean of the first volume prediction results is calculated, i.e., the first volume mean. This application embodiment does not specifically limit the calculation of the mean.

[0130] 830: Obtain the volume boost factor, and based on the first volume mean and the volume boost factor, obtain the volume prediction result with rereading effect.

[0131] Specifically, the volume boost factor can be calculated iteratively. For details on how to obtain the volume boost factor, please refer to [link to relevant documentation]. Figure 9 The embodiments described herein will not be repeated here to avoid repetition.

[0132] In one embodiment, the average first volume value corresponding to the first position information of the word to be emphasized is multiplied by a volume enhancement factor to obtain a first volume prediction result after the volume of the word to be emphasized is increased. Then, the first volume prediction result after the volume is increased replaces the first volume prediction result at the same position to obtain a volume prediction result with an emphasis effect.

[0133] Therefore, this embodiment of the application enhances the volume of the words to be stressed, making the stress effect of the words to be stressed more obvious, thus providing a guarantee for the subsequent generation of speech synthesis results with stress effect and enhancing the expressiveness of the speech synthesis results.

[0134] Figure 9 This is a flowchart illustrating a speech synthesis method provided in another exemplary embodiment of this application. Figure 9 The example is Figure 8 Further limitations on step 820 in the embodiments. For example... Figure 9 As shown, step 820 further includes the following.

[0135] 910: In at least one first prosodic prediction result, determine a second volume prediction result for the speech text corresponding to the second position information, and a third volume prediction result for the speech text corresponding to the third position information.

[0136] In one embodiment, the second position information is the position information of the word preceding the word to be emphasized, and the third position information is the position information of the word following the word.

[0137] Specifically, the second volume prediction result can be the second volume prediction result of the word preceding the word to be stressed, determined based on the second position information, from the volume prediction results output by the volume predictor of the speech synthesis model. This second volume prediction result can be understood as a segment of the volume prediction results corresponding to the word (or character) preceding the word to be stressed.

[0138] The third volume prediction result can be the third volume prediction result of the word adjacent to the word to be stressed, determined based on the third position information, within the volume prediction results output by the volume predictor of the speech synthesis model. This third volume prediction result can be understood as a segment of the volume prediction results corresponding to the word (or character) adjacent to the word to be stressed.

[0139] It should be noted that the second and third location information are related to... Figure 5 The embodiments are basically the same; please refer to [link / reference] for details. Figure 5 The embodiment describes this. Furthermore, when the word to be emphasized is the first word (or character), there is no need to calculate the second position information; when the word to be emphasized is the last word (or character), there is no need to calculate the third position information.

[0140] 920: Calculate the second volume mean of the second volume prediction result and the third volume mean of the third volume prediction result.

[0141] Specifically, the average of the second volume prediction result and the third volume prediction result are calculated respectively. The method of calculating the average is not specifically limited in the embodiments of this application.

[0142] 930: Based on the initial volume boost factor and the first volume mean, obtain the initial average reread volume value.

[0143] 940: Step a: Based on the first average volume value and the volume enhancement factor after the Nth update, obtain the average reread volume value after the Nth update. In one embodiment, N≥1.

[0144] Specifically, the formula for obtaining the initial average repetition volume value is detailed in formula (3) below:

[0145] E(0)′=E pred (0)×β (3)

[0146] Where E(0)′ is the initial average stress volume value of the word to be stressed; E pred (0) represents the first average volume value; β represents the initial volume increase factor.

[0147] It should be noted that the embodiments of this application can iterate the average value of the repeated volume by continuously adjusting the volume boost factor (i.e., β). The initial volume boost factor can be a preset value, and the subsequent adjustment of the volume boost factor can be done by increasing the preset value to obtain an updated volume boost factor. The embodiments of this application do not specifically limit the method of obtaining the volume boost factor.

[0148] In one embodiment, the average reread volume after the first update is obtained based on the average first volume value and the volume enhancement factor after the first update. ... The average reread volume after the Nth update is obtained based on the average first volume value and the volume enhancement factor after the Nth update, where N≥2. It should be noted that the formula for calculating the average reread volume after the Nth update is the same as that for calculating the initial average reread volume; please refer to formula (3) for details.

[0149] It should also be noted that when the average reread volume value meets the condition in step 940, the iteration ends, and the volume increase factor applied in the last calculation is taken as the final volume increase factor.

[0150] 950: Repeat step a to obtain the average reread volume value after the Nth update, until the average initial reread volume value after the Nth update is greater than the average second volume value and the average third volume value, and less than the pre-stored maximum volume value of the speaker to be synthesized. Then, use the volume increase factor after the Nth update as the volume increase factor.

[0151] Specifically, by employing a continuously updated volume boosting factor, step 930 is repeatedly executed to obtain the average repetition volume value after one update, ..., to obtain the average repetition volume value after N updates. This continues until the average repetition volume value obtained after N updates is greater than the average volume value of the preceding word adjacent to the word to be emphasized (i.e., the second average volume value) and the average volume value of the following word adjacent to the word to be emphasized (i.e., the third average volume value), and the average repetition volume value after N updates is less than the pre-stored maximum volume value of the speaker to be synthesized. At this point, the volume boosting factor applied when calculating the average repetition volume value after N updates is used as the final volume boosting factor, where N≥2.

[0152] Therefore, it can be seen that the embodiments of this application obtain the volume enhancement factor through iterative calculation, making the obtained volume enhancement factor more accurate. At the same time, the comparison with the pronunciation volume of adjacent words before and after the main words ensures the emphasis effect of the word to be emphasized in the sentence, avoids the problem of the volume of the word to be emphasized being too high, and thus provides a guarantee for generating more natural speech synthesis results.

[0153] In one embodiment of this application, at least one first prosodic prediction result includes at least one of pitch prediction result, pronunciation duration prediction result, and volume prediction result.

[0154] Specifically, the number of at least one first prosodic prediction results can be one or more. The at least one first prosodic prediction result may include pitch prediction result, articulation duration prediction result, and volume prediction result, or it may include pitch prediction result and volume prediction result, or it may include pitch prediction result and articulation duration prediction result. The embodiments of this application do not specifically limit the number of first prosodic prediction results or the content of the prediction results.

[0155] Preferably, at least one first prosodic prediction result in the embodiments of this application includes a pitch prediction result, a pronunciation duration prediction result, and a volume prediction result, for a total of three prosodic prediction results.

[0156] Therefore, it can be seen that the embodiments of this application comprehensively adjust the tone, pronunciation duration (i.e. speech rate) and volume of the words to be stressed, so that the generated speech synthesis result (i.e. synthesized speech) has a word or phrase level stress effect, thereby improving the expressiveness of the speech synthesis result.

[0157] In one embodiment of this application, determining the first position information of the word to be emphasized in the speech text includes: determining the first position information of the word to be emphasized in the speech text based on the word to be emphasized input by the user.

[0158] Specifically, users can specify which word or phrase in the speech text is to be stressed in the user interface. Then, the speech synthesis model can determine the position information (i.e., the first position information) of the word to be stressed in the speech text based on the user's input.

[0159] Therefore, the embodiments of this application can achieve the repetition effect of synthesized speech by having the user specify the words to be emphasized, thereby enhancing the controllability of the repetition effect and improving the repetition quality of the speech synthesis result.

[0160] In one embodiment of this application, the speech synthesis method is implemented by a speech synthesis model, which includes a text encoding module, a prosody predictor, and a speech synthesis module. The prosody predictor includes at least one of a pitch predictor, a volume predictor, and a duration predictor. The text encoding module is used to determine the first position information of the word to be stressed in the speech text. The prosody predictor is used to perform prosody prediction on the speech text to obtain at least one first prosody prediction result, and to adjust the prosody prediction result of the speech text corresponding to the first position information in the at least one first prosody prediction result to obtain at least one second prosody prediction result. The speech synthesis module is used to generate a speech synthesis result based on the at least one second prosody prediction result. Specifically, the speech synthesis module can generate a speech synthesis result based on the Mel spectrum decoded by the Mel decoder.

[0161] Figure 10 This is a flowchart illustrating a training method for a speech synthesis model provided in an exemplary embodiment of this application. Figure 10 As shown, the training method includes the following:

[0162] 1010: Determine the sample pitch spectrum of each speaker's speech sample data based on the sample data.

[0163] Specifically, the sample data may include speech sample data from multiple speakers. Before training the speech synthesis model, speech sample data for each speaker is first acquired, and then the sample pitch spectrum corresponding to each speaker's speech sample data is calculated. Each pitch value in the sample pitch spectrum can be the fundamental frequency (i.e., the fundamental frequency value).

[0164] In one embodiment, the sample pitch spectrum is at the phoneme level. This sample pitch spectrum can be obtained by first assigning a fundamental frequency value to each phoneme in the speech text, and then arranging the fundamental frequency values ​​corresponding to each phoneme in the same order according to the order of the phonemes in the speech text.

[0165] 1020: Calculate the sample mean and sample standard deviation of the sample pitch spectrum.

[0166] Specifically, the sample mean can be calculated based on the non-zero pitch values ​​in the sample pitch spectrum. This application does not specifically limit the method of calculating the sample mean and sample standard deviation.

[0167] 1030: The sample pitch spectrum is regularized based on the sample mean and sample standard deviation.

[0168] Specifically, the regularization process can be achieved by dividing the difference between each fundamental frequency value and the sample mean in the sample pitch spectrum by the sample standard value. The embodiments of this application do not specifically limit the regularization method.

[0169] It should be noted that the fundamental frequency data of human speech (e.g., speech sample data) is generally between 50Hz and 500Hz. However, in machine learning (e.g., training speech synthesis models), it is inconvenient to directly calculate the fundamental frequency data of human speech. Therefore, before using the fundamental frequency data of human speech to train the speech synthesis model, it is necessary to first perform regularization (i.e., normalization) processing on the fundamental frequency data of human speech.

[0170] 1040: The speech synthesis model is trained based on the pitch spectrum of the samples after regularization.

[0171] Specifically, the regularized sample pitch spectrum is input into the speech synthesis model for training. Furthermore, the speech synthesis model can utilize a loss function for backpropagation, continuously training the model until the desired speech synthesis model is obtained. The loss function can be flexibly set according to actual needs; this embodiment does not impose specific limitations on it.

[0172] Therefore, it can be seen that because multi-speaker speech synthesis models (such as the FastPitch multi-speaker model) use mean and variance data from all training audio samples for pitch regularization during training, without distinguishing between speakers, they suffer from fundamental frequency crosstalk. Therefore, this embodiment of the application avoids fundamental frequency crosstalk caused by mixed training of multiple speakers by performing pitch regularization on the speech sample data of each speaker.

[0173] Figure 11 This is a schematic diagram of the structure of a speech synthesis device 1100 provided in an exemplary embodiment of this application. Figure 11 As shown, the speech synthesis device 1100 includes: a first determining module 1110, an adjusting module 1120, and a generating module 1130.

[0174] The first determining module 1110 is used to determine the first position information of the word to be stressed in the speech text and perform prosodic prediction on the speech text to obtain at least one first prosodic prediction result; the adjusting module 1120 is used to adjust the prosodic prediction result of the speech text corresponding to the first position information in the at least one first prosodic prediction result to obtain at least one second prosodic prediction result, wherein the adjustment includes at least one of raising the pitch, increasing the volume, and reducing the pronunciation duration; the generating module 1130 is used to generate a speech synthesis result based on at least one second prosodic prediction result.

[0175] This application provides a speech synthesis device that obtains at least one first phonological prediction result of the speech text, adjusts the phonological prediction result at the corresponding position of the word to be stressed to obtain at least one second phonological prediction result, and obtains a speech synthesis result with stress effect based on at least one second phonological prediction result. This achieves word (or character) level stress control, improves the expressiveness of the speech synthesis result (i.e., synthesized speech), and makes the speech synthesis result more natural in dialogue applications.

[0176] According to one embodiment of this application, the adjustment module 1120 is used to determine the first tone prediction result of the speech text corresponding to the first position information in at least one first prosodic prediction result; obtain the tone adjustment value; and obtain the tone prediction result with repetition effect based on the first tone prediction result and the tone adjustment value.

[0177] According to one embodiment of this application, the adjustment module 1120 is used to determine, in at least one first prosodic prediction result, a second tone prediction result of the speech text corresponding to the second position information, and a third tone prediction result of the speech text corresponding to the third position information, wherein the second position information is the position information of the preceding word adjacent to the word to be stressed, and the third position information is the position information of the following word adjacent to the word to be stressed; calculate the first tone mean of the first tone prediction result, the second tone mean of the second tone prediction result, and the third tone mean of the third tone prediction result; and adjust the tone based on the first tone mean and an initial tone adjustment value. The standard deviation of the pitch of the speaker to be synthesized is stored in advance, and the initial mean of the stressed pitch value is obtained. Step a: Based on the first pitch mean, the pitch adjustment value after the Nth update, and the standard deviation of the pitch of the speaker to be synthesized in advance, the mean of the stressed pitch value after the Nth update is obtained, where N≥1. Step a is repeated to obtain the mean of the stressed pitch value after the Nth update until the mean of the stressed pitch value after the Nth update is greater than the second pitch mean, the third pitch mean, and the first pitch prediction result, and less than the maximum pitch value of the speaker to be synthesized in advance. The pitch adjustment value after the Nth update is then used as the pitch adjustment value.

[0178] According to one embodiment of this application, the adjustment module 1120 is used to determine the first pronunciation duration prediction result of the speech text corresponding to the first position information in at least one first prosodic prediction result; calculate the mean of the first pronunciation duration in the first pronunciation duration prediction result; obtain a reduction factor; and obtain a pronunciation duration prediction result with stress effect based on the mean of the first pronunciation duration and the reduction factor.

[0179] According to an embodiment of this application, the adjustment module 1120 is used to determine, in at least one first prosodic prediction result, a second pronunciation duration prediction result corresponding to the second position information and a third pronunciation duration prediction result corresponding to the third position information of the speech text, wherein the second position information is the position information of the word preceding the word to be stressed, and the third position information is the position information of the word following the word to be stressed; calculate the mean of the second pronunciation duration in the second pronunciation duration prediction result and the mean of the third pronunciation duration in the third pronunciation duration prediction result; obtain an initial mean of stressed pronunciation duration based on the mean of the first pronunciation duration and an initial deceleration factor; step a: obtain the mean of stressed pronunciation duration after the Nth update based on the mean of the first pronunciation duration and the deceleration factor after the Nth update, where N≥1; repeat step a to obtain the mean of stressed pronunciation duration after the Nth update until the mean of stressed pronunciation duration after the Nth update is greater than the mean of the second pronunciation duration and the mean of the third pronunciation duration, and use the deceleration factor after the Nth update as the deceleration factor.

[0180] According to one embodiment of this application, the adjustment module 1120 is used to determine a first volume prediction result of the speech text corresponding to the first position information in at least one first prosody prediction result; calculate a first volume mean of the first volume prediction result; obtain a volume enhancement factor; and obtain a volume prediction result with repetition effect based on the first volume mean and the volume enhancement factor.

[0181] According to an embodiment of this application, the adjustment module 1120 is used to determine, in at least one first prosody prediction result, a second volume prediction result of the speech text corresponding to the second position information and a third volume prediction result of the speech text corresponding to the third position information, wherein the second position information is the position information of the word preceding the word to be stressed, and the third position information is the position information of the word following the word to be stressed; calculate the second volume mean of the second volume prediction result and the third volume mean of the third volume prediction result; obtain the initial stressed volume mean based on the initial volume boosting factor and the first volume mean; step a: obtain the stressed volume mean after the Nth update based on the first volume mean and the volume boosting factor after the Nth update, wherein N≥1; repeat step a to obtain the stressed volume mean after the Nth update until the initial stressed volume mean after the Nth update is greater than the second volume mean and the third volume mean, and less than the pre-stored maximum volume value of the speaker to be synthesized, and use the volume boosting factor after the Nth update as the final volume boosting factor.

[0182] According to one embodiment of this application, at least one first prosodic prediction result includes at least one of pitch prediction result, pronunciation duration prediction result, and volume prediction result.

[0183] According to one embodiment of this application, the first determining module 1110 is used to determine the first position information of the word to be emphasized in the speech text based on the word to be emphasized input by the user.

[0184] According to one embodiment of this application, the speech synthesis model includes a text encoding module, a prosody predictor, and a speech synthesis module. The prosody predictor includes at least one of a pitch predictor, a volume predictor, and a duration predictor. The text encoding module is used to determine first position information of words to be stressed in the speech text. The prosody predictor is used to perform prosody prediction on the speech text to obtain at least one first prosody prediction result, and to adjust the prosody prediction result of the speech text corresponding to the first position information in the at least one first prosody prediction result to obtain at least one second prosody prediction result. The speech synthesis module is used to generate a speech synthesis result based on the at least one second prosody prediction result.

[0185] It should be understood that the specific working process and functions of the first determining module 1110, the adjusting module 1120, and the generating module 1130 in the above embodiments can be referred to the above. Figures 1 to 10 The description of the speech synthesis method provided in the embodiments will not be repeated here to avoid repetition.

[0186] Figure 12 This is a schematic diagram of the structure of a training device 1200 for a speech synthesis model provided in an exemplary embodiment of this application. Figure 12 As shown, the training device 1200 for the speech synthesis model includes: a second determination module 1210, a calculation module 1220, a regularization module 1230, and a training module 1240.

[0187] The second determining module 1210 is used to determine the sample pitch spectrum of each speaker's speech sample data based on the sample data; the calculation module 1220 is used to calculate the sample mean and sample standard deviation of the sample pitch spectrum; the regularization module 1230 is used to perform regularization processing on the sample pitch spectrum based on the sample mean and sample standard deviation; and the training module 1240 is used to train the speech synthesis model based on the regularized sample pitch spectrum.

[0188] This application provides a training apparatus for a speech synthesis model. Because multi-speaker speech synthesis models (such as the FastPitch multi-speaker model) use mean and variance data derived from all training audio sample data for pitch regularization during training, without distinguishing between speakers, this leads to fundamental frequency crosstalk. Therefore, this application avoids fundamental frequency crosstalk caused by mixed training of multiple speakers by performing pitch regularization processing on the speech sample data of each speaker.

[0189] Figure 13This is a block diagram of an electronic device 1300 for speech synthesis or training a speech synthesis model, provided in an exemplary embodiment of this application.

[0190] Reference Figure 13 The electronic device 1300 includes a processing component 1310, which further includes one or more processors, and memory resources represented by memory 1320 for storing instructions executable by the processing component 1310, such as application programs. The application programs stored in memory 1320 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1310 is configured to execute instructions to perform the aforementioned speech synthesis method or speech synthesis model training method.

[0191] Electronic device 1300 may also include a power supply component configured to perform power management of electronic device 1300, a wired or wireless network interface configured to connect electronic device 1300 to a network, and an input / output (I / O) interface. Electronic device 1300 can be operated based on an operating system stored in memory 1320, such as Windows Server. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0192] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the aforementioned electronic device 1300, enables the electronic device 1300 to execute a speech synthesis method, or a speech synthesis model training method, comprising: determining first position information of a word to be stressed in a speech text, and performing prosodic prediction on the speech text to obtain at least one first prosodic prediction result; adjusting the prosodic prediction result of the speech text corresponding to the first position information in the at least one first prosodic prediction result to obtain at least one second prosodic prediction result, wherein the adjustment includes at least one of raising the pitch, increasing the volume, and decreasing the pronunciation duration; and generating a speech synthesis result based on the at least one second prosodic prediction result. Alternatively, determining the sample pitch spectrum of each speaker's speech sample data based on sample data; calculating the sample mean and sample standard deviation of the sample pitch spectrum; performing regularization processing on the sample pitch spectrum based on the sample mean and sample standard deviation; and training a speech synthesis model based on the regularized sample pitch spectrum.

[0193] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0194] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0195] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0197] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0198] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0199] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program verification codes, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0200] It should be noted that in the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0201] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications or equivalent substitutions made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A speech synthesis method, characterized in that, include: Determine the first position information of the word to be stressed in the speech text, and perform prosodic prediction on the speech text to obtain at least one first prosodic prediction result, wherein the first prosodic prediction result includes at least one of pitch prediction result, volume prediction result and pronunciation duration prediction result; Based on the prosodic prediction results corresponding to the second position information and the prosodic prediction results corresponding to the third position information, the prosodic prediction results of the speech text corresponding to the first position information in the at least one first prosodic prediction results are re-adjusted to obtain at least one second prosodic prediction result. The re-adjustment includes at least one of raising the pitch, increasing the volume, and reducing the pronunciation duration. The second position information is the position information of the word preceding the word to be re-adjusted, and the third position information is the position information of the word following the word to be re-adjusted. Based on the at least one second prosodic prediction result, a speech synthesis result is generated.

2. The speech synthesis method according to claim 1, characterized in that, The step of adjusting the prosodic prediction result of the speech text corresponding to the first position information in the at least one first prosodic prediction result based on the prosodic prediction result corresponding to the second position information and the prosodic prediction result corresponding to the third position information to obtain at least one second prosodic prediction result includes: In the at least one first prosodic prediction result, determine the first pitch prediction result of the speech text corresponding to the first position information; Based on the prosodic prediction results corresponding to the second position information and the prosodic prediction results corresponding to the third position information, a pitch adjustment value is obtained, and based on the first pitch prediction result and the pitch adjustment value, a pitch prediction result with stress effect is obtained.

3. The speech synthesis method according to claim 2, characterized in that, The process of obtaining pitch adjustment values ​​based on the prosodic prediction results corresponding to the second position information and the prosodic prediction results corresponding to the third position information includes: In the at least one first prosodic prediction result, a second tone prediction result for the speech text corresponding to the second position information and a third tone prediction result for the speech text corresponding to the third position information are determined. Calculate the first pitch mean of the first pitch prediction result, the second pitch mean of the second pitch prediction result, and the third pitch mean of the third pitch prediction result; Based on the first pitch mean, the initial pitch adjustment value, and the pre-stored pitch standard deviation of the speaker to be synthesized, the initial stressed pitch mean is obtained; Step a: Based on the first pitch mean, the pitch adjustment value after the Nth update, and the pitch standard deviation of the speaker to be synthesized in advance, obtain the average stressed pitch value after the Nth update, where N≥1; Repeat step a to obtain the average stressed pitch value after the Nth update, until the average stressed pitch value after the Nth update is greater than the average second pitch value, the average third pitch value, and the first pitch prediction result, and less than the pre-stored maximum pitch value of the speaker to be synthesized, and use the pitch adjustment value after the Nth update as the pitch adjustment value.

4. The speech synthesis method according to claim 1, characterized in that, The step of adjusting the prosodic prediction result of the speech text corresponding to the first position information in the at least one first prosodic prediction result based on the prosodic prediction result corresponding to the second position information and the prosodic prediction result corresponding to the third position information to obtain at least one second prosodic prediction result includes: In the at least one first prosodic prediction result, determine the first pronunciation duration prediction result of the speech text corresponding to the first position information; Calculate the mean of the first pronunciation duration in the first pronunciation duration prediction results; Based on the prosodic prediction results corresponding to the second position information and the prosodic prediction results corresponding to the third position information, a reduction factor is obtained, and based on the mean of the first pronunciation duration and the reduction factor, a pronunciation duration prediction result with stress effect is obtained.

5. The speech synthesis method according to claim 4, characterized in that, The process of obtaining the deceleration factor based on the prosodic prediction results corresponding to the second position information and the prosodic prediction results corresponding to the third position information includes: In the at least one first prosodic prediction result, a second pronunciation duration prediction result for the speech text corresponding to the second position information and a third pronunciation duration prediction result for the speech text corresponding to the third position information are determined; Calculate the mean of the second pronunciation duration in the second pronunciation duration prediction result, and the mean of the third pronunciation duration in the third pronunciation duration prediction result; Based on the average duration of the first pronunciation and the initial deceleration factor, the initial average duration of stressed pronunciation is obtained; Step a: Based on the mean of the first pronunciation duration and the reduction factor after the Nth update, obtain the mean of the stressed pronunciation duration after the Nth update, where N≥1; Repeat step a to obtain the average duration of stressed pronunciation after the Nth update, until the average duration of stressed pronunciation after the Nth update is greater than the average duration of the second pronunciation and the average duration of the third pronunciation, and use the Nth update reduction factor as the reduction factor.

6. The speech synthesis method according to claim 1, characterized in that, The step of adjusting the prosodic prediction result of the speech text corresponding to the first position information in the at least one first prosodic prediction result based on the prosodic prediction result corresponding to the second position information and the prosodic prediction result corresponding to the third position information to obtain at least one second prosodic prediction result includes: In the at least one first prosody prediction result, a first volume prediction result for the speech text corresponding to the first position information is determined; Calculate the first volume average value of the first volume prediction result; Based on the prosody prediction results corresponding to the second position information and the prosody prediction results corresponding to the third position information, a volume enhancement factor is obtained, and based on the first volume mean and the volume enhancement factor, a volume prediction result with repetition effect is obtained.

7. The speech synthesis method according to claim 6, characterized in that, The step of obtaining the volume enhancement factor based on the prosody prediction results corresponding to the second position information and the prosody prediction results corresponding to the third position information includes: In the at least one first prosody prediction result, a second volume prediction result for the speech text corresponding to the second position information and a third volume prediction result for the speech text corresponding to the third position information are determined. Calculate the second average volume value of the second volume prediction result and the third average volume value of the third volume prediction result; Based on the initial volume increase factor and the first average volume value, obtain the initial average reread volume value; Step a: Based on the first average volume value and the volume increase factor after the Nth update, obtain the average reread volume after the Nth update, where N≥1; Repeat step a to obtain the average reread volume value after the Nth update, until the average initial reread volume value after the Nth update is greater than the average second volume value and the average third volume value, and less than the pre-stored maximum volume value of the speaker to be synthesized. Then, use the volume increase factor after the Nth update as the volume increase factor.

8. The speech synthesis method according to claim 1, characterized in that, The at least one first prosodic prediction result includes at least one of pitch prediction result, pronunciation duration prediction result, and volume prediction result.

9. The speech synthesis method according to claim 1, characterized in that, The determination of the first position information of the word to be stressed in the speech text includes: Based on the word to be emphasized input by the user, the first position information of the word to be emphasized in the speech text is determined.

10. The speech synthesis method according to any one of claims 1 to 9, characterized in that, The speech synthesis method is implemented by a speech synthesis model, which includes a text encoding module, a prosody predictor, and a speech synthesis module. The prosody predictor includes at least one of a pitch predictor, a volume predictor, and a duration predictor. The text encoding module is used to determine the first position information of the word to be emphasized in the speech text; The prosody predictor is used to perform prosody prediction on the speech text to obtain at least one first prosody prediction result, and to adjust the prosody prediction result of the speech text corresponding to the first position information in the at least one first prosody prediction result based on the prosody prediction result corresponding to the second position information and the prosody prediction result corresponding to the third position information to obtain at least one second prosody prediction result. The speech synthesis module is used to generate the speech synthesis result based on the at least one second prosodic prediction result.

11. A training method for a speech synthesis model, characterized in that, include: Determine the sample pitch spectrum of each speaker's speech sample data based on the sample data; Calculate the sample mean and sample standard deviation of the sample pitch spectrum; The sample pitch spectrum is regularized based on the sample mean and the sample standard deviation. The speech synthesis model is trained based on the regularized sample pitch spectrum; the speech synthesis model is the speech synthesis model as described in claim 10.

12. A speech synthesis device, characterized in that, include: The first determining module is used to determine the first position information of the word to be stressed in the speech text, and to perform prosodic prediction on the speech text to obtain at least one first prosodic prediction result, wherein the first prosodic prediction result includes at least one of pitch prediction result, volume prediction result and pronunciation duration prediction result. An adjustment module is used to adjust the prosodic prediction result of the speech text corresponding to the first position information in the at least one first prosodic prediction result based on the prosodic prediction result corresponding to the second position information and the prosodic prediction result corresponding to the third position information, so as to obtain at least one second prosodic prediction result. The adjustment includes at least one of raising the pitch, increasing the volume, and reducing the pronunciation duration. The second position information is the position information of the word preceding the word to be emphasized, and the third position information is the position information of the word following the word to be emphasized. A generation module is used to generate a speech synthesis result based on the at least one second prosodic prediction result.

13. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the speech synthesis method according to any one of claims 1 to 10, or for executing the training method of the speech synthesis model according to claim 11.

14. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions. The processor is used to execute the speech synthesis method according to any one of claims 1 to 10, or to execute the training method of the speech synthesis model according to claim 11.

Images