Speech synthesis method and device

Abstract

The present specification provides a speech synthesis method and device, wherein the speech synthesis method includes: obtaining pinyin information including a main phoneme and a secondary phoneme; inputting the main phoneme and the secondary phoneme into a speech synthesis model, and processing them through an encoder in the speech synthesis model to obtain a main phoneme coding vector and a secondary phoneme coding vector; splicing the main phoneme coding vector and the secondary phoneme coding vector, and generating a reference phoneme coding vector and an auxiliary phoneme coding vector based on the splicing result; processing the reference phoneme coding vector and the auxiliary phoneme coding vector through a decoder in the speech synthesis model, and generating an audio segment corresponding to the pinyin information based on the decoding processing result.

Description

Technical Field

[0001] This specification relates to the field of speech processing technology, and in particular to a speech synthesis method and device. Background Art

[0002] With the development of internet technology, speech synthesis technology is being applied in more and more scenarios. Erhua, a common pronunciation in modern Chinese, is a sound change caused by the tongue rolling back at the final vowel of a word. Existing techniques generally implement speech synthesis in a two-stage process: first, Chinese pinyin is split into initials and finals, then input into an acoustic model to construct a spectrogram, which is then input into a vocoder for processing to produce a sound signal. While speech synthesis can be achieved for most sentences, most sentences involving erhua ignore the pronunciation of the erhua, which greatly affects the playback quality of the synthesized sound signal. Therefore, an effective solution is urgently needed to address this issue. Summary of the Invention

[0003] In view of this, the embodiments of this specification provide a speech synthesis method. This specification also relates to a speech synthesis device, a computing device, and a computer-readable storage medium to address the technical deficiencies in the prior art.

[0004] According to a first aspect of the embodiments of this specification, there is provided a speech synthesis method, comprising:

[0005] Obtaining pinyin information including main phonemes and secondary phonemes;

[0006] Inputting the primary phoneme and the secondary phoneme into a speech synthesis model, and processing them through an encoder in the speech synthesis model to obtain a primary phoneme encoding vector and a secondary phoneme encoding vector;

[0007] splicing the primary phoneme coding vector and the secondary phoneme coding vector, and generating a reference phoneme coding vector and an auxiliary phoneme coding vector according to the splicing result;

[0008] The reference phoneme coding vector and the auxiliary phoneme coding vector are processed by a decoder in the speech synthesis model, and an audio segment corresponding to the pinyin information is generated according to the decoding processing result.

[0009] Optionally, obtaining pinyin information containing the main phoneme and the secondary phoneme includes:

[0010] Get the text to be processed containing erhua sound;

[0011] Determining pinyin information corresponding to characters associated with the erhua sound in the text to be processed;

[0012] Splitting the pinyin information to obtain initial consonant phonemes, final consonant phonemes, and erhua phonemes corresponding to the characters;

[0013] The initial consonant phoneme and the final phoneme are used as the main phoneme, and the erhua phoneme is used as the secondary phoneme.

[0014] Optionally, before the step of inputting the primary phoneme and the secondary phoneme into the speech synthesis model is performed, the method further includes:

[0015] Constructing the initial consonant feature corresponding to the initial consonant phoneme, the final consonant feature corresponding to the final consonant phoneme, and the erhua sound feature corresponding to the erhua sound phoneme based on a preset mapping relationship;

[0016] Accordingly, the inputting the main phoneme and the secondary phoneme into the speech synthesis model includes:

[0017] The initial consonant feature, the final consonant feature, and the erhua sound feature are input into the speech synthesis model.

[0018] Optionally, the processing by the encoder in the speech synthesis model to obtain the primary phoneme coding vector and the secondary phoneme coding vector includes:

[0019] Encoding the initial consonant feature, the final consonant feature, and the erhua sound feature respectively by the encoder in the speech synthesis model;

[0020] Obtaining, according to the processing results, an initial consonant coding vector corresponding to the initial consonant feature, a final vowel coding vector corresponding to the final vowel feature, and an erhua sound coding vector corresponding to the erhua sound feature;

[0021] The initial consonant encoding vector and the final vowel encoding vector are used as the main phoneme encoding vector, and the erhua sound encoding vector is used as the secondary phoneme encoding vector.

[0022] Optionally, the concatenating the primary phoneme encoding vector and the secondary phoneme encoding vector, and generating a reference phoneme encoding vector and an auxiliary phoneme encoding vector according to the concatenation result, includes:

[0023] splicing the vowel coding vector and the erhua sound coding vector, and generating the auxiliary sound coding vector according to the splicing result;

[0024] The initial consonant encoding vector is used as the reference encoding vector.

[0025] Optionally, processing the reference phoneme encoding vector and the auxiliary phoneme encoding vector by a decoder in the speech synthesis model, and generating an audio segment corresponding to the pinyin information according to a decoding processing result, includes:

[0026] Decoding the reference phoneme encoding vector and the auxiliary phoneme encoding vector by the decoder in the speech synthesis model to obtain a spectrogram corresponding to the pinyin information;

[0027] An audio segment corresponding to the pinyin information is generated based on the spectrogram.

[0028] Optionally, generating an audio segment corresponding to the pinyin information based on the spectrogram includes:

[0029] The spectrogram is input into a vocoder for processing to obtain the audio segment.

[0030] Optionally, after the step of obtaining the text to be processed containing the erhua sound is performed, the method further includes:

[0031] Determining reference pinyin information corresponding to reference characters in the text to be processed;

[0032] Creating a reference audio segment corresponding to the reference character based on the reference pinyin information;

[0033] The reference audio segment and the audio segment are spliced together, and a target audio segment corresponding to the text to be processed is generated according to the splicing result.

[0034] According to a second aspect of the embodiments of this specification, there is provided a speech synthesis apparatus, including:

[0035] An acquisition module is configured to acquire pinyin information including a primary phoneme and a secondary phoneme;

[0036] an encoding module configured to input the primary phoneme and the secondary phoneme into a speech synthesis model, and process them through an encoder in the speech synthesis model to obtain a primary phoneme encoding vector and a secondary phoneme encoding vector;

[0037] a splicing module configured to splice the primary phoneme code vector and the secondary phoneme code vector, and generate a reference phoneme code vector and an auxiliary phoneme code vector according to the splicing result;

[0038] The generation module is configured to process the reference phoneme encoding vector and the auxiliary phoneme encoding vector through the decoder in the speech synthesis model, and generate an audio segment corresponding to the pinyin information according to the decoding processing result.

[0039] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0040] memory and processor;

[0041] The memory is used to store computer-executable instructions, and the processor is used to implement the steps of the speech synthesis method when executing the computer-executable instructions.

[0042] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the speech synthesis method are implemented.

[0043] The speech synthesis method provided in this specification, after obtaining the pinyin information containing the main phoneme and the secondary phoneme, in order to be able to perform speech synthesis on the erhua sound and improve the playback effect of the synthesized audio, the main phoneme and the secondary phoneme can be input into the speech synthesis model, and processed separately by the encoder of the speech synthesis model to obtain the main phoneme coding vector and the secondary phoneme coding vector, and then the two are re-spliced to generate the reference phoneme coding vector and the auxiliary phoneme coding vector, and finally the two are processed by the decoder in the speech synthesis model, and an audio segment of the pinyin information can be created according to the processing results, so that in the speech synthesis process, the erhua sound can also be speech synthesized, so that the synthesized audio segment can play the sound signal with the erhua sound, effectively ensuring the integrity of the speech synthesis, and the synthesis effect of the erhua sound is better and more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of a speech synthesis method provided in one embodiment of this specification;

[0045] Figure 2 is a schematic diagram of a speech synthesis method provided in an embodiment of this specification;

[0046] Figure 3 This is a processing flow chart of a speech synthesis method applied to an erhua sound speech synthesis scenario provided by an embodiment of this specification;

[0047] Figure 4 This is a structural diagram of a speech synthesis device provided in one embodiment of this specification;

[0048] Figure 5 This is a structural block diagram of a computing device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0049] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0050] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0051] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0052] First, the terms involved in one or more embodiments of this specification are explained.

[0053] Erhua (erhua) is a sound change in the final vowel of some words in modern Mandarin and most Mandarin dialects, caused by rolling the tongue. An "r" is usually added after the vowel to indicate that the pronunciation can be erhua.

[0054] Pinyin tone: refers to the tones in Mandarin, usually called four tones, namely yinping (first tone), represented by "ˉ", corresponding to the number 1, such as lā, represented as la1; yangping (second tone), represented by "ˊ", corresponding to the number 2, such as lá, represented as la2; shangsheng (third tone), represented by "ˇ", corresponding to the number 3, such as lǎ, represented as la3; qusheng (fourth tone), represented by "ˋ", corresponding to the number 4, such as; là, represented as la4; in addition, it also includes light tone, in which case the final vowel is represented by 5, such as le5.

[0055] In this specification, a speech synthesis method is provided. This specification also relates to a speech synthesis device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0056] In actual applications, when performing speech synthesis on text containing erhua sounds, the pinyin of the erhua sound is usually represented by r, and then r and the final vowel of the previous character are input into the speech synthesis model as a whole for processing. The erhua sound data in the data is relatively sparse, resulting in the inability of the audio after speech synthesis to pronounce the erhua sound, resulting in the synthesized audio being inaccurate. Therefore, an effective solution is urgently needed to solve the above problem.

[0057] The speech synthesis method provided in this specification, after obtaining the pinyin information containing the main phoneme and the secondary phoneme, in order to be able to perform speech synthesis on erhua sounds as well and improve the playback effect of the synthesized audio, at this time, the main phoneme and the secondary phoneme can be input into the speech synthesis model, and they are respectively processed by the encoder of the speech synthesis model to obtain the main phoneme encoding vector and the secondary phoneme encoding vector. Then, the two are re - spliced to generate the reference phoneme encoding vector and the auxiliary phoneme encoding vector. Finally, the decoder in the speech synthesis model processes the two, and an audio segment of the pinyin information can be created according to the processing result, realizing that in the speech synthesis process, erhua sounds can also be synthesized, so that the synthesized audio segment can play the sound signal with erhua sounds, effectively ensuring the integrity of speech synthesis, and the synthesis effect of erhua sounds is better and more stable.

[0058] Figure 1 The flowchart of a speech synthesis method provided according to an embodiment of this specification is shown, which specifically includes the following steps:

[0059] Step S102, obtain the pinyin information containing the main phoneme and the secondary phoneme.

[0060] Specifically, the pinyin information specifically refers to the pinyin corresponding to the characters that need to perform speech synthesis, and the pinyin information contains the pinyin corresponding to the erhua sound; correspondingly, the main phoneme is specifically the initial and final sounds of the character corresponding to the pinyin information, and the secondary phoneme is specifically the phoneme corresponding to the erhua sound.

[0061] In practical applications, as a phoneme that also needs to be pronounced, when performing speech synthesis, if the pronunciation of the erhua sound is ignored, it may affect the integrity of speech synthesis; for example, for "laotour", its corresponding pinyin information is lao3, tour3 (where r represents the erhua sound). When performing speech synthesis, the erhua sound will be ignored, resulting in the synthesized speech only corresponding to "laotou"; or the erhua sound is synthesized with the third tone. Although the synthesized speech contains the three characters "laotour", the "r" is represented by a single byte and is not an erhua sound, resulting in a more rigid synthesized speech effect and unable to meet the user's usage requirements.

[0062] Based on this, in order to be able to perform speech synthesis on erhua sounds during speech synthesis, each phoneme will be encoded by the speech synthesis model respectively, and after encoding, the erhua sound and the final sound will be spliced. Then, the splicing result will be decoded to realize that the model outputs a spectrogram containing the erhua sound. Then, an audio segment containing the erhua sound can be created through the vocoder, making the final synthesis effect of the erhua sound better and more stable.

[0063] Further, when obtaining the pinyin information, in order to ensure that the synthesized speech contains the acoustic signals corresponding to erhua tones, the phonemes corresponding to the erhua tones will be separately split during the preprocessing stage. In this embodiment, the specific implementation method is as follows:

[0064] Obtain the text to be processed containing erhua tones; determine the pinyin information corresponding to the characters in the text to be processed that have an associated relationship with the erhua tones; split the pinyin information to obtain the initial phoneme, final phoneme, and erhua tone phoneme corresponding to the character; use the initial phoneme and the final phoneme as the main phonemes, and the erhua tone phoneme as the secondary phoneme.

[0065] Specifically, the text to be processed specifically refers to the text used for speech synthesis, and the audio after speech synthesis corresponds to the text content in the text to be processed; correspondingly, the characters having an associated relationship with the erhua tones specifically refer to the characters adjacent to the erhua tone before pronunciation; the initial phoneme specifically refers to the initial in the pinyin of the character, the final phoneme specifically refers to the final in the pinyin of the character, and the erhua tone phoneme specifically refers to the pinyin corresponding to the erhua tone.

[0066] Based on this, after obtaining the text to be processed containing erhua tones, in order to be able to output the audio containing erhua tones through speech synthesis, the pinyin information corresponding to the characters in the text to be processed that have an associated relationship with the erhua tones can be determined; then the pinyin information is split to obtain the initial phoneme and final phoneme corresponding to the character. Since the character is adjacent to the erhua tone, and the pinyin of the erhua tone is usually connected to the character at this position, the erhua tone phoneme corresponding to the split erhua tone is also obtained at the same time. Then, the initial phoneme and the final phoneme are used as the main phonemes, and the erhua tone phoneme is used as the secondary phoneme for subsequent speech synthesis processing.

[0067] Illustrate with an example. Refer to Figure 2 the schematic diagram shown. Receive the text to be processed uploaded by the user. Through semantic analysis of the text to be processed, it is determined that the character having an associated relationship with the erhua tone in the text to be processed is "明", the content containing the erhua tone is "明儿", and its corresponding pinyin information is "mingr2". By splitting the pinyin information, the initial phoneme m, the final phoneme ing2, and the erhua tone phoneme er6 are obtained for subsequent speech synthesis processing to obtain the audio segment corresponding to "明儿".

[0068] In summary, by splitting the pinyin corresponding to the characters having an associated relationship with the erhua tone and constructing the main phonemes and secondary phonemes in units of phonemes, it can effectively ensure that the erhua tone is separately treated as a phoneme for subsequent processing, so that when performing speech synthesis, considering the pronunciation characteristics of the erhua tone, an audio segment containing the erhua tone is output.

[0069] Step S104: input the primary phoneme and the secondary phoneme into a speech synthesis model, and process them through an encoder in the speech synthesis model to obtain a primary phoneme encoding vector and a secondary phoneme encoding vector.

[0070] Specifically, based on the above-mentioned acquisition of the main phoneme and the secondary phoneme, the main phoneme and the secondary phoneme can be further input into the speech synthesis model for processing, so as to complete the speech synthesis processing through the speech synthesis model.

[0071] The speech synthesis model can use the FastSpeech neural network, a feedforward structure based on the self-attention and 1D convolution in the Transformer. This network encodes and decodes each phoneme separately to complete the speech synthesis process. Accordingly, the primary phoneme encoding vector refers to the vector representation obtained by encoding the primary phoneme through the encoder in the speech synthesis model, and the secondary phoneme encoding vector refers to the vector representation obtained by encoding the secondary phoneme through the encoder in the speech synthesis model.

[0072] Furthermore, when the speech synthesis model processes the primary and secondary phonemes corresponding to the pinyin information, in order to include the erhua sound in the synthesized audio clip, each phoneme is encoded. In this embodiment, the specific implementation is as follows:

[0073] Based on a preset mapping relationship, the initial consonant feature corresponding to the initial consonant phoneme, the final consonant feature corresponding to the final consonant phoneme and the erhua sound feature corresponding to the erhua sound phoneme are constructed; and the initial consonant feature, the final consonant feature and the erhua sound feature are input into the speech synthesis model.

[0074] Specifically, the initial consonant feature refers to the vector expression corresponding to the initial consonant obtained after the initial consonant phoneme is converted, the final vowel feature refers to the vector expression corresponding to the final vowel obtained after the final vowel phoneme is converted, and the erhua sound feature refers to the vector expression corresponding to the erhua sound obtained after the erhua sound phoneme is converted.

[0075] Based on this, in order to support the speech synthesis model to encode and decode each phoneme separately, it is necessary to construct the main phonemes and the secondary phonemes as the same vector expressions as the speech synthesis model input. Therefore, based on the preset mapping relationship, the initial consonant features corresponding to the initial consonant phonemes, the final consonant features corresponding to the final consonant phonemes, and the erhua sound features corresponding to the erhua sound phonemes can be constructed. Then, the three can be input into the speech synthesis model to perform speech synthesis processing.

[0076] It should be noted that the preset mapping relationship refers to the pre-established relationship between each phoneme and the corresponding vector expression. Through this relationship, the vector expressions corresponding to the initial consonant, final vowel and erhua sound can be directly read, which can effectively improve the efficiency of constructing the vector expression.

[0077] In summary, by constructing the initial consonant features, final vowel features and erhua sound features, the phonemes can be directly processed after being input into the speech synthesis model, which effectively improves the speech synthesis efficiency and quickly completes the audio generation.

[0078] Furthermore, after the initial consonant features, final vowel features, and erhua sound features are input into the speech synthesis model, in order to reflect the erhua sound in the subsequently generated audio, each phoneme will be encoded separately. In this embodiment, the specific implementation method is as follows:

[0079] The encoder in the speech synthesis model encodes the initial consonant feature, the final consonant feature and the erhua sound feature respectively; according to the processing results, the initial consonant coding vector corresponding to the initial consonant feature, the final consonant coding vector corresponding to the final consonant feature and the erhua sound coding vector corresponding to the erhua sound feature are obtained; the initial consonant coding vector and the final consonant coding vector are used as the main phoneme coding vector, and the erhua sound coding vector is used as the secondary phoneme coding vector.

[0080] Specifically, the initial consonant encoding vector refers to the vector expression obtained after the encoder in the speech synthesis model encodes the initial consonant features, the final vowel encoding vector refers to the vector expression obtained after the encoder in the speech synthesis model encodes the final vowel features, and the erhua sound encoding vector refers to the vector expression obtained after the encoder in the speech synthesis model encodes the erhua sound features.

[0081] Specifically, based on the above-mentioned initial consonant features, final consonant features and erhua sound features, the three can be further input into the speech synthesis model at this time, and each feature can be encoded separately through the encoding layer in the speech synthesis model to obtain the initial consonant coding vector corresponding to the initial consonant feature, the final consonant coding vector corresponding to the final consonant feature, and the erhua sound coding vector corresponding to the erhua sound feature. The initial consonant coding vector and the final consonant coding vector are used as the main phoneme coding vector, the erhua sound coding vector and the secondary phoneme coding vector, so as to be used for decoding the spectrum diagram containing the erhua sound in the subsequent decoding stage.

[0082] Continuing with the above example, after obtaining the initial consonant phoneme m, the final vowel phoneme ing2, and the erhua sound phoneme er6, the initial consonant feature A corresponding to the initial consonant phoneme m, the final vowel feature B corresponding to the final vowel phoneme ing2, and the erhua sound feature C corresponding to the erhua sound phoneme er6 can be constructed based on the preset mapping relationship. Furthermore, the initial consonant feature A, the final vowel feature B, and the erhua sound feature C are respectively input into the encoding layer of the speech synthesis model, and each feature is encoded by the encoding layer to obtain the initial consonant encoding vector A'' corresponding to the initial consonant phoneme m, the final vowel encoding vector B'' corresponding to the final vowel phoneme ing2, and the erhua sound encoding vector C'' corresponding to the erhua sound phoneme er6, so as to be used for subsequent decoding to obtain a spectrum diagram containing the erhua sound.

[0083] In summary, the encoder in the speech synthesis model encodes each phoneme feature separately, so that the vector expression of erhua sound is taken into account during speech synthesis, so that the model can output a spectrum graph containing erhua sound information, so as to include erhua sound information during audio generation.

[0084] Step S106 , concatenating the primary phoneme coding vector and the secondary phoneme coding vector, and generating a reference phoneme coding vector and an auxiliary phoneme coding vector according to the concatenation result.

[0085] Specifically, based on the main phoneme coding vector and the secondary phoneme coding vector obtained above, in order to be able to integrate the erhua sound into the audio during the decoding stage, the baseline phoneme coding vector and the auxiliary phoneme coding vector can be re-spliced based on the main phoneme coding vector and the secondary phoneme coding vector, so as to realize the integration of the vector expression of the erhua sound into the final without changing the vector expression of the phoneme.

[0086] Furthermore, since the encoder encodes the initial consonant features, final consonant features, and erhua features separately, three encoding vectors are obtained. Erhua is usually produced in combination with a final consonant. Therefore, the encoding vector of the erhua can be integrated into the encoding vector of the final consonant. In this embodiment, the specific implementation is as follows:

[0087] The final vowel coding vector and the erhua sound coding vector are spliced, and the auxiliary sound coding vector is generated according to the splicing result; and the initial consonant coding vector is used as the reference coding vector.

[0088] Specifically, the primary phoneme coding vector refers to the initial consonant coding vector, and the auxiliary phoneme coding vector refers to the vector representation of the combination of the erhua coding vector and the final coding vector. Based on this, after obtaining the initial consonant coding vector, final vowel coding vector, and erhua coding vector output by the encoder in the speech synthesis model, the final vowel coding vector and erhua coding vector can be spliced to generate auxiliary coding vectors corresponding to the final vowel and erhua. At the same time, the initial consonant coding vector is used as the reference coding vector for subsequent decoding processing.

[0089] Continuing with the above example, after obtaining the initial consonant coding vector A” corresponding to the initial consonant phoneme m, the final vowel coding vector B” corresponding to the final vowel phoneme ing2, and the erhua sound coding vector C” corresponding to the erhua sound phoneme er6, the initial consonant coding vector A” corresponding to the initial consonant phoneme m is used as the baseline coding vector; at the same time, the final vowel coding vector B” and the erhua sound coding vector C” are spliced, and the auxiliary coding vector B”+C” is obtained according to the splicing result, which is used for the subsequent output of the audio clip containing the erhua sound through the speech synthesis model.

[0090] In summary, by reattaching the encoding vector of the erhua sound to the encoding vector of the final, the erhua sound is not decoded separately during the decoding stage, which can ensure that the output spectrogram is more stable and contains the information of the erhua sound.

[0091] Step S108: Process the reference phoneme encoding vector and the auxiliary phoneme encoding vector through the decoder in the speech synthesis model, and generate an audio segment corresponding to the pinyin information according to the decoding processing result.

[0092] Specifically, based on the above-mentioned reference coding vector and auxiliary coding vector, in order to be able to output an audio clip containing erhua sounds, the two will be decoded simultaneously by the decoder in the speech synthesis model, and then the audio clip will be obtained based on the decoding results; accordingly, the audio clip specifically refers to the acoustic signal corresponding to the pinyin information. By playing the audio clip, you can hear the same content as the characters corresponding to the pinyin information, and the content contains the pronunciation of erhua sounds.

[0093] Furthermore, after the encoding process, the reference phoneme encoding vector and the auxiliary phoneme encoding vector are decoded simultaneously by a decoder. At this time, a spectrogram output by the speech synthesis model is obtained. In order to be able to convert it into an audio clip, the spectrogram can be input into a vocoder for processing. In this embodiment, the specific implementation method is as follows:

[0094] The decoder in the voice synthesis model decodes the reference phoneme encoding vector and the auxiliary phoneme encoding vector to obtain the spectrogram corresponding to the pinyin information; the spectrogram is input into a vocoder for processing to obtain the audio clip.

[0095] Specifically, the spectrogram specifically refers to the form in which a sound signal is expressed by a two-dimensional signal. Based on this, after obtaining the reference encoding vector and the auxiliary encoding vector, the decoder in the voice synthesis model can decode the reference phoneme encoding vector and the auxiliary phoneme encoding vector to obtain the spectrogram output by the voice synthesis model according to the decoding result; then input it into the vocoder for processing to obtain the audio clip corresponding to the pinyin information.

[0096] Among them, a vocoder is a voice analysis and synthesis system of a certain model of voice signal. Only the model parameters are used in transmission, and the voice signal codec that uses model parameter estimation and voice synthesis technology during encoding and decoding is a codec for analyzing and synthesizing voice, also known as a voice analysis and synthesis system or a voice frequency band compression system. It is a powerful tool for compressing communication frequency bands and performing secure communication.

[0097] Continuing with the above example, after obtaining the reference encoding vector (initial consonant encoding vector A”) and the auxiliary encoding vector B” + C”, the two can be input into the decoder in the voice synthesis model for processing to generate a spectrogram as shown in Figure 2 Then input the spectrogram into the vocoder for voice conversion to obtain the audio clip corresponding to the content “tomorrow” with erhua sound. When this audio clip is played, it will contain the pronunciation of the erhua sound.

[0098] In summary, fusing the encoding vector corresponding to the erhua sound into the final vowel encoding vector in the decoding stage can effectively improve the stability and adaptability of the model, and can output a spectrogram containing the erhua sound, making the synthesized audio effect more stable.

[0099] In addition, since the text to be processed contains other characters in addition to the characters corresponding to the erhua sound, it is necessary to perform voice synthesis on these other characters and then splice the audio corresponding to the pinyin information to obtain the audio corresponding to the text to be processed. In this embodiment, the specific implementation method is as follows:

[0100] Determine the reference pinyin information corresponding to the reference characters in the text to be processed; create a reference audio clip corresponding to the reference characters based on the reference pinyin information; splice the reference audio clip and the audio clip, and generate the target audio clip corresponding to the text to be processed according to the splicing result.

[0101] Specifically, the benchmark characters specifically refer to the other characters that need to be pronounced in the text to be processed except the characters associated with the erhua sound. Correspondingly, the benchmark pinyin information specifically refers to the pinyin information corresponding to the other characters that need to be pronounced; correspondingly, the benchmark audio segment specifically refers to the audio corresponding to the other characters that need to be pronounced; the target audio segment specifically refers to the acoustic signal corresponding to the text to be processed, which includes the audio segments corresponding to all characters.

[0102] Based on this, while obtaining the audio clip containing the erhua sound, it is also necessary to simultaneously determine the benchmark pinyin information corresponding to the benchmark characters in the text to be processed, and then create a benchmark audio clip corresponding to the benchmark characters based on the benchmark pinyin information. The benchmark audio clip and the audio clip are then spliced according to the time information to obtain the target audio clip corresponding to the text to be processed.

[0103] It should be noted that the generation of the reference audio segment can be achieved by combining a speech synthesis model and a vocoder, or by using other speech synthesis technologies, which are not limited in this embodiment; and the process of generating the reference audio segment can be achieved simultaneously with the process of generating the audio segment containing the erhua sound. That is to say, after obtaining the text to be processed, the speech synthesis model + vocoder can be used to perform speech synthesis processing on normal characters (words that are not read with erhua sounds) and characters with associated erhua sounds (words that are read with erhua sounds) at the same time to obtain the audio segment corresponding to each character in the text to be processed. After that, the audio segments are spliced according to the order of arrangement of the characters in the text to be processed to obtain the target audio segment corresponding to the text to be processed.

[0104] Continuing with the above example, the text to be processed is "Let's go traveling tomorrow". While creating the audio clip corresponding to "tomorrow", the benchmark pinyin information corresponding to the benchmark characters (let's go traveling) in the text to be processed can be determined {zan-zan3, qu-qu4, lv-lv3 and you-you2}. Then, based on the benchmark pinyin information, the benchmark audio clips corresponding to the benchmark characters are created. The audio clip corresponding to "tomorrow" and the benchmark audio clip are then spliced together to obtain the target audio clip of the text to be processed "Let's go traveling tomorrow".

[0105] The speech synthesis method provided in this specification, after obtaining the pinyin information containing the main phoneme and the secondary phoneme, in order to be able to perform speech synthesis on the erhua sound and improve the playback effect of the synthesized audio, the main phoneme and the secondary phoneme can be input into the speech synthesis model, and processed separately by the encoder of the speech synthesis model to obtain the main phoneme coding vector and the secondary phoneme coding vector, and then the two are re-spliced to generate the reference phoneme coding vector and the auxiliary phoneme coding vector, and finally the two are processed by the decoder in the speech synthesis model, and an audio segment of the pinyin information can be created according to the processing results, so that in the speech synthesis process, the erhua sound can also be speech synthesized, so that the synthesized audio segment can play the sound signal with the erhua sound, effectively ensuring the integrity of the speech synthesis, and the synthesis effect of the erhua sound is better and more stable.

[0106] The following combined Figure 3 , taking the application of the speech synthesis method provided in this specification in the erhua sound speech synthesis scenario as an example, the speech synthesis method is further explained. Figure 3 A processing flow chart of a speech synthesis method applied to an erhua sound speech synthesis scenario provided by an embodiment of this specification is shown, which specifically includes the following steps:

[0107] Step S302: obtaining a text to be processed containing erhua sounds, and determining pinyin information corresponding to characters associated with erhua sounds in the text to be processed.

[0108] Step S304: split the pinyin information to obtain the initial consonant phonemes, final phonemes and erhua phonemes corresponding to the characters.

[0109] Step S306 , constructing initial consonant features corresponding to the initial consonant phonemes, final consonant features corresponding to the final consonant phonemes, and erhua features corresponding to the erhua phonemes based on the preset mapping relationship.

[0110] Step S308: input the initial consonant features, final vowel features, and erhua sound features into the speech synthesis model.

[0111] Step S310: Encode the initial consonant features, final consonant features, and erhua sound features respectively through the encoder in the speech synthesis model.

[0112] Step S312: obtaining, based on the processing results, an initial consonant coding vector corresponding to the initial consonant feature, a final vowel coding vector corresponding to the final vowel feature, and an erhua tone coding vector corresponding to the erhua tone feature.

[0113] Step S314: concatenate the vowel coding vector and the erhua coding vector to obtain an auxiliary sound coding vector, and use the initial consonant coding vector as a reference coding vector.

[0114] Step S316: decode the reference phoneme coding vector and the auxiliary phoneme coding vector through the decoder in the speech synthesis model to obtain a spectrogram.

[0115] Step S318: input the spectrogram into a vocoder for processing to obtain an audio segment.

[0116] Step S320 , determining reference pinyin information corresponding to reference characters in the text to be processed, and creating reference audio segments corresponding to the reference characters based on the reference pinyin information.

[0117] Step S322: splice the reference audio segment and the audio segment, and generate a target audio segment corresponding to the text to be processed according to the splicing result.

[0118] To sum up, after obtaining the pinyin information containing the main phonemes and the secondary phonemes, in order to be able to perform speech synthesis on the erhua sound and improve the playback effect of the synthesized audio, the main phonemes and the secondary phonemes can be input into the speech synthesis model, and processed separately by the encoder of the speech synthesis model to obtain the main phoneme coding vector and the secondary phoneme coding vector. Then, the two are re-spliced to generate the benchmark phoneme coding vector and the auxiliary phoneme coding vector. Finally, the two are processed by the decoder in the speech synthesis model. According to the processing results, an audio clip of the pinyin information can be created, so that in the speech synthesis process, the erhua sound can also be synthesized, so that the synthesized audio clip can play the sound signal with the erhua sound, which effectively ensures the integrity of the speech synthesis, and the synthesis effect of the erhua sound is better and more stable.

[0119] Corresponding to the above method embodiment, this specification also provides a speech synthesis device embodiment, Figure 4 FIG. 1 shows a schematic diagram of the structure of a speech synthesis device provided in an embodiment of this specification. Figure 4 As shown, the device includes:

[0120] An acquisition module 402 is configured to acquire pinyin information including a primary phoneme and a secondary phoneme;

[0121] The encoding module 404 is configured to input the primary phoneme and the secondary phoneme into a speech synthesis model, and process them through an encoder in the speech synthesis model to obtain a primary phoneme encoding vector and a secondary phoneme encoding vector;

[0122] a concatenation module 406 configured to concatenate the primary phoneme code vector and the secondary phoneme code vector, and generate a reference phoneme code vector and an auxiliary phoneme code vector according to the concatenation result;

[0123] The generation module 408 is configured to process the reference phoneme encoding vector and the auxiliary phoneme encoding vector through the decoder in the speech synthesis model, and generate an audio segment corresponding to the pinyin information according to the decoding processing result.

[0124] In an optional embodiment, the acquisition module 402 is further configured to:

[0125] Acquire a text to be processed containing an erhua sound; determine the pinyin information corresponding to the characters associated with the erhua sound in the text to be processed; split the pinyin information to obtain the initial consonant phonemes, final consonant phonemes and erhua sound phonemes corresponding to the characters; use the initial consonant phonemes and the final consonant phonemes as the main phonemes, and the erhua sound phonemes as the secondary phonemes.

[0126] In an optional embodiment, the speech synthesis device further includes:

[0127] A construction module is configured to construct, based on a preset mapping relationship, the initial consonant feature corresponding to the initial consonant phoneme, the final consonant feature corresponding to the final consonant phoneme, and the erhua sound feature corresponding to the erhua sound phoneme;

[0128] Accordingly, the encoding module 404 is further configured to:

[0129] The initial consonant feature, the final consonant feature, and the erhua sound feature are input into the speech synthesis model.

[0130] In an optional embodiment, the encoding module 404 is further configured to:

[0131] The encoder in the speech synthesis model encodes the initial consonant feature, the final consonant feature and the erhua sound feature respectively; according to the processing results, the initial consonant coding vector corresponding to the initial consonant feature, the final consonant coding vector corresponding to the final consonant feature and the erhua sound coding vector corresponding to the erhua sound feature are obtained; the initial consonant coding vector and the final consonant coding vector are used as the main phoneme coding vector, and the erhua sound coding vector is used as the secondary phoneme coding vector.

[0132] In an optional embodiment, the splicing module 406 is further configured to:

[0133] The final vowel coding vector and the erhua sound coding vector are spliced, and the auxiliary sound coding vector is generated according to the splicing result; and the initial consonant coding vector is used as the reference coding vector.

[0134] In an optional embodiment, the generating module 408 is further configured to:

[0135] The reference phoneme coding vector and the auxiliary phoneme coding vector are decoded by the decoder in the speech synthesis model to obtain a spectrogram corresponding to the pinyin information; and an audio segment corresponding to the pinyin information is generated based on the spectrogram.

[0136] In an optional embodiment, the generating module 408 is further configured to:

[0137] The spectrogram is input into a vocoder for processing to obtain the audio segment.

[0138] In an optional embodiment, the speech synthesis device further includes:

[0139] The determination module is configured to determine the benchmark pinyin information corresponding to the benchmark characters in the text to be processed; create a benchmark audio segment corresponding to the benchmark characters based on the benchmark pinyin information; splice the benchmark audio segment and the audio segment, and generate a target audio segment corresponding to the text to be processed based on the splicing result.

[0140] The speech synthesis device provided in this specification, after obtaining the pinyin information containing the main phoneme and the secondary phoneme, in order to be able to perform speech synthesis on the erhua sound and improve the playback effect of the synthesized audio, the main phoneme and the secondary phoneme can be input into the speech synthesis model, and processed separately by the encoder of the speech synthesis model to obtain the main phoneme coding vector and the secondary phoneme coding vector, and then the two are re-spliced to generate the reference phoneme coding vector and the auxiliary phoneme coding vector, and finally the two are processed by the decoder in the speech synthesis model, and an audio segment of the pinyin information can be created according to the processing results, so that in the speech synthesis process, the erhua sound can also be speech synthesized, so that the synthesized audio segment can play the sound signal with the erhua sound, effectively ensuring the integrity of the speech synthesis, and the synthesis effect of the erhua sound is better and more stable.

[0141] The above is a schematic diagram of a speech synthesis device according to this embodiment. It should be noted that the technical solution of the speech synthesis device and the technical solution of the speech synthesis method described above are based on the same concept. For details not described in detail in the technical solution of the speech synthesis device, please refer to the description of the technical solution of the speech synthesis method described above.

[0142] Figure 5 1 shows a block diagram of a computing device 500 according to an embodiment of the present disclosure. Components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0143] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)), whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0144] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 5 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0145] Computing device 500 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 500 can also be a mobile or stationary server.

[0146] The processor 520 is configured to execute the following computer-executable instructions:

[0147] Obtaining pinyin information including main phonemes and secondary phonemes;

[0148] Inputting the primary phoneme and the secondary phoneme into a speech synthesis model, and processing them through an encoder in the speech synthesis model to obtain a primary phoneme encoding vector and a secondary phoneme encoding vector;

[0149] splicing the primary phoneme coding vector and the secondary phoneme coding vector, and generating a reference phoneme coding vector and an auxiliary phoneme coding vector according to the splicing result;

[0150] The reference phoneme coding vector and the auxiliary phoneme coding vector are processed by a decoder in the speech synthesis model, and an audio segment corresponding to the pinyin information is generated according to the decoding processing result.

[0151] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above-mentioned speech synthesis method are based on the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-mentioned speech synthesis method.

[0152] An embodiment of the present specification further provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to:

[0153] Obtaining pinyin information including main phonemes and secondary phonemes;

[0154] Inputting the primary phoneme and the secondary phoneme into a speech synthesis model, and processing them through an encoder in the speech synthesis model to obtain a primary phoneme encoding vector and a secondary phoneme encoding vector;

[0155] splicing the primary phoneme coding vector and the secondary phoneme coding vector, and generating a reference phoneme coding vector and an auxiliary phoneme coding vector according to the splicing result;

[0156] The reference phoneme coding vector and the auxiliary phoneme coding vector are processed by a decoder in the speech synthesis model, and an audio segment corresponding to the pinyin information is generated according to the decoding processing result.

[0157] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the speech synthesis method described above are based on the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the speech synthesis method described above.

[0158] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0159] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0160] It should be noted that for the aforementioned method embodiments, for ease of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that this specification is not limited to the order of the actions described, because according to this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this specification.

[0161] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0162] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of this specification, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A speech synthesis method, characterized in that: include: Acquire pinyin information including main phonemes and secondary phonemes, wherein the main phonemes are initial consonant phonemes and final consonant phonemes, and the secondary phonemes are erhua phonemes; Inputting the primary phoneme and the secondary phoneme into a speech synthesis model, and processing them through an encoder in the speech synthesis model to obtain a primary phoneme encoding vector and a secondary phoneme encoding vector; Splicing the primary phoneme coding vector and the secondary phoneme coding vector, and generating a reference phoneme coding vector and an auxiliary phoneme coding vector according to the splicing result, including: splicing the vowel coding vector and the erhua code vector, and generating the auxiliary phoneme coding vector according to the splicing result; and using the initial consonant coding vector as the reference phoneme coding vector; The reference phoneme coding vector and the auxiliary phoneme coding vector are processed by a decoder in the speech synthesis model, and an audio segment corresponding to the pinyin information is generated according to the decoding processing result.

2. The speech synthesis method according to claim 1, wherein: The obtaining of pinyin information including the main phoneme and the secondary phoneme includes: Get the text to be processed containing erhua sound; Determining pinyin information corresponding to characters associated with the erhua sound in the text to be processed; Splitting the pinyin information to obtain initial consonant phonemes, final consonant phonemes, and erhua phonemes corresponding to the characters; The initial consonant phoneme and the final phoneme are used as the main phonemes, and the erhua phoneme is used as the secondary phoneme.

3. The speech synthesis method according to claim 2, wherein: Before the step of inputting the primary phoneme and the secondary phoneme into the speech synthesis model is performed, the method further includes: Constructing the initial consonant feature corresponding to the initial consonant phoneme, the final consonant feature corresponding to the final consonant phoneme, and the erhua sound feature corresponding to the erhua sound phoneme based on a preset mapping relationship; Accordingly, the inputting the main phoneme and the secondary phoneme into the speech synthesis model includes: The initial consonant feature, the final consonant feature, and the erhua sound feature are input into the speech synthesis model.

4. The speech synthesis method according to claim 3, wherein: The processing by the encoder in the speech synthesis model to obtain the primary phoneme coding vector and the secondary phoneme coding vector includes: Encoding the initial consonant feature, the final consonant feature, and the erhua sound feature respectively by the encoder in the speech synthesis model; Obtaining, according to the processing results, an initial consonant coding vector corresponding to the initial consonant feature, a final vowel coding vector corresponding to the final vowel feature, and an erhua sound coding vector corresponding to the erhua sound feature; The initial consonant encoding vector and the final vowel encoding vector are used as the main phoneme encoding vector, and the erhua sound encoding vector is used as the secondary phoneme encoding vector.

5. The speech synthesis method according to claim 1, wherein: The processing of the reference phoneme coding vector and the auxiliary phoneme coding vector by a decoder in the speech synthesis model, and generating an audio segment corresponding to the pinyin information according to a decoding processing result, includes: Decoding the reference phoneme encoding vector and the auxiliary phoneme encoding vector by the decoder in the speech synthesis model to obtain a spectrogram corresponding to the pinyin information; An audio segment corresponding to the pinyin information is generated based on the spectrogram.

6. The speech synthesis method according to claim 5, characterized in that Generating an audio segment corresponding to the pinyin information based on the spectrogram includes: The spectrogram is input into a vocoder for processing to obtain the audio segment.

7. The speech synthesis method according to any one of claims 2 to 6, characterized in that: After the step of obtaining the text to be processed containing the erhua sound is executed, the method further includes: Determining reference pinyin information corresponding to reference characters in the text to be processed; Creating a reference audio segment corresponding to the reference character based on the reference pinyin information; The reference audio segment and the audio segment are spliced together, and a target audio segment corresponding to the text to be processed is generated according to the splicing result.

8. A speech synthesis device, characterized in that: include: An acquisition module is configured to acquire pinyin information including a main phoneme and a secondary phoneme, wherein the main phoneme is an initial phoneme and a final phoneme, and the secondary phoneme is an erhua phoneme; an encoding module configured to input the primary phoneme and the secondary phoneme into a speech synthesis model, and process them through an encoder in the speech synthesis model to obtain a primary phoneme encoding vector and a secondary phoneme encoding vector; a splicing module configured to splice the primary phoneme code vector and the secondary phoneme code vector, and generate a reference phoneme code vector and an auxiliary phoneme code vector according to the splicing result; The concatenation module is further configured to concatenate the vowel code vector and the erhua code vector, and generate the auxiliary phoneme code vector according to the concatenation result; and use the initial consonant code vector as the reference phoneme code vector; The generation module is configured to process the reference phoneme encoding vector and the auxiliary phoneme encoding vector through the decoder in the speech synthesis model, and generate an audio segment corresponding to the pinyin information according to the decoding processing result.

9. A computing device, characterized in that It comprises a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the speech synthesis method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by the processor, the steps of the speech synthesis method described in any one of claims 1 to 7 are implemented.

Images