Training method and device of speech synthesis system and speech synthesis method and device

Abstract

The present invention proposes a training method and device for a speech synthesis system, and a speech synthesis method and device, which belong to the field of speech human-computer interaction. The method adopts independent modeling of initials, finals and tones, so that the speech synthesis system only needs less training corpus to cover all modeling units. The method comprises: collecting the training speech of the target speaker to form training audio, and performing text annotation on the training audio; converting the obtained text annotation into pronunciation unit annotation, the pronunciation unit annotation includes the corresponding initials and finals with tones; disassembling the pronunciation unit annotation and converting it into pronunciation unit vectors and tone vectors without tones; and forming training samples for training the speech synthesis system based on the training audio, pronunciation unit vectors and tone vectors. The present invention adopts independent modeling of initials, finals and tones, so that the speech synthesis system only needs less training corpus to cover all modeling units.

Description

Technical Field

[0001] The present invention belongs to the field of speech human-computer interaction, and in particular relates to a training method and device for a speech synthesis system, and a speech synthesis method and device. Background Art

[0002] Voice interaction is gradually recognized by people and widely used in various life scenarios due to its natural and friendly interaction mode, such as intelligent robots, car voice, smart TV and audio smart toys, virtual people, etc. Speech synthesis technology, namely TTS technology, represents the speaking ability and expression ability of smart devices, and is an important technology in the field of voice interaction.

[0003] With the development of TTS technology, the use of high-quality training corpus can achieve speech synthesis effects that are indistinguishable from real people. However, speech synthesis service providers currently only provide a few fixed speakers. Although some manufacturers also provide speaker customization services, the price is expensive. One reason for the high price of speaker customization services is that speech synthesis currently requires a lot of corpus. Generally, professional broadcasters are required to go to a professional recording environment to record dozens of hours of corpus, and then train new speakers. The customization cycle is also very long.

[0004] The mainstream technology of TTS technology at present is to use sequence-to-sequence acoustic model and neural network vocoder, and the acoustic model and vocoder are trained independently. Among them, the acoustic model realizes the conversion of text information into acoustic features, and the vocoder realizes the conversion of acoustic features into audio. However, there are still some serious problems with speech synthesis at present: such as very poor sound quality, the speaking speed and intonation of all speakers are averaged, and the speaker's personality is lost, so it is difficult to apply it in actual scenarios. Summary of the invention

[0005] To solve the above technical problems, the present invention proposes a training method and device for a speech synthesis system, and a speech synthesis method and device, which use independent modeling of consonants, finals and tones, so that the speech synthesis system only needs less training corpus to cover all modeling units.

[0006] A first aspect of the present invention discloses a training method for a speech synthesis system, the method comprising:

[0007] Collecting the training speech of the target speaker to form training audio, and performing text annotation on the training audio;

[0008] Convert the obtained text annotations into pronunciation unit annotations, wherein the pronunciation unit annotations include corresponding initial consonants and finals with tones;

[0009] Decomposing the pronunciation unit annotation and converting it into a pronunciation unit vector without tone and a tone vector;

[0010] Based on the training audio, the pronunciation unit vector and the tone vector, a training sample for training the speech synthesis system is formed.

[0011] According to the training method of the first aspect of the present invention, after converting the text annotation into the pronunciation unit annotation and before decomposing and converting the pronunciation unit annotation into the pronunciation unit vector without tone and the tone vector, it also includes:

[0012] The standard pauses between words in the training audio are detected using an alignment technology based on speech recognition, and the pronunciation unit annotations are modified according to the detected standard pauses between words so that the pronunciation unit annotations contain information about the standard pauses between words.

[0013] According to the training method of the first aspect of the present invention, before forming the training samples for training the speech synthesis system, the training method further comprises:

[0014] Based on the pronunciation unit, the audio energy of each word corresponding to the pronunciation unit label is calculated, and based on the fact that the audio energy of the stressed word is greater than the average audio energy of each word in the sentence, the information of the stressed word in the sentence is obtained, and then converted into a corresponding stress vector;

[0015] The training sample also includes the rereading vector.

[0016] According to the training method of the first aspect of the present invention, before forming the training samples for training the speech synthesis system, the training method further comprises:

[0017] Obtaining a corresponding sentence pattern identifier according to an end symbol of a sentence in the text annotation;

[0018] The training sample also includes the rereading vector.

[0019] According to the training method of the first aspect of the present invention, before forming the training samples for training the speech synthesis system, the method further comprises:

[0020] Screening the voiced segments in the training audio;

[0021] Using voiceprint extraction technology, extracting the voiceprint of the voiced segment and averaging it as a speaker code;

[0022] The sample input also includes the speaker code.

[0023] According to the training method of the first aspect of the present invention, the pronunciation unit vector corresponds to the tone vector one by one.

[0024] According to the training method of the first aspect of the present invention, the pronunciation unit vector includes letter vector units corresponding to pinyin letters and silent units, and the silent units are used to represent starting and ending silence and pause punctuation marks.

[0025] According to the training method of the first aspect of the present invention, in step S3, the tone vector defines 6 tone identifiers, which respectively correspond to the silent units and the silent tones of initials, and the first, second, third, fourth and light tones of finals, wherein the silent units and the silent tones of initials correspond to the same tone identifier.

[0026] According to the training method of the first aspect of the present invention, the speech synthesis system is a VITS-based speech synthesis system.

[0027] The training method according to the first aspect of the present invention further comprises:

[0028] The pronunciation unit vector and the tone vector of the training sample are encoded using different embedding network layers respectively;

[0029] Superimposing a second feature vector obtained after encoding the tone vector onto a first feature vector obtained after encoding the pronunciation unit vector, and inputting the result into a neural network of the speech synthesis system;

[0030] After the neural network is trained, the speech synthesis system learns the rule that the pronunciation unit changes intonation according to the intonation.

[0031] According to the training method of the first aspect of the present invention, the training sample includes the training audio and the stress vector, and the training audio is preprocessed as follows before input: obtaining standard pauses between words, performing display modeling on the standard pauses between words, obtaining stressed words, and performing display modeling on stress; the training method further includes:

[0032] The rereading vectors of the training samples are encoded using an independent embedding network layer;

[0033] Superimposing the third feature vector formed by encoding the rereading vector onto the first feature vector encoded by the pronunciation unit vector, and inputting the third feature vector into the neural network of the speech synthesis system;

[0034] After the neural network is trained, the speech synthesis system learns the rule that the pronunciation unit is stressed according to the stress vector.

[0035] According to the training method of the first aspect of the present invention, the training sample includes a sentence identifier, and the training method further includes:

[0036] Encoding the sentence identifiers of the training samples using an independent embedding network layer;

[0037] Expanding the fourth feature vector after the sentence identification is encoded and superimposing it on the first feature vector after the pronunciation unit vector is encoded, and inputting it into the neural network of the speech synthesis system;

[0038] After the neural network is trained, the speech synthesis system learns the rules of changing the tone according to the sentence pattern.

[0039] According to the training method of the first aspect of the present invention, the training sample includes a speaker code, and the training method further includes:

[0040] encoding the speaker of the training sample and processing it using a linear neural network;

[0041] The speaker code is processed by a linear neural network and replaces the one-hot speaker code of VITS.

[0042] A second aspect of the present invention further provides a training device for a speech synthesis system, the training device comprising:

[0043] A collection module, used for collecting the training speech of the target speaker to form training audio;

[0044] A text annotation module, used for performing text annotation on the training audio;

[0045] A pronunciation annotation module, used for converting the obtained text annotation into pronunciation unit annotation, wherein the pronunciation unit annotation includes corresponding initial consonants and finals with tones;

[0046] A labeling and disassembling module, which disassembles the pronunciation unit labeling and converts it into a pronunciation unit vector without tone and a tone vector;

[0047] The output module forms a training sample for training the speech synthesis system based on the training audio, the pronunciation unit vector and the tone vector.

[0048] A third aspect of the present invention provides a speech synthesis method, which performs speech synthesis using a speech synthesis system formed by any of the training methods described above.

[0049] The speech synthesis method according to the third aspect of the present invention comprises:

[0050] Get the speaker code of the target speaker;

[0051] Obtaining a sentence to be used for synthesizing speech;

[0052] Obtaining a sentence pattern identifier according to the end symbol of the sentence;

[0053] Converting Chinese characters of the sentence into pronunciation unit vectors without tones and tone vectors;

[0054] Generate a stress vector with all zeros or other default values ​​according to the length of the pronunciation unit vector; or generate a corresponding stress vector according to the stressed word in the sentence;

[0055] The pronunciation unit vector, the tone vector, the stress vector, the sentence pattern identifier and the speaker code are used as input and sent to a trained speech synthesis model to generate corresponding speech.

[0056] A fourth aspect of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the training methods described above, or the steps in any one of the speech synthesis methods described above are implemented.

[0057] A fifth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps of any of the training methods described above, or the steps of any of the speech synthesis methods described above.

[0058] The training method of the speech synthesis system of the present application particularly relates to a personalized voice customization using a small amount of corpus. Through the method of the present application, by recording five minutes of corpus of the speaker to be customized, the personalized customization of the target speaker can be achieved. The speaker's timbre and rhythm are basically consistent with those of a real person, and the sound quality is high-fidelity; the use of the method of the present application can greatly reduce the customization cost, and can provide customized services for terminal individuals, which can make the customization of the speaker of speech synthesis widely popularized; the method of the present application can be used in all products with voice interaction, such as intelligent robots, car voice, smart TVs and speakers, smart toys, virtual people, etc. The method of the present application also has broad application prospects in the field of voice media, such as automatic generation of audio novels, short video dubbing, voice customer service, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 is a flow chart of a training method for a speech synthesis system according to an embodiment of the present invention;

[0061] Figure 2A training flow chart of a personalized speaker for a speech synthesis system based on VITS according to an embodiment of the present invention;

[0062] Figure 3 A speech synthesis flow chart of a personalized speaker of a speech synthesis system based on VITS according to an embodiment of the present invention;

[0063] Figure 4 The figure is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0065] The multi-speaker speech synthesis technology using independently trained acoustic models and vocoder models (typical acoustic models include: Tactron2, FastSpeech, GlowSpeech, etc.; typical vocoders include: waveRnn, waveNet, LPCNet, MelGAN, Hifi-GAN, etc.) has very poor synthesized sound quality, and the speaking speed and intonation of all speakers show a phenomenon of being averaged (i.e., the speaker's individuality is lost); the main reason is that there is a mismatch between the independently trained acoustic model and the independently trained vocoder, and the vocoder cannot distinguish the average caused by the speaker. The VITS speech synthesis system can solve the above problems, but it does not focus on the problem of speaker customization for the Chinese speech synthesis system. Therefore, using VITS to implement speaker customization for the Chinese speech synthesis system will face these problems: how to model Chinese pronunciation units; how to control the Chinese pronunciation rhythm; how to add new personalized speakers. This application uses VITS to implement speaker customization for the Chinese speech synthesis system, and will provide solutions to these problems.

[0066] The above-mentioned VITS is a completely end-to-end speech synthesis system first proposed in a paper titled Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech. It adopts a posterior encoder based on VAE (Variational Autoencoder), a priori encoder based on Normalizing Flow, and a Hifi-GAN decoder to realize a complete end-to-end speech synthesis system to solve the problem of mismatch between acoustic model and vocoder. It adopts stochastic duration predictor to model the duration of pronunciation unit, which solves the problem of speech rate averaging in current mainstream technology. At the same time, it adopts a Hifi-GAN decoder based on speaker condition and its adversarial training technology to ensure the speech quality while ensuring the personalized pronunciation details of the speaker.

[0067] It should be noted that although this article describes the solution of the present invention based on the VITS speech synthesis system, the technical solution of the present invention may also be used in combination with other technologies, and this article does not make specific limitations on this.

[0068] The embodiment of the present invention provides a method for training a speech synthesis system, the method generally comprising: a step of obtaining a training sample of a target speaker, and a step of training the speech synthesis system based on the training sample; Figure 1 As shown, the step of obtaining the target speaker training sample includes:

[0069] Step 1: collecting the training speech of the target speaker to form training audio, and performing text annotation on the training audio;

[0070] Step 2: converting the obtained text annotation into pronunciation unit annotation, wherein the pronunciation unit annotation includes corresponding initial consonants and finals with tones;

[0071] Step 3: Decompose the pronunciation unit annotation and convert it into a pronunciation unit vector without tone and a tone vector;

[0072] Step 4: Based on the training audio, pronunciation unit vector and tone vector, a training sample for training the speech synthesis system is formed.

[0073] Among them, the pronunciation unit vector can correspond to the tone vector one by one. For example, a word in the speech corresponds to a pronunciation unit vector and a tone vector, and there is a corresponding relationship between the two, which can indicate how the word is pronounced and how the tone is.

[0074] Among them, the pronunciation unit vector may include letter vector units corresponding to pinyin letters and non - pronunciation units. The non - pronunciation units are used to represent the starting and ending silences and the punctuation marks for pauses.

[0075] Among them, in step S3, the tone vector can define 6 tone identifiers, which respectively correspond to the non - pronunciation unit and the voiceless tone of the initial consonant, the first tone, the second tone, the third tone, the fourth tone, and the light tone of the final consonant. Among them, the non - pronunciation unit and the voiceless tone of the initial consonant correspond to the same tone identifier.

[0076] Currently, the research on speech synthesis mainly focuses on English speech synthesis systems. For example, VITS only provides English pronunciation modeling methods. The biggest difference between Chinese pronunciation and English pronunciation is that Chinese pronunciation is based on pinyin and pinyin has tones, while English pronunciation has no tones. Common English synthesis systems use English phonetic symbols for modeling. Common Chinese synthesis systems use pinyin with tones for modeling. The larger the modeling granularity, the more training corpus is needed to cover all modeling units. Therefore, the modeling method using pinyin with tones disassembles pinyin into two modeling units: one is the initial consonant, and the other is the final consonant with tone. Usually, when designing TTS pronunciation units, some sounds will be further subdivided based on standard initial and final consonants. For example, the final consonant i in zi and zhi is defined as two modeling units because of the very large difference in pronunciation. For simplicity, taking the number of standard initial and final consonants as an example to illustrate the problem: there are 23 initial consonants, 24 final consonants, and 5 tones (the first tone, the second tone, the third tone, the fourth tone, and the light tone). Then the number of modeling units is 23 + 24 * 5 = 143. Using this method, there is a serious problem, that is, it is very difficult for the training corpus to cover all these pronunciation units. When actually used, the synthesized speech of modeling units not appearing in the training corpus and rare modeling units with few occurrences is completely wrong noise. This modeling method has been used since the era of digital signal processing speech synthesis. Digital signal processing technology cannot model the tone change of pinyin and can only perform bundled modeling on initial and final consonants and tones.

[0077] In the method for forming the speech synthesis system of this embodiment, the method of independent modeling of initial and final consonants and tones is adopted, and the neural network Embedding technology is used to enable the speech synthesis system to learn the tone change rules of pinyin by itself. Similarly, taking the standard initial and final consonants as an example, there are 23 + 24 = 47 pronunciation units, and the non - pronunciation unit sil represents the starting and ending silences and the punctuation marks for pauses. 6 tones are defined, which are respectively the non - pronunciation unit and the voiceless tone of the initial consonant, the first tone, the second tone, the third tone, the fourth tone, and the light tone of the final consonant, and {0, 1, 2, 3, 4, 5} are used as ids in sequence. Taking the synthesis of "ni hao" as an example, its pinyin is "ni3 hao3", and an input vector {sil, n, i3, h, ao3, sil} is implemented according to the mainstream modeling method.

[0078] In the method of this embodiment, two parallel input vectors {sil, n, i, h, ao, sil} and {0, 0, 3, 0, 3, 0} are modeled. During training, the Embedding technology is used to encode the two input vectors into two feature vectors {x, c}, and then the feature vector c of the tone is superimposed on x, so that c serves as a condition and disturbance of x. After training, the speech synthesis system can automatically learn the tone change rules of the finals. Such a modeling method can ensure that all modeled units can be covered by the training corpus as much as possible, and can also reduce the training corpus required for personalized customization by the speaker.

[0079] In order to prove the effectiveness of this modeling method, the inventors of the present application tested the creation of pronunciation inputs that do not exist in Chinese at all. The system formed by the method of the present application can still produce speech with imperfect pitch, but the speech produced by the mainstream modeling method is completely incomprehensible noise. A specific example is that the pronunciation of "driving" is changed from "kai1 che1" to "kai1che5". In addition, if all the pitch input vectors are set to zero when synthesizing speech, the synthesized result is that the audio pronunciation is like Chinese read by a native English speaker and there is no tone change. This also proves from the side that the modeling method of the present application is more in line with objective reality: pitch should be modeled as a condition and disturbance of the final.

[0080] In some embodiments, after step 2 and before step 3, the target speaker training sample acquisition step further includes:

[0081] The standard pauses between words in the training audio are detected using the alignment technology based on speech recognition, and the pronunciation unit annotations obtained in step 2 are modified according to the detected standard pauses between words, so that the modified pronunciation unit annotations contain information about the standard pauses between words.

[0082] At present, the corpus for speech synthesis is usually recorded by professional announcers, and its speech speed, volume, and emotion are very stable and consistent; however, when implementing the customization of the speaker, the target speaker usually has not received professional pronunciation training, and often has problems such as stuck words and pauses, and also has habitual rereading and emotion adjustment according to the sentence structure. For the pauses between words that appear in the training corpus of the target speaker, that is, the situation of fluency, since VITS uses monotonic alignment search to achieve the alignment of pronunciation and text annotation, the pauses between words will be assigned to the pronunciation units on both sides. When synthesizing speech, these pronunciation units will have occasional pauses, resulting in a decrease in the fluency of the synthesized speech. Therefore, the pauses between words in the training audio must be explicitly modeled, and the pauses must not affect the pronunciation units on both sides of the pauses between words; accurate pauses between words can be achieved based on manual annotation, but the cost is too high.

[0083] This embodiment uses the alignment technology based on speech recognition to detect the standard pauses between words in the training audio, and according to the detected standard pauses between words, the pronunciation unit annotation obtained in step 2 is modified so that the modified pronunciation unit annotation contains the information of the standard pauses between words. The standard pauses between words here refer to normal pauses between words, and do not include or exclude the abnormal pauses such as stuck words, unintentional pauses, coughs, etc. due to lack of professional pronunciation training. This step can adopt the method of adding pause identifiers between words to the pronunciation unit annotations to add the relevant information of the standard pauses between words to the pronunciation unit annotations.

[0084] This embodiment can use the alignment technology based on speech recognition to detect the pauses between words in the training audio, and make a display pause mark in the corresponding position in the original text mark; specifically, this embodiment uses MFA (montreal forcedaligner) to complete this work. MFA is a speech-to-text alignment tool based on KALDI. After the pauses between words are modeled explicitly, the monotone consistency search algorithm of VITS will not assign the pauses to the pronunciation units, and there will be no abrupt pauses when performing speech synthesis.

[0085] In some embodiments, in some embodiments, before step 3, the method further includes:

[0086] Based on the pronunciation unit, the audio energy of each character corresponding to the pronunciation unit annotation is calculated, and based on the fact that the audio energy of the stressed character is greater than the average audio energy of each character in the sentence, the information of the stressed character in the sentence is obtained, and then converted into a corresponding stress vector; in step 3, the training sample also includes the stress vector.

[0087] The training samples also include stress vectors, which means that the training samples input into the speech synthesis system contain stress information corresponding to the training audio. Specifically, the training samples can be directly multiple vectors, which contain stress vectors corresponding to the training audio. The training samples can also be generated based on the stress vectors so that the training samples contain the stress information of each sentence of the training audio.

[0088] For the stress that appears in the training corpus of the target speaker, if the stressed and unstressed pronunciation units are modeled in a non - distinguishable and mixed way, when synthesizing speech, problems such as the averaging of stress and non - stress and the random appearance of stress will occur, resulting in a decrease in the naturalness of the synthesized speech; therefore, the stressed and unstressed pronunciation units should be modeled in a distinguishable way, similar to the method of tones. Accurate stress can be achieved based on manual annotation, but the cost is too high; in a sentence, the energy of the stressed words is usually significantly greater than that of the non - stressed words, and stress can be identified based on this. Since the alignment technology based on speech recognition is used in this embodiment to obtain the audio corresponding to each word, the energy of each word can be calculated, so as to approximately obtain which are the stressed words and which are the non - stressed words. Similarly, in this embodiment, the non - stressed and stressed can be modeled with {0, 1}. If a word is stressed, both its initial and final consonants are stressed. After the stress vector is encoded by the Embedding technology, it is superimposed on the feature vector x of the pronunciation unit; taking the synthesis of "today's weather" as an example, its pinyin is "jin1 tian1 de5 tian1 qi4". Assuming that the stress occurs on 'tian', then according to the method of this embodiment, the input pronunciation unit vector is {sil, j, in, t, ian, d, e, t, ian, q, i, sil}, the tone vector is {0, 0, 1, 0, 1, 0, 5, 0, 1, 0, 4, 0}, and the stress vector is {0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0}.

[0089] In some embodiments, before step 3, the target speaker training sample acquisition step further includes: obtaining the corresponding sentence pattern identifier (sentence pattern id) according to the end symbol of the annotated sentence in the text annotation; in step 3, the training sample further includes the sentence pattern identifier.

[0090] The training sample further includes the sentence pattern identifier, which means that the training sample input into the speech synthesis system contains the sentence pattern identifiers corresponding to each sentence of the training audio. Specifically, the training sample can directly be a multi - vector, and the multi - vector contains the sentence pattern identifier, or the training sample can be generated based on the sentence pattern identifier so that the training sample contains the tone information corresponding to the sentence pattern identifier.

[0091] Common Chinese sentence patterns are divided into declarative sentences, interrogative sentences, exclamatory sentences, and imperative sentences. When the target speaker records the corpus, they will habitually use a tone that conforms to the sentence pattern for pronunciation. In order to improve the tone expression ability of the speech synthesis system, the declarative sentences, interrogative sentences, exclamatory sentences, and imperative sentences in this application embodiment are modeled with {0, 1, 2}; exclamatory sentences and imperative sentences have similarities and often end with an exclamation mark, so they are encoded together with 2; interrogative sentences often end with a question mark; sentences whose ending symbols are not exclamation marks and question marks are encoded with the declarative sentence 0. After the sentence pattern id is encoded by the Embedding technology, it is replicated and extended according to the dimension of the feature vector x of the pronunciation unit and then superimposed on the feature vector x.

[0092] In some embodiments, before step 3, it also includes: screening the voiced segments in the training audio; using voiceprint extraction technology to extract the voiceprints of the voiced segments and average them as the speaker code; in step 3, the sample input also includes the speaker code.

[0093] The training sample also includes a speaker code, which means that the training sample input into the speech synthesis system contains the speaker code corresponding to each sentence of the training audio. Specifically, the training sample can be directly a multiple vector containing the speaker code, or the training sample can be generated based on the speaker code so that the training sample contains the speaker identification information corresponding to the speaker code.

[0094] The original VITS uses one-hot speaker encoding to embed into the speech synthesis system through Embedding technology to realize multiple speakers in the speech synthesis system; the one-hot speaker encoding method is determined by the number of speakers, so it does not support adding new speakers to the trained model. If new speakers must be added, the entire system must be trained from scratch. This application uses voiceprint encoding to encode speakers, but in actual use, due to the unreliability of voiceprint encoding itself, some speakers interfere with each other and the sound quality is poor.

[0095] After analysis, it is found that general voiceprint systems use moderate-dimensional acoustic features to extract voiceprint codes for real-time performance; since the voice code extraction process of this application does not require real-time performance, high-dimensional acoustic features are used to extract voiceprint codes. Improving the accuracy of voiceprint coding can increase information sharing between similar speakers and reduce mutual interference between dissimilar speakers.

[0096] Voiceprint encoding generally extracts speech in segments and then averages them over all speech segments; therefore, silent segments without any features are usually discarded directly; since speech includes voiced and unvoiced sounds: when making voiced sounds, the vocal cords resonate with the pitch period F0, the entire vocal tract participates in the pronunciation, the voiceprint features are obvious, and there are great differences between different speakers; while when making unvoiced sounds, the vocal cords do not work, only part of the vocal tract organs participate in the pronunciation, the voiceprint features are not obvious, and there are small differences between different speakers, such as [s]; therefore, further, the present application can use the pitch period (F0) detection technology to determine the voiced segments, and then extract the voiceprint encoding of the voiced segments, and globally average them as the speaker encoding of the present application. In order to illustrate the adverse effects of unvoiced sounds on voiceprint coding, two sentences A1 and A2 from female speaker A and one sentence B1 from male speaker B are collected. The voiceprint codes extracted based on the whole sentence are: A1_x, A2_x, B1_x, and the voiceprint codes extracted based on voiced sounds are: A1_f, A2_f, B1_f. Then the cosine similarities between the voiceprint codes are calculated, which are: cosine_similarity(A1_x, A1_f)=0.8769, cosine_similarity(A2_x, A2_f)=0.9405, cosine_similarity(A1_x, A2_x)=0.6210, cosine_simil arity(A1_f, A2_f)=0.6868, cosine_similarity(A1_x, B1_x)=-0.0558, cosine_similarity(A1_f, B1_f)=-0.0753; It can be concluded that: there are obvious differences between voiceprint coding based on whole sentence extraction and voiced sound extraction; between different sentences of the same speaker, the similarity of voiceprint coding based on voiced sound extraction is higher than that of voiceprint coding based on whole sentence extraction; between different speakers, the difference of voiceprint coding based on voiced sound extraction is greater than that of voiceprint coding based on whole sentence extraction; therefore, in summary, voiceprint coding based on voiced sound extraction is better than voiceprint coding based on whole sentence extraction. The personalized speaker addition scheme of the present application can be summarized as using voiceprint coding based on voiced sound as the speaker coding, embedding it into the speech synthesis system for speaker training, and thus completing the addition of new speakers.

[0097] Among them, the present application is formed based on the VITS speech synthesis system. Based on the training samples, the steps of training the VITS speech synthesis system include: encoding the pronunciation unit vector and the tone vector of the training sample using different embedding network layers (Embedding); superimposing the second feature vector obtained after the tone vector encoding on the first feature vector obtained after the pronunciation unit vector encoding, and inputting the neural network of the speech synthesis system; after the neural network is trained, the speech synthesis system learns the law of the pronunciation unit changing tone according to the tone.

[0098] The training sample may also include the training audio and stress vector. The training audio is preprocessed as follows before input: obtaining standard pauses between words and display modeling of standard pauses between words; obtaining stressed words and display modeling of stress; the step of training the speech synthesis system based on the training sample also includes: encoding the stress vector of the training sample using an independent Embedding network layer; superimposing the third feature vector formed by encoding the stress vector on the first feature vector after encoding the pronunciation unit vector, and inputting it into the neural network of the speech synthesis system; after the neural network is trained, the speech synthesis system learns the rule that the pronunciation unit is stressed according to the stress vector.

[0099] The step of training the speech synthesis system based on the training samples also includes: encoding the sentence identifiers of the training samples using an independent Embedding network layer; expanding the fourth eigenvector after the sentence identifier is encoded and superimposed on the first eigenvector after the pronunciation unit vector is encoded, and inputting the neural network of the speech synthesis system; after the neural network is trained, the speech synthesis system learns the law of changing the tone according to the sentence pattern. Here, expansion refers to the operation performed without changing its information for the convenience of matrix calculation.

[0100] Among them, the step of training the speech synthesis system based on the training samples also includes: encoding the speaker of the training samples and processing them using a linear neural network (linear layer); after the speaker code is processed by the linear neural network, replacing the one-hot speaker code of VITS.

[0101] In summary, the present application provides a method for forming a speech synthesis system based on VITS, especially a method for customizing speakers of a Chinese speech synthesis system. When implementing multi-speaker TTS, the sound quality, timbre and rhythm of each speaker are basically consistent with the training speech, and the use of GPU for reasoning can meet real-time requirements; and after implementing the basic multi-speaker TTS model, the newly added speaker only needs 5 minutes of corpus for training to achieve personalized customization of the speaker.

[0102] The speaker customization service of the speech synthesis system has a wide market demand and huge commercial value. The VITS-based speech synthesis system of the present invention can realize personalized customization of the speaker, and can realize speaker customization using very little training corpus. The voice quality and rhythm are highly consistent with the training corpus, and a high-fidelity speaker can be customized using 5 minutes of high-fidelity training corpus. The customization cost of the speaker is greatly reduced, and customization services can be provided to terminal individual consumers, and can be widely used in smart devices and voice media. At the same time, the technology also has social value, such as customizing special pronunciation systems for disabled people with acquired pronunciation disorders.

[0103] Compared with the mainstream method of bundling finals and tones for modeling, the method of independently modeling vowels and tones proposed in the present application has fewer modeling units, and it is easier to cover the modeling units on less training corpus, reducing the error of pronouncing uncommon modeling units. Therefore, this modeling method is more suitable for speaker customization of speech synthesis system; after independently modeling the pronunciation units and tones using the neural network Embedding technology, the feature vector of the tone vector is added to the feature vector of the pronunciation unit vector in the feature space. After training, the neural network of the speech synthesis system can automatically learn the tone change rules of the pronunciation units.

[0104] This application also uses alignment technology based on speech recognition to pre-process the training corpus, obtains pauses between words and stress marks, and then performs display modeling, as well as a sentence modeling method based on ending symbols, which can improve the fluency of speech synthesis and the ability to express tone.

[0105] This application is also based on voiceprint coding based on voiced sound extraction, which is more reliable than the mainstream voiceprint coding based on whole sentence extraction. It can increase information sharing between similar speakers and reduce mutual interference between dissimilar speakers, thereby improving the accuracy of pronunciation and improving sound quality.

[0106] A second aspect of the present invention further provides a training device for a speech synthesis system, the training device comprising:

[0107] A collection module, used for collecting the training speech of the target speaker to form training audio;

[0108] A text annotation module, used for performing text annotation on the training audio;

[0109] A pronunciation annotation module, used for converting the obtained text annotation into pronunciation unit annotation, wherein the pronunciation unit annotation includes corresponding initial consonants and finals with tones;

[0110] A labeling and disassembling module, which disassembles the pronunciation unit labeling and converts it into a pronunciation unit vector without tone and a tone vector;

[0111] The output module forms a training sample for training the speech synthesis system based on the training audio, the pronunciation unit vector and the tone vector.

[0112] The training device according to the present invention also includes: a pause module, which is used to detect standard pauses between words in the training audio based on speech recognition alignment technology, and modify the pronunciation unit annotation according to the detected standard pauses between words so that the pronunciation unit annotation contains information about the standard pauses between words.

[0113] The training device according to the present invention further includes: a stress module, which is used to calculate the audio energy of each word corresponding to the pronunciation unit label based on the pronunciation unit, and based on the fact that the audio energy of the stressed word is greater than the average audio energy of each word in the sentence, obtain the information of the stressed word in the sentence, and then convert it into a corresponding stress vector. The training sample also includes the stress vector.

[0114] The training device according to the present invention further comprises: a sentence pattern module, which is used to obtain a corresponding sentence pattern identifier according to the end symbol of the sentence in the text annotation. The training sample also comprises the rereading vector.

[0115] The training device according to the present invention further comprises: a speaker encoding module for screening the voiced segments in the training audio, using voiceprint extraction technology to extract the voiceprints of the voiced segments and average them as speaker codes. The sample input also comprises the speaker code.

[0116] According to the training device of the present invention, the pronunciation unit vector corresponds to the tone vector one by one.

[0117] According to the training device of the present invention, the pronunciation unit vector includes letter vector units corresponding to pinyin letters and silent units, and the silent units are used to represent the silence of the start and end and the punctuation marks of the pause.

[0118] According to the training device of the present invention, the tone vector defines 6 tone identifiers, which respectively correspond to the silent units and the silent tones of initials, and the first, second, third, fourth and light tones of finals, wherein the silent units and the silent tones of initials correspond to the same tone identifier.

[0119] According to the training device of the present invention, the speech synthesis system is a VITS-based speech synthesis system.

[0120] The training device according to the present invention also includes: a training module, which is used to encode the pronunciation unit vector and the tone vector of the training sample using different embedded network layers respectively; superimpose the second feature vector obtained after encoding the tone vector on the first feature vector obtained after encoding the pronunciation unit vector, and input it into the neural network of the speech synthesis system; after the neural network is trained, the speech synthesis system learns the rule that the pronunciation unit changes according to the tone.

[0121] According to the training device of the present invention, the training sample includes the training audio and the stress vector, and the training audio is preprocessed as follows before input: obtaining standard pauses between words, display modeling of the standard pauses between words, obtaining stressed words, and display modeling of stress; the training module is also used to: encode the stress vector of the training sample using an independent embedding network layer; superimpose the third feature vector formed after encoding the stress vector on the first feature vector after encoding the pronunciation unit vector, and input it into the neural network of the speech synthesis system; after the neural network is trained, the speech synthesis system learns the rules of stressing the pronunciation unit according to the stress vector.

[0122] According to the training device of the present invention, the training samples include sentence identifiers, and the training module is also used to: encode the sentence identifiers of the training samples using an independent embedding network layer; expand the fourth feature vector after the sentence identifier is encoded and superimposed on the first feature vector after the pronunciation unit vector is encoded, and input it into the neural network of the speech synthesis system; after the neural network is trained, the speech synthesis system learns the rules of changing the tone according to the sentence pattern.

[0123] According to the training device of the present invention, the training sample includes a speaker code, and the training module is further used to: process the speaker code of the training sample using a linear neural network; after the speaker code is processed by the linear neural network, replace the one-hot speaker code of VITS.

[0124] The third aspect of the present invention further provides a speech synthesis method, which uses a speech synthesis system obtained by any of the training methods described above to perform speech synthesis. Specifically, the method comprises:

[0125] Get the speaker code of the target speaker;

[0126] Obtaining a sentence to be used for synthesizing speech;

[0127] Obtaining a sentence pattern identifier according to the end symbol of the sentence;

[0128] Converting Chinese characters of the sentence into pronunciation unit vectors without tones and tone vectors;

[0129] Generate a stress vector with all zeros or other default values ​​according to the length of the pronunciation unit vector; or generate a corresponding stress vector according to the stressed word in the sentence;

[0130] The pronunciation unit vector, the tone vector, the stress vector, the sentence pattern identifier and the speaker code are used as input and sent to a trained speech synthesis model to generate corresponding speech.

[0131] A fourth aspect of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the training methods described above, or the steps in any one of the speech synthesis methods described above are implemented.

[0132] A fifth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps of any of the training methods described above, or the steps of any of the speech synthesis methods described above.

[0133] The technical solution of the present application is further described below in conjunction with specific embodiments.

[0134] The personalized speaker implementation of the speech synthesis system includes two stages: target speaker speech training and target speaker speech synthesis, which will be explained below.

[0135] The specific implementation method of the target speaker voice training stage is as follows:

[0136] Step 1: Complete the training speech collection and text annotation of the target speaker;

[0137] Step 2: Convert the text annotations into pronunciation unit annotations of initials and finals with tones;

[0138] Step 3: Use the MFA tool to align the target speaker’s training speech and pronunciation unit annotations;

[0139] Step 4: Based on the alignment results, possible pauses between words are obtained, the pronunciation unit annotations in step 2 are modified, and pauses between words are displayed and annotated, such as using pause for annotation;

[0140] Step 5: According to the alignment results, calculate the average energy of the words composed of pronunciation units, obtain the possible stressed words, and obtain the stress vector corresponding to the sentence;

[0141] Step 6: Get the sentence ID according to the end symbol of the marked sentence;

[0142] Step 7: Use the fundamental pitch F0 detection algorithm to filter the voiced segments in the training audio;

[0143] Step 8: Use voiceprint extraction technology to extract the voiceprint of the voiced segment and average it as the speaker code;

[0144] Step 9: Decompose the pronunciation unit annotation in step 4 into a pronunciation unit vector without tone and a tone vector;

[0145] Step 10: Send {pronunciation unit vector, tone vector, stress vector, sentence id, speaker code, audio} as a sample to the VITS speech synthesis system for training, as follows: Figure 2 shown.

[0146] The specific implementation method of the target speaker speech synthesis is as follows Figure 3 As shown, including:

[0147] Step 1: The speaker code of the target speaker is known (i.e. Figure 2 and Figure 3 Speaker encoding in );

[0148] Step 2: Enter the sentence to be used for synthesized speech;

[0149] Step 3: Get the sentence id according to the sentence ending symbol;

[0150] Step 4: Convert Chinese characters into pronunciation unit vectors and tone vectors without tones;

[0151] Step 5: Generate a default stress vector of all zeros according to the length of the pronunciation unit vector, or generate a stress vector according to the word that actually needs to be stressed;

[0152] Step 6: Take {pronunciation unit vector, tone vector, stress vector, sentence id, speaker code} as input and send it to the trained speech synthesis model to generate the corresponding speech.

[0153] Among them, about Figure 2 The training process of the VITS speech synthesis system in Figure 3 For the application of speech synthesis models, please refer to the international conference paper ICML 2021: International Conference on Machine Learning, "Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech", J Kim, J Kong, J Son.

[0154] This embodiment also provides a method for implementing personalized customization of a speech synthesis system for a speaker, including:

[0155] VITS-based speech synthesis system;

[0156] Methods for independent modeling of vowels, finals and tones;

[0157] Define 6 tones, which are the silent unit and the no tone of the initial consonant, the first tone, the second tone, the third tone, the fourth tone, and the light tone of the final, and use {0, 1, 2, 3, 4, 5} as ids respectively;

[0158] Decompose pinyin into pronunciation unit vector and tone vector;

[0159] The pronunciation unit vector and the tone vector are encoded using different Embedding network layers;

[0160] Superimposing the feature vector encoded by the tone vector onto the feature vector encoded by the pronunciation unit vector;

[0161] After training with the neural network, the speech synthesis system automatically learns the rules of pronunciation units changing according to pitch;

[0162] The training corpus is preprocessed using the alignment technology based on speech recognition: the pauses between words are obtained and modeled explicitly; the stressed words are obtained and modeled explicitly for stress;

[0163] The reread vector is encoded using an independent Embedding network layer;

[0164] Superimposing the feature vector encoded by the rereading vector onto the feature vector encoded by the pronunciation unit vector;

[0165] After training with the neural network, the speech synthesis system automatically learns the rules of pronunciation unit stress according to the stress vector;

[0166] Sentence modeling is performed based on the ending symbol: sentences ending with an exclamation mark are represented by 2; sentences ending with a question mark are represented by 1; other sentences are represented by 0; sentence ids are encoded using an independent Embedding network layer;

[0167] Expand the feature vector after the sentence id is encoded and superimpose it on the feature vector after the pronunciation unit vector is encoded;

[0168] After training with the neural network, the speech synthesis system automatically learns the rules of changing the tone of voice according to the sentence structure;

[0169] Voiceprint coding based on voiced sound extraction: Use the fundamental tone F0 detection technology to determine the voiced sound in the audio; use the voiceprint extraction technology to extract the voiceprint code corresponding to the voiced sound; average all the voiceprint codes of the target speaker based on voiced sound extraction as the speaker code; the speaker code is processed by the linear neural network and replaces the original VITS one-hot speaker code.

[0170] The third aspect of the present invention also provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps in any one of the training methods described above, or the steps in any one of the speech synthesis methods described above.

[0171] Figure 4 FIG. 1 is a structural diagram of one embodiment of the present invention. Figure 4 As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the housing of the electronic device, or an external keyboard, touch pad or mouse, etc.

[0172] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0173] The fourth aspect of the present invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method of any one of the first aspect or the second aspect of the present disclosure are implemented.

[0174] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A training method for a speech synthesis system, characterized in that: The training method comprises: Collecting the training speech of the target speaker to form training audio, and performing text annotation on the training audio; Convert the obtained text annotations into pronunciation unit annotations, wherein the pronunciation unit annotations include corresponding initial consonants and finals with tones; Decomposing the pronunciation unit annotation and converting it into a pronunciation unit vector without tone and a tone vector; Based on the training audio, the pronunciation unit vector and the tone vector, a training sample for training the speech synthesis system is formed.

2. The training method according to claim 1, characterized in that: After converting the text annotations into pronunciation unit annotations and before decomposing and converting the pronunciation unit annotations into pronunciation unit vectors without tones and tone vectors, the method further includes: The standard pauses between words in the training audio are detected using an alignment technology based on speech recognition, and the pronunciation unit annotations are modified according to the detected standard pauses between words, so that the modified pronunciation unit annotations contain information about the standard pauses between words.

3. The training method according to claim 1, characterized in that: Before forming the training samples for training the speech synthesis system, the training method further includes: Based on the pronunciation unit annotation, the audio energy of each word corresponding to the pronunciation unit annotation is calculated, and based on the fact that the audio energy of the stressed word is greater than the average audio energy of each word in the sentence, information of the stressed word in the sentence is obtained, and then converted into a corresponding stress vector; The training sample also includes the rereading vector.

4. The training method according to claim 1, characterized in that: Before forming the training samples for training the speech synthesis system, the training method further includes: Obtaining a corresponding sentence pattern identifier according to an end symbol of a sentence in the text annotation; The training sample also includes the sentence pattern identifier.

5. The training method according to claim 1, characterized in that: Before forming the training samples for training the speech synthesis system, the method further includes: Screening the voiced segments in the training audio; Using voiceprint extraction technology, extracting the voiceprint of the voiced segment and averaging it as a speaker code; The training sample also includes the speaker code.

6. The training method according to claim 1, characterized in that: The pronunciation unit vectors correspond one-to-one to the tone vectors.

7. The training method according to claim 6, characterized in that: The pronunciation unit vector includes letter vector units corresponding to pinyin letters and silent units, and the silent units are used to represent the silence of the start and end and the punctuation marks of the pause.

8. The training method according to claim 7, characterized in that: The tone vector defines 6 tone identifiers, which correspond to the silent units and the unvoiced tone of the initials, and the first, second, third, fourth and light tones of the finals, respectively. Among them, the silent units and the unvoiced tone of the initials correspond to the same tone identifier.

9. The training method according to any one of claims 1 to 8, characterized in that: The speech synthesis system is a speech synthesis system based on VITS.

10. The training method according to claim 9, characterized in that: The training sample includes the pronunciation unit vector and the tone vector, and the training method further includes: The pronunciation unit vector and the tone vector of the training sample are encoded using different embedding network layers respectively; Superimposing a second feature vector obtained after encoding the tone vector onto a first feature vector obtained after encoding the pronunciation unit vector, and inputting the second feature vector into a neural network of the speech synthesis system; After the neural network is trained, the speech synthesis system learns the law of tone change of the pronunciation unit according to the tone.

11. The training method according to claim 10, characterized in that: The training sample includes the training audio and the stress vector, and the training audio is preprocessed as follows before input: obtaining a standard pause between words, and performing display modeling on the standard pause between words based on the obtained pause between words; Obtain stressed words and perform explicit modeling of stress based on them; The training method further comprises: The rereading vectors of the training samples are encoded using an independent embedding network layer; Superimposing the third feature vector formed by encoding the rereading vector onto the first feature vector encoded by the pronunciation unit vector, and inputting the third feature vector into the neural network of the speech synthesis system; After the neural network is trained, the speech synthesis system learns the rule of stressing the pronunciation unit according to the stress vector.

12. The training method according to claim 11, characterized in that: The training sample includes a sentence identifier, and the training method further includes: Encoding the sentence identifiers of the training samples using an independent embedding network layer; Expanding the fourth feature vector after the sentence identification is encoded and superimposing it on the first feature vector after the pronunciation unit vector is encoded, and inputting it into the neural network of the speech synthesis system; After the neural network is trained, the speech synthesis system learns the rules of changing the tone according to the sentence pattern.

13. The training method according to claim 12, characterized in that: The training sample includes a speaker code, and the training method further includes: encoding the speaker of the training sample and processing it using a linear neural network; The speaker code is processed by a linear neural network and replaces the one-hot speaker code of VITS.

14. A training device for a speech synthesis system, characterized in that: The training device comprises: A collection module, used for collecting the training speech of the target speaker to form training audio; A text annotation module, used for performing text annotation on the training audio; A pronunciation annotation module, used for converting the obtained text annotation into pronunciation unit annotation, wherein the pronunciation unit annotation includes corresponding initial consonants and finals with tones; A labeling and disassembling module, which disassembles the pronunciation unit labeling and converts it into a pronunciation unit vector without tone and a tone vector; The output module forms a training sample for training the speech synthesis system based on the training audio, the pronunciation unit vector and the tone vector.

15. A speech synthesis method, characterized in that: A speech synthesis system obtained by using the training method described in any one of claims 1 to 13 performs speech synthesis.

16. The speech synthesis method according to claim 15, characterized in that: include: Get the speaker code of the target speaker; Obtaining a sentence to be used for synthesizing speech; Obtaining a sentence pattern identifier according to the end symbol of the sentence; Converting Chinese characters of the sentence into pronunciation unit vectors without tones and tone vectors; Generate a stress vector with all zeros or other default values ​​according to the length of the pronunciation unit vector; or generate a corresponding stress vector according to the stressed word in the sentence; The pronunciation unit vector, the tone vector, the stress vector, the sentence pattern identifier and the speaker code are taken as input and sent to a trained speech synthesis system to generate corresponding speech.

17. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the steps in the training method described in any one of claims 1 to 13, or the steps in the speech synthesis method described in any one of claims 15 to 16.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the training method described in any one of claims 1 to 13, or the steps in the speech synthesis method described in any one of claims 15 to 16 are implemented.

Images