A method and device for synthesizing speech of a specific speaker

Abstract

The present application provides a method and device for synthesizing speech for a specific speaker, which encodes a phoneme sequence according to the position of a target type text in an annotated text to obtain a position-encoded text with positional features, and uses the phoneme sequence and the position-encoded text to train a speech synthesis model. Correspondingly, the speech synthesis model designed by the present application splices the vector representation of the phoneme sequence and the vector representation of the position-encoded text through a splicing module and inputs them into an encoder, so that the encoder can learn the rhythmic latent features and the positional latent features of the target type text in parallel without increasing the complexity and input them into a variance adapter. The variance adapter strengthens the fusion of the speech features and the above features, and while ensuring that the model complexity is small, the trained model synthesizes the speech of a specific speaker with not only a sense of frustration, but also a sense of emphasis on the position of the target type text.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a method and device for synthesizing speech of a specific speaker. Background Art

[0002] Speech Synthesis, also known as text-to-speech technology, refers to the technology that converts any text into fluent speech through computer analysis. As one of the core technologies for realizing human-computer voice interaction systems, speech synthesis is an important direction in speech processing technology, and its application value is increasingly valued.

[0003] In existing speech synthesis technology, the synthesized speech is mostly dull and rigid, lacking a sense of pause and emphasis. Especially when designing a speech synthesis system for Chinese text, if the input text is simply matched to the pronunciation library one by one, and the pronunciations of all the characters are connected in series to form the speech output, such speech is mechanical and stiff, without intonation, which makes the user's listening experience very poor.

[0004] Therefore, it is worth studying how to improve the naturalness of synthesized speech and enhance user experience satisfaction. Summary of the invention

[0005] The present application provides a method and device for synthesizing speech for a specific speaker, which can increase the emotional characteristics of the model-synthesized speech while ensuring the training efficiency of the speech synthesis model, and improve the sense of pause, emphasis and naturalness of the model-synthesized speech.

[0006] In a first aspect, the present application provides a method for synthesizing speech for a specific speaker, comprising:

[0007] Obtaining an identity tag of each speaker among multiple speakers, a standard Mel spectrum corresponding to the speech audio data of each speaker, and annotated text corresponding to the speech audio data of each speaker and its corresponding phoneme sequence;

[0008] Determine the position of the target type text in the annotated text, and encode the phoneme sequence according to the position to obtain the position encoded text;

[0009] Constructing a speech synthesis model, which includes a phoneme embedding layer, a position embedding layer, a concatenation module, an encoder, a variance adapter, a Mel decoder and a vocoder, wherein the phoneme embedding layer is used to obtain a vector representation of a phoneme according to a phoneme sequence, the position embedding layer is used to obtain a vector representation of a position according to a position-encoded text, the concatenation module is used to concatenate the vector representation of the phoneme and the vector representation of the position, the encoder is used to extract the rhythmic latent features and the positional latent features of the annotated text according to the vector representation of the phoneme and the vector representation of the position, the variance adapter is used to enhance and expand the speech features, the rhythmic latent features and the positional latent features, the Mel decoder is used to implement parallel decoding according to the output of the variance adapter to obtain the Mel spectrum of the synthesized speech, and the vocoder is used to map the Mel spectrum of the synthesized speech into a sound waveform to obtain the synthesized speech;

[0010] The speech synthesis model is trained according to the phoneme sequence and the standard Mel spectrum, and the loss is calculated according to the error between the Mel spectrum of the synthesized speech and the standard Mel spectrum until the model converges to obtain a trained speech synthesis model;

[0011] Obtain the input text to be synthesized and the identity tag of the target speaker;

[0012] The input text and the target speaker's identity label are input into the trained speech synthesis model to synthesize speech with the target speaker's speech characteristics.

[0013] In one example, the method further includes: performing pinyin processing on the input text to obtain a first pinyin text;

[0014] Determine a target type text in an input text; identify polyphones in the target type text; re-determine a first phonetic text based on the polyphones in the target type text; and obtain a phoneme sequence corresponding to the input text based on the first phonetic text.

[0015] In one example, identifying polyphones in a target type text includes: querying a polyphone dictionary for corresponding pinyins according to the target type text, wherein the polyphone dictionary is used to characterize a mapping relationship between words including polyphones and pinyins, and the update of the polyphone dictionary supports hot update.

[0016] In one example, the annotated text is text with a prosodic tag, and the method further includes:

[0017] Constructing a phoneme alignment dictionary, which is used to represent the mapping relationship between pinyin and phonemes and the mapping relationship between prosodic labels and newly defined phonemes. The newly defined phonemes are sounds used to distinguish the pause durations of different prosodic labels.

[0018] performing phonetic processing on the text with the prosody tag to obtain a second phonetic text with the prosody tag;

[0019] The second phonetic text with the prosodic tag is converted into phonetic text data by using a phonetic alignment dictionary, wherein the phonetic text data includes phonemes corresponding to the phonetic part of the second phonetic text and newly defined phonemes corresponding to the prosodic tag part of the second phonetic text;

[0020] The method based on phoneme identification mapping converts phoneme text data into phoneme sequences.

[0021] In one example, encoding a phoneme sequence according to a position to obtain a positional encoded text includes:

[0022] Determine the mark corresponding to the target type text according to the position;

[0023] Encode the phoneme sequence using one-hot encoding based on the token and phoneme sequence to obtain positional encoded text.

[0024] In one example, a vocoder uses a generative adversarial network model for efficient and high-fidelity speech synthesis, where the upsampling parameters of the upsampling module of the model's generator are [10, 4, 4], the convolution kernel size in the upsampling module is [20, 8, 8], the number of hidden layers of the multi-receptive field fusion module of the model's generator is 128, and the residual expansion size of the multi-receptive field fusion module is [[1, 2], [2, 6], [3, 12]].

[0025] In one example, the target type is a name type or a place name type.

[0026] In one example, the speech features include pitch features, energy features, and sound length features of the speech audio data.

[0027] In one example, the method includes: building a speech synthesis model under the Tensorflow framework.

[0028] In a second aspect, the present application provides a speech synthesis device for a specific speaker, comprising:

[0029] A data acquisition module, used to acquire an identity tag of each speaker among multiple speakers, a standard Mel spectrum corresponding to the speech and audio data of each speaker, and annotated text corresponding to the speech and audio data of each speaker and its corresponding phoneme sequence;

[0030] A preprocessing module, used for determining the position of the target type text in the annotated text, and encoding the phoneme sequence according to the position to obtain the position encoded text;

[0031] A model building module, used for building a speech synthesis model, the speech synthesis model includes a phoneme embedding layer, a position embedding layer, a concatenation module, an encoder, a variance adapter, a Mel decoder and a vocoder, wherein the phoneme embedding layer is used for obtaining a vector representation of a phoneme according to a phoneme sequence, the position embedding layer is used for obtaining a vector representation of a position according to a position-encoded text, the concatenation module is used for concatenating the vector representation of the phoneme and the vector representation of the position, the encoder is used for extracting a rhythmic latent feature and a positional latent feature of annotated text according to the vector representation of the phoneme and the vector representation of the position, the variance adapter is used for enhancing and expanding the speech feature, the rhythmic latent feature and the positional latent feature, the Mel decoder is used for implementing parallel decoding according to the output of the variance adapter to obtain a Mel spectrum of synthesized speech, and the vocoder is used for mapping the Mel spectrum of synthesized speech into a sound waveform to obtain synthesized speech;

[0032] The model training module is used to train the speech synthesis model according to the phoneme sequence and the standard Mel spectrum, and calculate the loss according to the error between the Mel spectrum of the synthesized speech and the standard Mel spectrum until the model converges to obtain the trained speech synthesis model;

[0033] The data acquisition module is also used to obtain the input text to be synthesized and the identity tag of the target speaker;

[0034] The speech synthesis module is used to input the input text and the identity label of the target speaker into the trained speech synthesis model to synthesize speech with the speech characteristics of the target speaker.

[0035] The present application analyzes a large amount of speech scene data and finds that when people speak and communicate, emotional features such as joy, anger, sorrow, and happiness are not very obvious in intonation or tone. Instead, the sense of frustration or emphasis of language is an important measure of emotional expression and speech authenticity. Accordingly, the embodiment of the present application encodes the phoneme sequence according to the position of the target type text in the annotated text to obtain a position-coded text with position features, and uses the phoneme sequence and the position-coded text to train the speech synthesis model. The model learns the position features from the position-coded text instead of directly learning the position features from the input text, which can improve the performance of the model while ensuring the efficiency of model training. Correspondingly, the speech synthesis model designed by the present application splices the vector representation of the phoneme sequence and the vector representation of the position-coded text through a splicing module and then inputs them into the encoder, so that the encoder can learn the rhythmic latent features and the positional latent features of the target type text in parallel without increasing the complexity and input them into the variance adapter. The variance adapter strengthens the fusion of speech features and the above features, while ensuring that the model complexity is small, so that the trained model synthesizes the speech of a specific speaker with not only a sense of pause, but also a sense of emphasis on the position of the target type of text. This method ensures the sound quality of the synthesized speech, further improves the naturalness and thus improves the experience, while also making the model training method simple and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0037] Figure 1 is a schematic flow chart of a method for synthesizing speech for a specific speaker provided in an embodiment of the present application;

[0038] Figure 2 This is a schematic diagram of a speech synthesis system framework provided in an embodiment of the present application;

[0039] Figure 3 is a schematic diagram of an algorithm structure of a speech synthesis model provided by the present application according to some embodiments;

[0040] Figure 4 This is a schematic diagram of the voice data structure of a single speaker provided in this application;

[0041] Figure 5 This is a schematic diagram of a text data structure corresponding to a speech data structure of a single speaker provided in this application;

[0042] Figure 6This is a schematic diagram of the structure of speech data of multiple speakers provided by this application;

[0043] Figure 7 This is another example of a multi-speaker speech data structure diagram provided by the present application;

[0044] Figure 8 This is an example of a pinyin text data diagram provided in an embodiment of the present application;

[0045] Fig. 9 It is a schematic diagram of a speech synthesis device for a specific speaker provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0047] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0048] At present, although emotional information is added to some speech synthesis technologies, these technologies generally have some problems, such as difficulty in model training, reduced sound quality information and unclear emotional information. This application has conducted corresponding analysis on some speech scenes and found that when ordinary people speak, their emotional characteristics are not completely expressed in joy, anger, sorrow and happiness. When people speak and communicate, most of them are in a very normal speaking atmosphere. In this case, the sense of frustration and emphasis of speech and language are important criteria for measuring emotional expression and speech authenticity.

[0049] Therefore, how to improve the sense of pause and emphasis of synthesized speech and improve the naturalness of speech has become an urgent problem to be solved.

[0050] In order to facilitate understanding of the solutions in this application, some technical concepts are briefly introduced below:

[0051] Duration: The duration of a sound wave refers to the duration of the sound wave vibration, which is determined by the duration of the vibration of the sound-producing body. The longer the vibration time, the longer the sound wave.

[0052] Pitch: Sometimes called tone, the pitch is determined by the frequency of the sound wave vibration. The higher the vibration frequency, the higher the pitch. In Chinese, the tone of Chinese characters and the intonation of sentences are mainly determined by pitch.

[0053] Energy: It is the way to express the intensity of a sound, indicating the strength of the sound. The energy of a sound has the function of distinguishing the meaning of words and certain grammatical functions in Chinese, such as determining the difference between unstressed and stressed tones.

[0054] Due to its special complexity, Chinese has many characteristics such as grammatical structure, grammatical rules, acoustic characteristics, and rhythmic structure. In Chinese, a syllable is generally a Chinese character. Tone is an indispensable part of the syllable structure. Tone is usually used to indicate the rise and fall of a syllable when it is pronounced. In addition to being mainly determined by the change in pitch, the formation of tone is also reflected in the change in length. During the pronunciation process, the pronunciation body can adjust the changes in pitch and length at any time, thus forming different tones. Tones play an important role in distinguishing meanings. For example, tones are used to distinguish the meanings of words such as "subject matter" and "genre", "practice" and "contact" in Chinese phonetics. In Chinese, there are pauses in the middle of the pronunciation of continuous sentences, and different words in a sentence will use light or heavy tones according to the semantics of the context. These grammatical structures, grammatical rules, acoustic characteristics, and rhythmic structures of Chinese jointly form the ups and downs, emphasis on tone, tone emotion, and rhythmic rhythm of Chinese pronunciation.

[0055] In order to solve the above problems, the present application embodiment provides a method 100 for synthesizing speech for a specific speaker and a system framework. Figure 1 and Figure 2 The method is described, wherein Figure 1 is a schematic flow chart of a method for synthesizing speech for a specific speaker provided in an embodiment of the present application. Figure 2 This is a schematic diagram of a speech synthesis system framework provided in an embodiment of the present application.

[0056] S110, obtaining training data.

[0057] Specifically, the training data includes an identity tag of each speaker among multiple speakers, a standard Mel-spectrogram corresponding to the speech audio data of each speaker, and annotated text corresponding to the speech audio data of each speaker and its corresponding phoneme sequence.

[0058] The identity tag of each speaker and the phoneme sequence corresponding to the annotated text of the speaker's speech audio are bound together as training data to train the speech synthesis model.

[0059] Exemplarily, speech audio data is input into a Mel filter to obtain the standard Mel spectrum, and the standard Mel spectrum is used to calculate the loss using a loss function with the Mel spectrum for synthesized speech output by the model, so that the model converges.

[0060] In one example, the method of obtaining the phoneme sequence includes:

[0061] S111, constructing a phoneme alignment dictionary.

[0062] Among them, the phoneme alignment dictionary is used to represent the mapping relationship between pinyin and phonemes and the mapping relationship between prosodic labels and newly defined phonemes. The newly defined phonemes are sounds used to distinguish the pause durations of different prosodic labels.

[0063] S112, performing pinyin processing on the text with the prosody tag to obtain a second pinyin text with the prosody tag.

[0064] S113, obtaining phoneme text data.

[0065] Specifically, the second pinyin text with prosodic tags is converted into phoneme text data using a phoneme alignment dictionary, and the phoneme text data includes phonemes corresponding to the pinyin part of the second pinyin text and newly defined phonemes corresponding to the prosodic tag part of the second pinyin text.

[0066] S114, converting the phoneme text data into a phoneme sequence based on a phoneme identification mapping method.

[0067] Among them, the phoneme identification mapping method uses the corresponding mapping file to map the phonemes in the phoneme alignment dictionary to corresponding sequences to generate a phoneme sequence that can be recognized by the model.

[0068] By adopting the above method, the present application creatively converts the prosody label into a specific definition factor included in the phoneme sequence, so that the model can learn the speech characteristics of the prosody label while ensuring the speech characteristics of the speaker's speech audio. This method can simply and efficiently obtain a variety of prosody features, further improving the naturalness of the model's synthesized speech.

[0069] S120, obtaining position-coded text.

[0070] Specifically, the position of the target type text in the annotated text is determined, and the phoneme sequence is encoded according to the position to obtain the position-encoded text.

[0071] In one example, the target type text includes a name type text or a place name type text.

[0072] In one example, a method for obtaining position-encoded text includes determining a tag corresponding to a target type text according to a position, and encoding a phoneme sequence using a one-hot encoding method according to the tag and a phoneme sequence to obtain the position-encoded text.

[0073] The position refers to the position of the target type text in the annotated text, for example, from the nth column of the mth row to the nth + 3th column of the m+1th row. The form of the mark can be set as required, and this application does not limit this.

[0074] The above method obtains the position encoded text by marking the position and encoding it using the one-hot encoding method according to the mark. It can simply and efficiently obtain the encoded data with the position characteristics of the target type of text, thereby improving the efficiency of the early work of model training.

[0075] S130, constructing a speech synthesis model.

[0076] Specifically, the constructed speech synthesis model includes a phoneme embedding layer, a position embedding layer, a concatenation module, an encoder, a variance adapter, a Mel decoder and a vocoder.

[0077] Figure 3 is a schematic diagram of an algorithm structure of a speech synthesis model provided by the present application according to some embodiments, such as Figure 2 As shown:

[0078] The phoneme embedding layer is used to obtain the vector representation of the phoneme according to the phoneme sequence. The vector representation is a vector matrix.

[0079] The position embedding layer is used to obtain the vector representation of the position based on the position encoding text;

[0080] The concatenation module is used for concatenating the vector representation of the phoneme and the vector representation of the position to obtain a concatenated vector matrix;

[0081] The input of the encoder is a concatenated vector matrix, which includes a vector matrix of phonemes and a vector matrix of positions. The encoder extracts the rhythmic latent features and positional latent features of the annotated text according to the vector matrix of phonemes and the vector matrix of positions, and inputs these features into the variance adapter;

[0082] The variance adapter is used to enhance and expand the speech features, prosodic latent features and position latent features;

[0083] The positional encoding is used to construct a matrix with the same dimension as the output of the variance adapter, which is then added to the matrix with the same dimension as the output of the variance adapter to obtain the input of the Mel decoder using multi-headattention.

[0084] The Mel decoder is used to implement parallel decoding on its input to obtain the Mel spectrum of the synthesized speech;

[0085] The vocoder is used to map the Mel spectrum of the synthesized speech into a sound waveform to obtain the synthesized speech.

[0086] In one example, the vocoder uses a generative adversarial network model (HiFi-GAN) for efficient high-fidelity speech synthesis, and the vocoder structure of the model includes a first convolution layer, a first leaky-relu activation function layer, a transposed convolution layer, a multi-receptive field fusion (multi-receptive field fusion, MRF) layer, a second leaky-relu activation function layer, a second convolution layer, and a tanh activation function layer. The upsampling parameters of the upsampling module of the generator in the model of this embodiment are [10, 4, 4], the convolution kernel size in the upsampling module is [20, 8, 8], the number of hidden layers of the MRF module of the generator of the model is 128, and the residual expansion size of the multi-receptive field fusion module is [[1, 2], [2, 6], [3, 12]].

[0087] Among them, the upsampling parameters of the generator are sampling parameters used to upsample the Mel spectrum parameters and restore them to speech. The upsampling parameters and other parameters such as the upsampling convolution kernel parameters jointly affect the duration and accuracy of the final speech.

[0088] Compared with the original HiFi-GAN model, the optimized HiFi-GAN model in this application reduces the upsampling parameters of the generator, the convolution kernel size of the upsampling structure, and the residual expansion size parameters of the MRF, thereby improving the voice and efficiency while ensuring the voice quality.

[0089] In one example, the speech features include pitch features, energy features, and sound length features of the speech audio data.

[0090] In one example, a speech synthesis model is built under the Tensorflow framework.

[0091] Among the existing deep learning frameworks, such as Tensorflow, Pytorch, and Paddlepaddle, Tensorflow is more optimized than other frameworks in terms of model training and deployment. For example, the Tensorflow framework is not only portable, efficient, and scalable, but can also run on different computers, from smartphones to computer clusters; its API is more stable and compatible; its distributed architecture does not take too much time to train models for large data sets, because it can run on multiple CPUs, GPUs, or a mixture of both at the same time; at the same time, due to its larger development team, the efficiency of the framework is constantly improving, and its deployment solutions are more complete, such as tensorflow-serving, which has more commercial support, making commercial deployment more convenient and easy. At the same time, its tensorRT can further improve operating efficiency, which is unmatched by other frameworks. Therefore, deploying the speech synthesis model under the Tensorflow framework can not only improve the model training speed, but also make the commercial deployment of the model more convenient and easy.

[0092] S140, training a speech synthesis model.

[0093] Specifically, the phoneme sequence and the standard Mel spectrum are input into the model as training data to train the speech synthesis model, wherein the parameters of the loss function of the model include the Mel spectrum and the standard Mel spectrum of the synthesized speech, and the parameters of the model are updated according to the loss function until the model converges to obtain a trained speech synthesis model.

[0094] S150, obtaining input data.

[0095] Specifically, the input data includes input text to be speech synthesized and an identity tag of a target speaker.

[0096] In one example, the input file is preprocessed, including the following steps:

[0097] S151, obtaining a first pinyin text.

[0098] The first pinyin text is obtained by pinyinizing the input text.

[0099] S152, determining the target type text in the input text.

[0100] S153, identifying polyphonetic characters in the target type text.

[0101] S154, re-determine the first pinyin text.

[0102] Specifically, the first pinyin text is re-determined according to the polyphonetic characters of the target type text.

[0103] S155, obtaining a phoneme sequence corresponding to the input text according to the first pinyin text.

[0104] In one example, the method of re-determining the first pinyin text includes firstly performing pinyin processing on the input text to obtain the first pinyin text, then determining the target type text in the input text, and then identifying whether the target type text includes polyphones. If it does, querying a polyphone dictionary based on the polyphone to determine its corresponding pinyin, wherein the polyphone dictionary is used to characterize the mapping relationship between words including polyphones and pinyin, and the update of the polyphone dictionary supports hot update. Then, the first pinyin text is modified according to the pinyin corresponding to the queried polyphone to obtain a new first pinyin text.

[0105] The method of modifying the first pinyin text according to the pinyin corresponding to the queried polyphonetic character is as follows:

[0106] Determine the standard pinyin corresponding to the queried polyphone, and then determine whether the standard pinyin and the pinyin corresponding to the polyphone part in the target type text are the same. If different, modify the pinyin corresponding to the polyphone part in the target type text according to the standard pinyin to obtain a new first pinyin text.

[0107] It should be understood that if they are the same, there is no need to modify the first pinyin text.

[0108] Since Chinese has the characteristic of polyphones, especially for texts such as names and place names, the ratio of polyphones is relatively high. Therefore, polyphone recognition is performed on the target type text. If polyphones exist, the pinyin text is re-determined to obtain an accurate phoneme sequence. This solution effectively solves the problem of inaccurate speech synthesis of polyphone texts in Chinese, thereby improving the accuracy of speech synthesis.

[0109] S160, synthesizing speech using a speech synthesis model according to the input data.

[0110] Specifically, the input text and the identity tag of the target speaker are input into the trained speech synthesis model to synthesize speech with the speech characteristics of the target speaker.

[0111] Because in the general context of humans, words with specific names will have a sense of emphasis, such as names, place names or product names, etc. Accordingly, the present application utilizes the position of the target type text in the annotated text, and combines the phoneme sequence data of the annotated text to obtain the position-coded text, and uses the phoneme sequence and the position-coded text to train the speech synthesis model. While learning the implicit features of rhythm and the speech features of different speakers, the speech synthesis model can also learn the implicit features of the position of the target type text, so that the trained model synthesizes the speech of a specific speaker not only with a sense of pause, but also with a sense of emphasis on the position of the target type text. This method ensures the sound quality of the synthesized speech, further improves the naturalness and thus improves the experience, while also making the training of the model simple and efficient.

[0112] The method 100 is described below in conjunction with a specific embodiment. For example, the method 200 includes:

[0113] S210, obtaining training data.

[0114] This application prepares about 300 voices of speaker A, and extracts about 2 seconds of the speaker's voice from each voice. And annotates the content of each voice. The specific format is as follows Figure 4 and Figure 5 As shown, Figure 4 is a schematic diagram of the voice data structure of a single speaker provided in this application. Figure 5 This is a schematic diagram of a text data structure corresponding to a speech data structure of a single speaker provided in this application.

[0115] like Figure 4 As shown in the figure, "speaker01" represents the identity label of speaker A, i.e., ID, which contains a "labels.txt" and "wav" folders. The "wav" file stores the audio data of the speaker's voice, and "labels.txt" is the annotation text corresponding to the audio data. The specific content of the text is as follows: Figure 5 As shown. Figure 5 In each row, Figure 4 The name of each voice (spk_***) and the corresponding text content are separated by a space. The "#1, #2, #3" are prosody tags.

[0116] For multiple speakers, just list them one by one, such as Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of speech data of multiple speakers provided in this application.

[0117] The following takes the training of a speech synthesis model using the speech data of speaker A as an example for explanation.

[0118] S220, preprocessing the training data.

[0119] (1) Perform target type text recognition on the annotated text. The following takes name recognition as an example. The name text is recognized based on the position of the mark. The mark is as follows: Figure 5 as shown in the “[…]” in the figure.

[0120] (2) The sampling rate of the voice audio data is uniformly converted, that is, converted into the sampling rate defaulted by the system. Here, the default sampling rate set by this application is 24kHz.

[0121] (3) The annotated text is processed into pinyin, and one voice audio corresponds to one pinyin text.

[0122] The structure of the phonetic data after pinyinization is as follows: Figure 7 As shown, Figure 7 is another example of a multi-speaker speech data structure diagram provided by this application. Figure 7 In each speaker folder, there is no "labels.txt" file, but there is an additional "spk_00*.lab" file, which is the pinyin text content of the corresponding speech. Figure 8 As shown, Figure 8 This is an example of a pinyin text data graph provided in an embodiment of the present application, including the pinyin part and the prosodic tag part of the text excluding the prosodic tag.

[0123] It can be seen that when the annotated text is converted to pinyin, the prosodic labels remain unchanged. The purpose of this part is to treat the corresponding prosodic labels as phonemes when converting pinyin to phonemes, and then give them corresponding phonetic features when aligning phonemes.

[0124] S230, obtaining a phoneme sequence.

[0125] First, a phoneme alignment dictionary is constructed, and the phoneme alignment dictionary is shown in Table 1 below. Then, the phoneme sequence is obtained by querying the phoneme alignment dictionary according to the pinyin text with prosody tags.

[0126] Table 1

[0127]

[0128]

[0129] In this embodiment, the prosody label is mainly converted as a blank sound of different durations. The setting method here is not unique. This solution mainly distinguishes different prosody by setting different numbers of "sil" sounds. It should be noted that the "sil" sound here needs to be distinguished from the blank sound of punctuation marks.

[0130] Taking Figure 5 the corresponding pinyin text "ni2hao3 qing3 wen4 ni2 shi4#3zhang1 san1#3ma5" of the labeled text "spk_005 Hello, are you #3[Zhang San]#3?" in [reference] as an example, by querying Table 1, it is determined that the corresponding phoneme text data is "n i2 h ao3 q ing3 w en4 n i2 sh i4 sil sil sil zh ang1 s an1 silsil sil m a5". Among them, the phonemes corresponding to the pinyin part of this phoneme text data (i.e., the phoneme text data of the second pinyin text) include "n i2 h ao3 q ing3 w en4 n i2 sh i4 zh ang1 s an1 m a5", and the newly defined phonemes corresponding to the prosody label part include "sil sil sil" and "sil sil sil".

[0131] Furthermore, the phoneme sequence is transformed into a phoneme sequence recognizable by the model based on the method of phoneme identification mapping.

[0132] S240, obtain the position encoding text.

[0133] Determine whether each labeled text includes the marker "[…]" of the name position. If so, perform one-hot encoding on the phoneme sequence corresponding to the labeled text to obtain the position encoding text.

[0134] Taking Figure 5 the labeled text "spk_005" in [reference] as an example, the position encoding text after one-hot encoding the phoneme sequence "n i2h ao3 q ing3 w en4 n i2 sh i4 sil sil sil zh ang1 s an1 sil sil sil m a5" according to the position marker "[Zhang San]" is "000000000000000111100000". The model can extract the position features of this sequence based on the "1111" therein, thereby strengthening the emphasis on the speech at this position.

[0135] S250, extract the speech duration file.

[0136] In this embodiment, the general model built into the Montreal Forced Aligner (MFA) algorithm is used to predict the phoneme duration in each piece of speech. The MFA algorithm can generate a phoneme sequence according to the above phoneme alignment dictionary at the same time.

[0137] S260, extract the speech features.

[0138] The speech features are extracted according to the following formula based on the speech audio data and the phoneme duration data obtained by MFA.

[0139] Extract the pitch feature according to formula (1):

[0140]

[0141] Among them, s i is the value of the sampling point of the i-th audio in a sound frame, and n is the number of sampling points contained in each sound frame. median is the median of all the sampling points.

[0142] Extract energy features according to formula (2):

[0143]

[0144] Note that the pitch features and energy features here are extracted according to each phoneme, that is, each phoneme corresponds to a pitch and energy.

[0145] According to formula (3), the duration feature of each phoneme is extracted.

[0146] duration=(P e -P s )*sample_rate*hop_size formula (3)

[0147] Among them, P e is the end time of a phoneme, P s is the start time of the phoneme, and these two times can be obtained through the speech duration file generated by MFA. sample_rate represents the sampling frequency of the speech audio (the sampling rate of 24kHz is taken as an example in this embodiment), and hop_size represents the number of samples staggered between two adjacent sampling windows of the speech. Note that the number of sound length features of each speech should be consistent with the number of phonemes in the speech.

[0148] Finally, the state characteristic data of the pitch is calculated according to formula (4), namely:

[0149]

[0150] Among them, pitch min Indicates the minimum value of pitch, pitch max Indicates the maximum value of pitch, pitch mean Indicates the average value, pitch std Represents standard deviation.

[0151] Similarly, the state characteristic data of energy calculated according to formula (5) is as follows:

[0152]

[0153] S270, training a speech synthesis model.

[0154] Specifically, Figure 3 As shown in the figure, the speech synthesis model is built under the TensorFlow framework. Figure 3 The model shown inputs the above-obtained phoneme sequence, name position coding text, and the pitch, energy and length feature data of the speech obtained in S260, wherein the phoneme embedding layer encodes the phoneme sequence into a first vector matrix; the position embedding layer encodes the name position coding text into a second vector matrix; the splicing module splices the first vector matrix and the second vector matrix into a third vector matrix and inputs the third vector matrix to the encoder. The encoder processes the phoneme sequence part and the position part in the third vector matrix in parallel, and extracts the rhythmic latent features and positional latent features of the annotated text at the same time; the variance adapter enhances and expands the pitch, energy and length feature data of the speech, the rhythmic latent features and the positional latent features. The Mel decoder is used to implement parallel decoding according to the output of the variance adapter to obtain the Mel spectrum of the synthesized speech. The loss is calculated according to the error between the Mel spectrum of the synthesized speech and the standard Mel spectrum according to the loss function until the model converges to obtain the trained speech synthesis model.

[0155] In the above embodiment, when performing text encoding of speech audio annotation, the prosody tag is used as a special phoneme and the phoneme of the spoken content as a phoneme sequence to participate in the training of the model, which improves the sense of frustration of the speech synthesized by the model. Although the emotional information is added, the difficulty of training the model is not increased. In addition, the present application found in the experiment that in general contexts, there is an emphasis on the tone of the name. For example, "Excuse me, are you Zhang San?" For the speech in the position of "Zhang San", the current speech synthesis technology does not focus on this sentence. In the above embodiment, the position of "Zhang San" is extracted for the model to learn, so that the pronunciation of the model in the name position is strengthened. The emphasis is strengthened, thereby further improving the naturalness of the synthesized speech and making it closer to the real speech.

[0156] Further targeting the situation of polyphonetic characters in Chinese, the present application also provides a method for processing input text to be synthesized speech, as shown in the following method 300.

[0157] S310: Obtain input text of speech to be synthesized.

[0158] For example, the input text is "I am Jie Meili".

[0159] S320, performing pinyin processing on the input text.

[0160] The first pinyin text after pinyinizing "我是解美丽" is "wo3 shi4 jie3 mei3 li4".

[0161] S330, identify the name text in the input text.

[0162] Identify the name type text as "谢美丽" according to the text at the marked position.

[0163] S340, identify polyphonic characters.

[0164] Query the polyphonic characters in the name text according to the polyphonic character table shown in Table 2, and determine the pinyin of the polyphonic character.

[0165] Table 2

[0166] Polyphonetic characters Name Pinyin Non-name pinyin untie xie4 jie3 one shan4 dan1 … … …

[0167] According to Table 2, the pinyin of the surname "解" is "xie4".

[0168] In addition, Table 2 supports hot update, so there is no need to modify other modules every time Table 2 is updated.

[0169] S350, re-determine the first pinyin text.

[0170] Re-determine the first pinyin text as "wo3 shi4 xie4 mei3 li4" according to the pinyin of the surname "谢" in Table 2.

[0171] S360, determine the phoneme sequence.

[0172] Determine the phoneme sequence of the first pinyin text as "w o3 sh i4 x ie4 m ei3 li4" according to the phoneme alignment dictionary.

[0173] The above embodiments perform polyphonic character recognition and adjustment of the pinyin text on the input text, well taking into account the characteristics of Chinese polyphonic characters, and can improve the accuracy of the phoneme sequence, thereby improving the accuracy and naturalness of the model synthesized speech.

[0174] According to the above specific speaker voice synthesis method, the present application also provides a specific speaker voice synthesis device, as Fig. 9 shown, Fig. 9 is a schematic diagram of a specific speaker voice synthesis device provided by an embodiment of the present application. The device includes:

[0175] The data acquisition module 410 is used to acquire the identity tag of each speaker among the multiple speakers, the standard Mel spectrum corresponding to the speech audio data of each speaker, and the annotated text corresponding to the speech audio data of each speaker and its corresponding phoneme sequence;

[0176] A preprocessing module 420 is used to determine the position of the target type text in the annotated text, and encode the phoneme sequence according to the position to obtain the position encoded text;

[0177] A model building module 430 is used to build a speech synthesis model, which includes a phoneme embedding layer, a position embedding layer, a concatenation module, an encoder, a variance adapter, a Mel decoder and a vocoder, wherein the phoneme embedding layer is used to obtain a vector representation of a phoneme according to a phoneme sequence, the position embedding layer is used to obtain a vector representation of a position according to a position-encoded text, the concatenation module is used to concatenate the vector representation of the phoneme and the vector representation of the position, the encoder is used to extract the rhythmic latent features and the positional latent features of the annotated text according to the vector representation of the phoneme and the vector representation of the position, the variance adapter is used to enhance and expand the speech features, the rhythmic latent features and the positional latent features, the Mel decoder is used to implement parallel decoding according to the output of the variance adapter to obtain the Mel spectrum of the synthesized speech, and the vocoder is used to map the Mel spectrum of the synthesized speech into a sound waveform to obtain the synthesized speech;

[0178] A model training module 440 is used to train a speech synthesis model according to a phoneme sequence and a standard Mel spectrum, and calculate a loss according to an error between the Mel spectrum of the synthesized speech and the standard Mel spectrum until the model converges to obtain a trained speech synthesis model;

[0179] The data acquisition module 410 is also used to acquire the input text to be speech synthesized and the identity tag of the target speaker;

[0180] The speech synthesis module 450 is used to input the input text and the identity tag of the target speaker into the trained speech synthesis model to synthesize speech with the speech characteristics of the target speaker.

[0181] In one example, the apparatus further comprises:

[0182] The text-to-pinyin module 460 is used to perform pinyin processing on the input text to obtain a first pinyin text;

[0183] A target type text determination module 470 is used to determine a target type text in an input text;

[0184] Polyphone recognition module 480, used to recognize polyphones in target type text;

[0185] A phonetic text determination module 490, for re-determining the first phonetic text according to the polyphonetic characters of the target type text;

[0186] The phoneme sequence determination module 4100 is used to obtain the phoneme sequence corresponding to the input text according to the first pinyin text.

[0187] In one example, the apparatus further comprises:

[0188] A phoneme alignment dictionary construction module 4110 is used to construct a phoneme alignment dictionary, which is used to characterize the mapping relationship between pinyin and phonemes and the mapping relationship between prosodic labels and newly defined phonemes, where the newly defined phonemes are sounds used to distinguish pause durations of different prosodic labels;

[0189] The pinyin text determination module 490 is further used to perform pinyin processing on the text with the prosody tag to obtain a second pinyin text with the prosody tag;

[0190] The phoneme text data determination module 4120 is used to convert the second pinyin text with the prosody tag into phoneme text data by using the phoneme alignment dictionary, wherein the phoneme text data includes the phonemes corresponding to the pinyin part of the second pinyin text and the newly defined phonemes corresponding to the prosody tag part of the second pinyin text;

[0191] The phoneme sequence determination module 4100 is also used to convert the phoneme text data into a phoneme sequence based on a phoneme identifier mapping method.

[0192] For other implementations of the device, please refer to method 100, method 200 and method 300, which will not be described in detail here.

[0193] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0194] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0195] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0196] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0197] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0198] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for synthesizing speech for a specific speaker, characterized in that: include: Obtaining an identity tag of each speaker among multiple speakers, a standard Mel spectrum corresponding to the speech audio data of each speaker, and annotated text corresponding to the speech audio data of each speaker and its corresponding phoneme sequence; Determine the position of the target type text in the annotated text, and encode the phoneme sequence according to the position to obtain a position-encoded text, wherein the target type is a name type or a place name type; Constructing a speech synthesis model, the speech synthesis model includes a phoneme embedding layer, a position embedding layer, a splicing module, an encoder, a variance adapter, a Mel decoder and a vocoder, wherein the phoneme embedding layer is used to obtain a vector representation of a phoneme according to the phoneme sequence, the position embedding layer is used to obtain a vector representation of a position according to the position-encoded text, the splicing module is used to splice the vector representation of the phoneme and the vector representation of the position, the encoder is used to extract the rhythmic latent features and the positional latent features of the annotated text according to the vector representation of the phoneme and the vector representation of the position, the variance adapter is used to enhance and expand the speech features, the rhythmic latent features and the positional latent features, the Mel decoder is used to implement parallel decoding according to the output of the variance adapter to obtain the Mel spectrum of the synthesized speech, and the vocoder is used to map the Mel spectrum of the synthesized speech into a sound waveform to obtain the synthesized speech; Training the speech synthesis model according to the phoneme sequence and the standard Mel spectrum, and calculating the loss according to the error between the Mel spectrum of the synthesized speech and the standard Mel spectrum until the model converges, thereby obtaining the trained speech synthesis model; Acquire an input text to be speech synthesized and an identity tag of a target speaker, perform phonetic processing on the input text to obtain a first phonetic text; determine a target type text in the input text; identify polyphones of the target type text; re-determine the first phonetic text based on the polyphones of the target type text; and acquire a phoneme sequence corresponding to the input text based on the first phonetic text; The input text and the identity tag of the target speaker are input into the trained speech synthesis model to synthesize speech with the speech characteristics of the target speaker.

2. The method according to claim 1, characterized in that The identifying of the polyphones in the target type text includes: searching for corresponding pinyins in a polyphone dictionary according to the target type text, wherein the polyphone dictionary is used to represent a mapping relationship between words including polyphones and pinyins.

3. The method according to claim 1, characterized in that The annotated text is text with a prosody tag, and the method further includes: Constructing a phoneme alignment dictionary, wherein the phoneme alignment dictionary is used to characterize the mapping relationship between pinyin and phonemes and the mapping relationship between prosody labels and newly defined phonemes, wherein the newly defined phonemes are sounds used to distinguish pause durations of different prosody labels; Performing pinyin processing on the text with the prosody tag to obtain a second pinyin text with the prosody tag; The second phonetic text with the prosody tag is converted into phonetic text data using the phonetic alignment dictionary, wherein the phonetic text data includes phonemes corresponding to the phonetic part of the second phonetic text and newly defined phonemes corresponding to the prosody tag part of the second phonetic text; The phoneme text data is converted into the phoneme sequence based on a phoneme identifier mapping method.

4. The method according to claim 1, characterized in that: The step of encoding the phoneme sequence according to the position to obtain position-encoded text comprises: Determine a mark corresponding to the target type text according to the position; The phoneme sequence is encoded using a one-hot encoding method according to the tag and the phoneme sequence to obtain a positional encoded text.

5. The method according to claim 1, characterized in that: The vocoder uses a generative adversarial network model for efficient and high-fidelity speech synthesis. The upsampling parameters of the upsampling module of the generator of the model are [10, 4, 4], the convolution kernel size in the upsampling module is [20, 8, 8], the number of hidden layers of the multi-receptive field fusion module of the generator of the model is 128, and the residual expansion size of the multi-receptive field fusion module is [[1, 2], [2, 6], [3, 12]].

6. The method according to claim 1, characterized in that The speech features include pitch features, energy features and sound length features of the speech audio data.

7. The method according to claim 1, characterized in that The constructing of the speech synthesis model comprises: Build a speech synthesis model under the Tensorflow framework.

8. A speech synthesis device for a specific speaker, characterized in that: include: A data acquisition module, used to acquire an identity tag of each speaker among multiple speakers, a standard Mel spectrum corresponding to the speech and audio data of each speaker, and annotated text corresponding to the speech and audio data of each speaker and its corresponding phoneme sequence; A preprocessing module, used for determining the position of the target type text in the annotated text, and encoding the phoneme sequence according to the position to obtain a position-encoded text, wherein the target type is a name type or a place name type; A model construction module, used for constructing a speech synthesis model, the speech synthesis model includes a phoneme embedding layer, a position embedding layer, a splicing module, an encoder, a variance adapter, a Mel decoder and a vocoder, wherein the phoneme embedding layer is used to obtain a vector representation of a phoneme according to the phoneme sequence, the position embedding layer is used to obtain a vector representation of a position according to the position-encoded text, the splicing module is used to splice the vector representation of the phoneme and the vector representation of the position, the encoder is used to extract the rhythmic latent features and the positional latent features of the annotated text according to the vector representation of the phoneme and the vector representation of the position, the variance adapter is used to enhance and expand the speech features, the rhythmic latent features and the positional latent features, the Mel decoder is used to implement parallel decoding according to the output of the variance adapter to obtain the Mel spectrum of the synthesized speech, and the vocoder is used to map the Mel spectrum of the synthesized speech into a sound waveform to obtain the synthesized speech; A model training module, used for training the speech synthesis model according to the phoneme sequence and the standard Mel spectrum, and calculating the loss according to the error between the Mel spectrum of the synthesized speech and the standard Mel spectrum, until the model converges, thereby obtaining the trained speech synthesis model; The data acquisition module is also used to acquire an input text to be speech synthesized and an identity tag of a target speaker, perform phonetic processing on the input text to obtain a first phonetic text; determine a target type text in the input text; identify polyphones of the target type text; re-determine the first phonetic text according to the polyphones of the target type text; and acquire a phoneme sequence corresponding to the input text according to the first phonetic text; The speech synthesis module is used to input the input text and the identity tag of the target speaker into the trained speech synthesis model to synthesize speech with the speech characteristics of the target speaker.

Images