A training method for a speech synthesis model and a speech synthesis method

Abstract

The present application is applicable to the field of computer technology, and in particular, to a training method for a speech synthesis model and a speech synthesis method. The training method for the speech synthesis model includes: annotating the pronunciation of a first speech sample to obtain a text sequence of the first speech sample; inputting the first speech sample and the text sequence of the first speech sample in pairs into a preset speech synthesis model for processing to obtain an output audio corresponding to the text sequence of the first speech sample and an audio feature of the first speech sample; adjusting the parameters of the speech synthesis model according to the audio feature of the first speech sample and the output audio until the preset training conditions are met to obtain a trained speech synthesis model. In the present application, pronunciation annotation can be performed based on sample speech of a dialect, and then a speech synthesis model capable of synthesizing a speech with the pronunciation characteristics of the dialect can be obtained, thereby improving the quality of dialect speech synthesis.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a training method for a speech synthesis model and a speech synthesis method. Background Art

[0002] Speech synthesis technology refers to the technology of generating artificial speech through mechanical and electronic methods. Text-to-speech (TTS) technology belongs to speech synthesis. It is a technology that converts text information generated by the computer itself or input from the outside into understandable and fluent spoken output. At present, speech synthesis technology based on deep learning is gradually maturing and can synthesize common languages ​​such as Mandarin and English. However, the quality of speech synthesis for dialects is still not high. Summary of the invention

[0003] The embodiments of the present application provide a training method for a speech synthesis model, a readable storage medium, and a terminal device, which can improve the quality of dialect speech synthesis.

[0004] In a first aspect, an embodiment of the present application provides a method for training a speech synthesis model, comprising:

[0005] Performing pronunciation annotation on the first voice sample to obtain a text sequence of the first voice sample;

[0006] Inputting the first speech sample and the text sequence of the first speech sample into a preset speech synthesis model in pairs to obtain output audio corresponding to the text sequence of the first speech sample and audio features of the first speech sample;

[0007] According to the audio features of the first speech sample and the output audio, the parameters of the speech synthesis model are adjusted to obtain a trained speech synthesis model.

[0008] Furthermore, the speech synthesis model includes an audio processing module, a text encoding module, a decoding module and a synthesis module;

[0009] The step of inputting the first speech sample and the text sequence of the first speech sample in pairs into a preset speech synthesis model for processing to obtain an output audio corresponding to the text sequence of the first speech sample and an audio feature of the first speech sample includes:

[0010] Inputting the first speech sample into the audio processing module for processing to obtain audio features of the first speech sample;

[0011] Inputting the text sequence of the first speech sample into the text encoding module for processing to obtain a feature vector corresponding to the text sequence;

[0012] Inputting the feature vector corresponding to the text sequence into the decoding module for processing to obtain a frequency spectrum corresponding to the feature vector;

[0013] The frequency spectrum is input to the synthesis module for processing to obtain the output audio.

[0014] Furthermore, the step of annotating the pronunciation of the first voice sample to obtain a text sequence of the first voice sample includes:

[0015] Constructing a first data set according to the first speech sample;

[0016] Pronunciation annotation is performed on the first voice sample based on the first data set to obtain a text sequence of the first voice sample.

[0017] Specifically, since the first data set already includes the pronunciation labels of each character in the dialect, the pronunciation labels of each character in the text corresponding to the first voice sample can be determined by traversing the first data set, and the text sequence of the first voice sample can be determined based on the preset encoding rules.

[0018] Further, constructing the first data set according to the first speech sample includes:

[0019] Pronunciation annotation is performed on the text corresponding to the first voice sample according to the first voice sample, and a pronunciation label of each word in the first data set is determined.

[0020] Furthermore, the step of performing pronunciation annotation on the first voice sample based on the first data set to obtain a text sequence of the first voice sample includes:

[0021] Traversing the first data set, and performing pronunciation labeling on each word of the first speech sample;

[0022] The pronunciation sequence number corresponding to the pronunciation label of each word is determined according to a preset coding rule to obtain a text sequence of the first voice sample.

[0023] Furthermore, the training method of the speech synthesis model further includes:

[0024] Constructing a second data set according to a second speech sample and the first data set;

[0025] Performing pronunciation annotation on the second voice sample based on the second data set to obtain a text sequence of the second voice sample;

[0026] The preset speech synthesis model is trained based on the second speech sample and a text sequence of the second speech sample.

[0027] Furthermore, the above-mentioned constructing a second data set according to the second speech sample and the first data set includes:

[0028] Performing pronunciation annotation on the text corresponding to the second voice sample according to the second voice sample and determining a pronunciation label for each word in the second voice sample;

[0029] The pronunciation label of each word in the text corresponding to the second voice sample is added to the first data set to obtain a second data set.

[0030] Furthermore, the training of the preset speech synthesis model based on the second speech sample and the text sequence of the second speech sample includes:

[0031] Inputting the text sequence of the second voice sample into a preset speech synthesis model for processing to obtain corresponding output audio;

[0032] Processing the second speech sample by using the preset speech synthesis model to obtain a frequency spectrum feature corresponding to the second speech sample;

[0033] According to the frequency spectrum features corresponding to the second speech sample and the output audio, the parameters of the speech synthesis model are adjusted to obtain a trained speech synthesis model.

[0034] In a second aspect, an embodiment of the present application provides a speech synthesis method, comprising:

[0035] Get the text to be synthesized;

[0036] The text to be synthesized is input into a trained speech synthesis model for processing, and the output audio output by the speech synthesis model is obtained. The speech synthesis model is a speech synthesis model trained by any of the above-mentioned speech synthesis model training methods.

[0037] Furthermore, the speech synthesis model includes a text encoding module, a decoding module and a synthesis module;

[0038] The step of inputting the text to be synthesized into a trained speech synthesis model for processing, and obtaining the output audio output by the speech synthesis model, includes:

[0039] Inputting the text to be synthesized into the text encoding module for processing to obtain a feature vector corresponding to the text to be synthesized;

[0040] Inputting the feature vector corresponding to the text to be synthesized into the decoding module for processing to obtain a frequency spectrum corresponding to the feature vector;

[0041] The frequency spectrum is input to the synthesis module for processing to obtain the output audio.

[0042] Furthermore, the step of inputting the text to be synthesized into the text encoding module for processing to obtain a feature vector corresponding to the text to be synthesized includes:

[0043] Generate a corresponding text sequence according to the text to be synthesized;

[0044] Feature extraction is performed on the text sequence to obtain a feature vector corresponding to the text to be synthesized.

[0045] A third aspect of an embodiment of the present application provides a terminal device, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein when the processor executes the computer-readable instructions, the processor implements the steps of any one of the above-mentioned speech synthesis model training methods or the steps of any one of the above-mentioned speech synthesis methods.

[0046] In a fourth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the steps of the training method of the speech synthesis model described in any one of the first aspects or the steps of any one of the speech synthesis methods described above.

[0047] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0048] Compared with the prior art, the embodiments of the present application have the following beneficial effects: it is possible to annotate the pronunciation of speech samples of the dialect, and to train a language synthesis model based on the text sequence of the speech samples and the collected speech samples, thereby obtaining a speech synthesis model that can synthesize a speech with the pronunciation characteristics of the dialect, thereby improving the quality of dialect speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 is a flow chart for implementing a method for training a speech synthesis model provided in one embodiment of the present application;

[0051] Figure 2 It is a flowchart for implementing the process of forming a first data set in the training method of a speech synthesis model provided in one embodiment of the present application;

[0052] Figure 3It is a flowchart for implementing a process of determining a text sequence of a first speech sample in a training method for a speech synthesis model provided in an embodiment of the present application;

[0053] Figure 4 It is a flowchart for implementing S103 of the training method of the speech synthesis model provided in one embodiment of the present application;

[0054] Figure 5 is a flowchart of a method for training a speech synthesis model provided in another embodiment of the present application;

[0055] Figure 6 is a flowchart of a speech synthesis method provided by another embodiment of the present application;

[0056] Figure 7 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0058] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0059] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0060] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0061] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0062] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0063] The speech synthesis method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0064] See also Figure 1 , Figure 1 is a flow chart of the implementation of the training method of the speech synthesis model provided in the embodiment of the present application, such as Figure 1 As shown, the training method of the above speech synthesis model may include:

[0065] S101: Annotate a first voice sample with pronunciation to obtain a text sequence of the first voice sample.

[0066] Specifically, by collecting a first voice sample, the first voice sample is then annotated with pronunciation according to the pronunciation of each character in the first voice sample. Among them, the first voice sample can be a voice sample of the most common version of a certain dialect, such as a voice sample of the most common version of the dialect recorded for more than 10 hours. During recording, it is necessary to cover all pronunciations and words of the dialect as much as possible, and to cover commonly used Chinese characters as much as possible. According to the pronunciation of each character in the first voice sample, each character in the text corresponding to the first voice sample is annotated with pronunciation, and then the pronunciation labels of each character in the text corresponding to the first voice sample are determined. Since the first voice sample covers all pronunciations, words and commonly used Chinese characters of the dialect, the pronunciation labels of each character in the dialect can be basically determined. It should be noted that the above-mentioned pronunciation label refers to a label used to identify the pronunciation method of each character in the dialect, and each character has its corresponding pronunciation label. It is understandable that due to the existence of polyphonic characters, one character can also correspond to multiple pronunciation labels. It should be noted that the pronunciation annotation of each word in the text corresponding to the first voice sample means: annotating each word in the text corresponding to the first voice sample through the pronunciation of each word in the audio of the first voice sample, including annotating the initial consonant, final consonant, tone and pronunciation method of the word. Here, the initial consonant and final consonant of each word can be annotated based on the customized initial consonant table and final consonant table. At the same time, the tone can be customized according to the pronunciation tone contained in the corresponding dialect. The pronunciation method can also be set according to the pronunciation characteristics of the dialect, such as erhua sound, stress, prolonged sound, etc. The above-mentioned initial consonant table and final consonant table can be based on the initial consonants and final consonants in Mandarin Pinyin, and then add the initial consonants and final consonants missing in Mandarin Pinyin (for example, the initial consonants and final consonants are set with reference to the pronunciation method of English phonetic symbols).

[0067] Specifically, since the first voice sample already includes the pronunciation labels of each character in the dialect, the pronunciation labels of each character in the text corresponding to the first voice sample can be determined by traversing the pronunciation labels of each character in the text corresponding to the first voice sample, and the text sequence of the first voice sample can be determined based on preset encoding rules.

[0068] It should be noted that the above-mentioned preset coding rule is to encode initial consonants, finals, tones, and pronunciation methods using digital coding. Specifically, the data range can be divided according to the types of initial consonants, finals, tones, and pronunciation methods, and the values ​​within a specific range are used to represent initial consonants, finals, tones, and pronunciation methods. Exemplarily, the above-mentioned preset coding rule is to use reserved bits from 0 to 39 to represent the initial consonant sequence number; use reserved bits from 40 to 99 to represent the final sequence number, use reserved bits from 100 to 105 to represent the tone sequence number, and use reserved bits from 106 to 110 to represent the pronunciation method sequence number. Representing the pronunciation of a dialect by means of data coding increases the scalability of pronunciation representation. If it is found that there are new initial consonants, finals, tones, or pronunciation methods in this dialect, only a new sequence number needs to be added. For the above reserved bits, if there are many forms of initial consonants and finals in a certain dialect, it is only necessary to increase the corresponding reserved range.

[0069] As a possible implementation manner, the above S101 may include the following steps:

[0070] Constructing a first data set according to the first speech sample;

[0071] Pronunciation annotation is performed on the first voice sample based on the first data set to obtain a text sequence of the first voice sample.

[0072] Specifically, a first speech sample is collected, and then a first data set is formulated for the dialect corresponding to the speech sample based on the first speech sample. It should be noted that the first data set refers to a data set containing each word in the first speech sample and the pronunciation label corresponding to each word. Each word and the pronunciation label corresponding to each word are stored correspondingly.

[0073] Specifically, the construction process of the first data set can be performed by a preprocessing device (including but not limited to a terminal device with computing capabilities such as a computer and a server) by: Figure 2 The processing shown yields:

[0074] S1011: Perform pronunciation annotation on the text corresponding to the first voice sample according to the first voice sample and determine the pronunciation label of each word in the first data set.

[0075] Generally, the original data format of the voice sample is WAV audio format, which is the audio format closest to lossless, so its size is relatively large. In practical applications, in order to reduce the subsequent calculation amount, the voice sample can be converted from WAV audio format to PCM audio format in advance. Preferably, considering that the voice sample may contain silent signals, these silent signals generally appear in the time period before the user speaks, the time period after the user speaks, and the time period when the user pauses in the middle of speaking, and do not contain any useful information. Therefore, these silent signals can be removed from the voice sample to reduce interference with the final result.

[0076] When the first voice sample is collected, the first voice sample and the text corresponding to the first voice sample are stored accordingly. After the first voice sample is collected, the corresponding text is annotated word by word based on the pronunciation of the first voice sample, that is, each word in the text is annotated with pronunciation according to the pronunciation of the first voice sample. The pronunciation annotation includes initial consonants, final consonants, tones, and pronunciation methods. The pronunciation annotation of each word in the text is performed based on a custom initial consonant table, final consonant table, tone table, and pronunciation method table.

[0077] In practical applications, the above-mentioned self-defined initial consonant table and final consonant table can take the initial consonants and final consonants in Mandarin Pinyin as the basis, and then add the initial consonants and final consonants missing in Mandarin Pinyin. The above-mentioned tone table can also take the tones of Mandarin Pinyin as the basis, and then add the tones missing in Mandarin Pinyin. The above-mentioned pronunciation table can be set according to the pronunciation characteristics of the dialect, such as erhua, stress, prolonged sound, etc.

[0078] For example, taking Shaanxi dialect as an example, the initial consonant table, final consonant table, tone table and pronunciation table of Shaanxi dialect are defined as follows:

[0079] Initial consonant table: bpmfdtnlgkhjqx zh ch sh rzcsyw ng (where ng can refer to the English phonetic symbol / / pronunciation. );

[0080] Table of finals: aoeiuv ai ei ui ao ou iu ie ve er an en in un vn ang enging ong or (the pronunciation of or can refer to the English phonetic symbol / (r) / pronunciation. );

[0081] Tone table: 100 (first tone), 101 (second tone), 102 (third tone), 103 (fourth tone), 104 (light tone);

[0082] Pronunciation methods: 107 (stressed sound), 108 (prolonged sound), 109 (r-ending). It should be noted that since some pronunciations in dialects are different from those in Mandarin, initials determined according to the pronunciation of English phonetic symbols (such as "ng" shown in the above example) can be added to the custom initial table, and finals determined according to the pronunciation of English audio (such as "or" shown in the above example) can be added to the custom final table to ensure that the pronunciation of the dialect can be more comprehensively reflected.

[0083] Since the first speech sample has been collected to cover as many common characters in the dialect as possible, the pronunciation label of each character in the first dataset can be determined based on the above pronunciation annotation process, and then the first dataset corresponding to the dialect can be constructed.

[0084] The pronunciation label of each character in the dataset includes initials, finals, tones, and pronunciation methods. Exemplarily, for the dataset corresponding to the Shaanxi dialect, the pronunciation labels of each character in the corresponding "I love Tiananmen in Beijing" are as follows:

[0085] I: ng e 103 107

[0086] Love: ng ai 100

[0087] North: b ei 102

[0088] Beijing: j ing 102

[0089] Sky: t i an 102

[0090] Peace: ng an 102

[0091] Gate: m en 102

[0092] It should be noted that when the pronunciation method is omitted, it is defaulted to the normal pronunciation. It should also be noted that since the pronunciations of the characters "I", "love", and "peace" in the Shaanxi dialect start with vowels or semi-vowels, their pronunciations are different from those of "wo", "ai", and "an" in Mandarin. According to the Shaanxi dialect, the pronunciation of "I" is "nge", the pronunciation of "love" is "ngai", and the pronunciation of "peace" is "ngan".

[0093] It should also be noted that there are cases of polyphonic characters in the dialect, so one character can correspond to multiple pronunciation labels. Exemplarily: The pronunciation labels of the character "after" in the Shaanxi dialect are as follows:

[0094] {

[0095] "After": {

[0096] "her 103": ["the day after tomorrow"],

[0097] "hor 103": ["back"],

[0098] "hou 100": ["behind"]

[0099] }

[0100] }

[0101] Specifically, since the first data set already includes the pronunciation labels of each character in the dialect, the pronunciation labels of each character in the text corresponding to the first voice sample can be determined by traversing the first data set, and the text sequence of the first voice sample can be determined based on the preset encoding rules.

[0102] Specifically, the text sequence determination process of obtaining the first voice sample can be performed by a preprocessing device (including but not limited to a terminal device with computing capabilities such as a computer and a server) by: Figure 3 The processing shown yields:

[0103] S1021: Traverse the first data set and perform pronunciation labeling on each word of the first speech sample.

[0104] Specifically, by traversing the first data set, the pronunciation of each word in the text corresponding to the first voice sample is annotated with a pronunciation tag.

[0105] Exemplarily, if the corresponding text in the first voice sample is “I love Tiananmen Square in Beijing”, the pronunciation label of “I” is “ng e 103 107”; the pronunciation label of “love” is “ng ai 100”; the pronunciation label of “bei” is “b ei 102”; the pronunciation label of “jing” is “j ing 102”; the pronunciation label of “tian” is “ti an 102”; the pronunciation label of “an” is “ngan 102”; the pronunciation label of “men” is “m en 102”; that is, the pronunciation label corresponding to “I love Tiananmen Square in Beijing” is “ng e103 107ng ai100b ei 102j ing 102t i an 102ng an 102m en 102”.

[0106] S1022: Determine the pronunciation sequence number corresponding to the pronunciation label of each word according to a preset coding rule to obtain a text sequence of the first voice sample.

[0107] Specifically, due to the complexity of dialects, in order to more conveniently express the pronunciation of dialects, it is not necessary to use symbols that specifically specify tones and finals to express them. It is only necessary to know how many initials, finals, tones, and pronunciation methods there are in the dialect. Different digital numbers are used to express different contents, that is, the pronunciation numbers corresponding to the pronunciation labels of each word are determined according to the preset coding rules. It should be noted that the preset coding rules are to use digital coding to encode initials, finals, tones, and pronunciation methods. Specifically, the data range can be divided according to the types of initials, finals, tones, and pronunciation methods, and the values ​​within a specific range are used to represent initials, finals, tones, and pronunciation methods. Exemplarily, the preset coding rules are to use reserved bits from 0 to 39 to represent the initial consonant number; use reserved bits from 40 to 99 to represent the final number, use reserved bits from 100 to 105 to represent the tone number, and use reserved bits from 106 to 110 to represent the pronunciation method number. Representing the pronunciation of a dialect by means of data encoding increases the scalability of pronunciation representation. If new initials, finals, tones or pronunciation methods are found in this dialect, only a new serial number needs to be added. For the above reserved positions, if there are many forms of initials and finals in a certain dialect, only the corresponding reserved range needs to be increased.

[0108] In a specific application, by specifying the serial numbers corresponding to each initial consonant, each final consonant, each tone, and each pronunciation mode, the corresponding serial numbers can be determined based on the pronunciation labels, and then the text sequence corresponding to the first voice sample can be determined.

[0109] For example, the pronunciation tag corresponding to “I love Beijing Tiananmen” in Shaanxi dialect is: ng e 103107ngai 100b ei 102j ing 102t i an 102ng an 102m en 102; and its corresponding text sequence is “23 42103 107 23 46 100 0 47 102 11 62 102 5 43 55 102 23 55 102 2 56 102”.

[0110] S102: Input the first speech sample and the text sequence of the first speech sample in pairs into a preset speech synthesis model for processing to obtain output audio corresponding to the text sequence of the first speech sample and audio features of the first speech sample.

[0111] In this embodiment, by inputting the text sequence of the first voice sample into a preset speech synthesis model for processing, the corresponding output audio can be obtained, and the speech synthesis model can also obtain the audio features corresponding to the first voice sample by processing the first voice sample, and use the spectral features of the first voice sample as a standard for verifying the output audio, so as to verify whether the model training is completed. In the application, the above-mentioned speech synthesis model can be Tacotron, which is an end-to-end deep learning TTS model. The core structure of the model is an encoder to decoder model of an attention mechanism. During the training process, the model is trained based on the collected first voice sample and the text sequence of the first voice sample. Among them, the above-mentioned speech synthesis model includes an audio processing module, a text encoding module, a decoding module and a synthesis module.

[0112] Specifically, S102 may include: Figure 4 The process shown:

[0113] S1031: Input the first speech sample to the audio processing module for processing to obtain audio features of the first speech sample.

[0114] Specifically, for the first speech sample, in order to more intuitively obtain the changing law of its frequency domain, the audio features are extracted based on the audio processing module. The above audio features can use Mel spectrum, which is a relatively common audio feature. For sound, it is actually a one-dimensional time domain signal. It is difficult to intuitively see the changing law of the frequency domain. Its frequency domain information can be obtained by using Fourier transform, but the time domain information is lost, and the change of the frequency domain with the time domain cannot be seen, so it is impossible to describe the sound well. Mel spectrum is a spectrum used to represent short-term audio information. Its principle is based on the logarithmic spectrum represented by nonlinear Mel scale and its linear cosine transform. In a specific implementation, the speech sample can be first converted from time domain to frequency domain by short-time Fourier transform, and then its logarithmic energy spectrum is convolved with a triangular filter group distributed according to Mel scale, and finally the vector composed of the output of each filter is discrete cosine transformed to obtain its Mel spectrum.

[0115] The so-called short-time Fourier transform (STFT) is to perform Fourier transform on short-time signals. The long-time signal is divided into short-time signals by framing. The long-time signal is windowed, and then the Fourier transform (FFT) is performed on each frame of data. Finally, the Fourier transform results of each frame of data are stacked along another dimension to obtain a two-dimensional signal form similar to a picture. Since the first speech sample mentioned above is a sound signal, the two-dimensional signal obtained by STFT expansion is the spectrogram. The spectrogram is often a large picture. In order to obtain the sound features of the appropriate size, it is often transformed through a mel-scale filter bank to transform it into a mel spectrum. Performing cepstrum analysis on the mel spectrum (taking the logarithm and performing DCT transformation) will obtain the mel cepstrum.

[0116] S1032: Input the text sequence of the first voice sample into the text encoding module for processing to obtain a feature vector corresponding to the text sequence.

[0117] Specifically, by one-hot encoding the text sequence of the first voice sample, its corresponding one-hot vector can be obtained. One-hot encoding is a binary representation of a categorical variable. By using one-hot encoding, the value range of discrete features (text features) can be extended to the Euclidean space, so that the discrete features (text features) can correspond to a value in the Euclidean space, and the features contained in the text sequence of the first voice sample can be better extracted to obtain a feature vector corresponding to the text sequence.

[0118] One-hot encoding is to represent the above text sequence using a one-hot vector containing 0 and 1. Then the one-hot vector corresponding to the text sequence is input into the text encoding module for processing.

[0119] In this embodiment, the text encoding module includes a preprocessing module and a feature extraction module.

[0120] The above pre-processing module (per-net) includes two fully connected layers, whose neurons are 256 and 128 respectively. The number of hidden units in the first layer is consistent with the number of input units, and the number of hidden units in the second layer is half of the number of hidden units in the first layer. Both use the relu activation function and 0.5 dropout, and introduce nonlinear transformation through the pre-processing module to accelerate the convergence of the model.

[0121] The above feature extraction module (CBHG) consists of 1-D convolution bank, highway network, and bidirectional GRU.

[0122] Among them, the 1-D convolution bank consists of 1 convolution layer, 1 pooling layer and 2 one-dimensional convolution layers. The first convolution layer includes K one-dimensional filters of different sizes, where the filter size is 1, 2, 3...K. These convolution kernels of different sizes extract context information of different lengths. The outputs of the k convolution kernels of different sizes are stacked together, that is, the extracted context information is combined together, and then input into the pooling layer for processing to increase local invariance. The above pooling layer is a pooling layer with stride 1 and width 2. The output of the pooling layer is passed through two one-dimensional convolution layers. The filter size of the first convolution layer is 3, the stride is 1, and the activation function used is ReLu; the filter size of the second convolution layer is 3, the stride is 1, and it is only used as a linear transformation. Finally, the output of the convolution layer is residually connected with the vector input to the feature extraction module, that is, the output of the convolution layer and the vector input to the feature extraction module are added together. Using residual connection can alleviate the gradient diffusion problem caused by neural networks.

[0123] The structure of each layer of the highway network is: put the input into two one-layer fully connected networks at the same time. The activation functions of these two networks use ReLu and sigmoid functions respectively. Assuming that the input is input, the output of ReLu is output1, and the output of sigmoid is output2, then the output of the highway layer is output=output1*output2+input*(1-output2).

[0124] Bidirectional GRU is a bidirectional GRU, which inputs the output into the bidirectional GRU to extract the time sequence of the text. Among them, GRU is a variant of RNN, and it uses a threshold mechanism like LSTM.

[0125] After the feature extraction module (CBHG), a feature vector corresponding to the text sequence can be obtained.

[0126] S1033: Inputting the feature vector corresponding to the text sequence into the decoding module for processing to obtain a frequency spectrum corresponding to the feature vector.

[0127] Specifically, based on the attention mechanism, the feature vector of the text sequence is sent to the decoding module for processing to obtain its corresponding spectrum.

[0128] The above decoding module includes pre-net, Attention-RNN, and Decoder-RNN. Among them, the structure of Pre-net is the same as the preprocessing module (per-net) in the text encoding module. The structure of Attention-RNN is a layer of RNN containing 256 GRUs. It takes the output of pre-net and the output of attention as input, and outputs to decoder-RNN after passing through GRU units. Decoder-RNN is a two-layer residual GRU. Its output is the sum of the input and the output after passing through the GRU unit. The initial input is all 0 vectors. When predicting, the Mel spectrum of r frames will be predicted each time, which can effectively reduce the amount of calculation and accelerate the convergence of the model.

[0129] S1034: Input the frequency spectrum to the synthesis module for processing to obtain the output audio.

[0130] The above-mentioned synthesis module includes a post-processing unit and a synthesis unit. The output Mel spectrum is input into the post-processing unit for conversion. After obtaining the corresponding linear spectrum, the Griffin-Lim algorithm is finally used to synthesize the linear spectrum output by the post-processing net into the output audio.

[0131] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0132] S103: According to the audio features of the first speech sample and the output audio, the parameters of the speech synthesis model are adjusted to obtain a trained speech synthesis model.

[0133] Specifically, the spectral features of the first speech sample and the spectral sequence of the output audio output by the speech synthesis model can be compared first. If the two are inconsistent, it means that the output of the model is still inaccurate. The training loss value of the speech synthesis model can be calculated, and the model parameters of the speech synthesis model can be adjusted.

[0134] In this embodiment, assuming that the model parameter of the speech synthesis model is W1, the training loss value is back-propagated to modify the model parameter W1 of the speech synthesis model to obtain the modified parameter W2. After modifying the parameters, the step of processing the first speech sample and the text sequence of the first speech sample is continued, that is, the next training process is started. In this training process, a new set of the first speech samples and the text sequence of the first speech samples can be processed, and the training loss value corresponding to the text sequence can be calculated. The training loss value is back-propagated to modify the model parameter W2 of the speech synthesis model to obtain the modified parameter W3, ..., and so on. The above process is repeated continuously. Each training process can be trained for a new set of first speech samples, and the model parameters are modified until the preset training conditions are met. The training condition can be that the number of training times reaches a preset number threshold, and optionally, the number threshold can be 100,000 times; the training condition can also be that the speech synthesis model converges; since the number of training times may not reach the number threshold, but the speech synthesis model has converged, unnecessary work may be repeated; or the speech synthesis model cannot converge, which may lead to an infinite loop and the training process cannot be terminated. Based on the above two situations, the training condition can also be that the number of training times reaches the number threshold or the speech synthesis model converges. When the training conditions are met, a trained speech synthesis model can be obtained.

[0135] In another embodiment, due to the difference in the same dialect due to different regions, the pronunciation of some words may be different. Figure 5 As shown, the training method of the above speech synthesis model may also include the following steps:

[0136] S104: construct a second data set according to a second speech sample and the first data set.

[0137] Specifically, the second voice sample is a variant of the first voice sample in a different region of the same dialect. The dialect voice samples are supplemented by recording multiple voice samples of the dialect in other regions.

[0138] The second data set is supplemented with the pronunciation label of each word in the second speech sample on the basis of the first data set, thereby ensuring that the second data set contains as many pronunciations of each word in the dialect as possible.

[0139] As a possible implementation of the present application, the above S104 may include the following steps:

[0140] Performing pronunciation annotation on the text corresponding to the second voice sample according to the second voice sample and determining a pronunciation label for each word in the second voice sample;

[0141] The pronunciation label of each word in the text corresponding to the second voice sample is added to the first data set to obtain a second data set.

[0142] Specifically, the second data set can be constructed by adding the pronunciation labels of the characters in the second voice sample on the basis of the first data set.

[0143] S105: Perform pronunciation annotation on the second voice sample based on the second data set to obtain a text sequence of the second voice sample.

[0144] Here, since the second data set also includes the pronunciation labels of each word of the dialect, the specific pronunciation annotation process is similar to S102, and please refer to S102 for details.

[0145] S106: Training the preset speech synthesis model based on the audio features of the second speech sample and the text sequence of the second speech sample.

[0146] Here, by inputting the text sequence of the second voice sample into a preset speech synthesis model for processing, the corresponding output audio can be obtained. The speech synthesis model can also obtain the spectral features corresponding to the second voice sample by processing the second voice sample, and use the spectral features of the second voice sample as a standard for verifying the output audio, so as to verify whether the model training is completed.

[0147] As a possible implementation of the present application, the above S106 may include the following steps:

[0148] Inputting the text sequence of the second voice sample into a preset speech synthesis model for processing to obtain corresponding output audio;

[0149] Processing the second speech sample by using the preset speech synthesis model to obtain a frequency spectrum feature corresponding to the second speech sample;

[0150] According to the frequency spectrum features corresponding to the second speech sample and the output audio, the parameters of the speech synthesis model are adjusted to obtain a trained speech synthesis model.

[0151] Specifically, the specific process of adjusting the parameters of the speech synthesis model according to the spectral features corresponding to the second speech sample and the output audio to obtain the trained speech synthesis model can be seen in S103, which will not be repeated here.

[0152] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0153] In summary, the training method of the speech synthesis model provided in the embodiment of the present application can perform pronunciation annotation based on sample speech of the dialect, and train the language synthesis model based on the text sequence of the pronunciation annotation and the collected speech samples, thereby obtaining a speech synthesis model that can synthesize speech with the pronunciation characteristics of the dialect, thereby improving the quality of dialect speech synthesis.

[0154] See also Figure 6 Another embodiment of the present application provides a speech synthesis method, such as Figure 6 As shown, it includes:

[0155] S201: Obtain the text to be synthesized.

[0156] The text to be synthesized may be text data collected instantly by the user through input devices such as keyboards of terminal devices such as mobile phones and tablet computers. In a specific usage scenario of this embodiment, when the user wants to perform speech synthesis instantly, the speech synthesis mode of the terminal device can be turned on by clicking a specific physical key or virtual key before collecting the text to be synthesized. In this mode, the terminal device can process the text input by the user according to subsequent steps to obtain output audio corresponding to the text. The specific processing process will be described in detail later.

[0157] The text to be synthesized may also be text data originally stored in the terminal device, or text data obtained by the terminal device from a cloud server or other terminal device through a network. In another specific usage scenario of this embodiment, when a user wants to perform speech synthesis on an existing text to be synthesized, the user can open the speech synthesis mode of the terminal device by clicking a specific physical key or a virtual key, and select the text to be synthesized (the order of clicking the key and selecting the text can be interchanged, that is, the text can be selected first, and then the speech synthesis mode of the terminal device can be opened). Then, the terminal device can process the text to be synthesized according to the subsequent steps to obtain the output audio corresponding to the text to be synthesized. The specific processing process will be described in detail later.

[0158] S202: Input the text to be synthesized into a trained speech synthesis model for processing, and obtain output audio output by the speech synthesis model.

[0159] Among them, the speech synthesis model is the speech synthesis model described in the previous embodiment.

[0160] Specifically, by inputting the text to be synthesized into the speech synthesis model, the speech synthesis model can convert the text to be synthesized into its corresponding text sequence, determine its corresponding feature vector based on the text sequence, and then obtain the spectrum corresponding to the feature vector, perform speech synthesis based on the spectrum, and obtain the output audio corresponding to the text to be synthesized. After inputting into the preset speech synthesis model for processing, the corresponding output audio can be obtained.

[0161] In this embodiment, the speech synthesis model includes a text encoding module, a decoding module and a synthesis module, and the above S201 includes:

[0162] Inputting the text to be synthesized into the text encoding module for processing to obtain a feature vector corresponding to the text to be synthesized;

[0163] Inputting the feature vector corresponding to the text to be synthesized into the decoding module for processing to obtain a frequency spectrum corresponding to the feature vector;

[0164] The frequency spectrum is input to the synthesis module for processing to obtain the output audio.

[0165] In one embodiment, the step of inputting the text to be synthesized into the text encoding module for processing to obtain a feature vector corresponding to the text to be synthesized includes:

[0166] Generate a corresponding text sequence according to the text to be synthesized;

[0167] Feature extraction is performed on the text sequence to obtain a feature vector corresponding to the text to be synthesized.

[0168] The speech synthesis method provided in this embodiment is based on the speech synthesis model trained by the above-mentioned speech synthesis model training method. It can also construct a corresponding data set based on the dialect, and perform pronunciation annotation based on the data set, and train the language synthesis model based on the pronunciation-annotated text sequence and the collected speech samples, so as to obtain a speech synthesis model that can synthesize speech with the pronunciation characteristics of the dialect, thereby improving the quality of dialect speech synthesis.

[0169] Figure 7 A schematic block diagram of a terminal device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0170] like Figure 7As shown, the terminal device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps in the training method embodiments of the above-mentioned various speech synthesis models are implemented, for example Figure 1 Steps S101 to S103 are shown, or steps in the above-mentioned speech synthesis method.

[0171] Exemplarily, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 72 in the terminal device 7.

[0172] The terminal device 7 can be a computing device such as a mobile phone, a tablet computer, a desktop computer, a notebook, a PDA, a cloud server, etc. Those skilled in the art will understand that Figure 7 It is only an example of the terminal device 7 and does not constitute a limitation on the terminal device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 7 may also include input and output devices, network access devices, buses, etc.

[0173] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0174] The memory 71 may be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. The memory 71 may also be an external storage device of the terminal device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 7. Further, the memory 71 may also include both an internal storage unit and an external storage device of the terminal device 7. The memory 71 is used to store the computer program and other programs and data required by the terminal device 7. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0175] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0176] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0177] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0178] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0182] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for training a speech synthesis model, characterized in that: include: Constructing a first data set according to the first speech sample; Traversing the first data set, obtaining the pronunciation labels of each word in the first voice sample, and determining the pronunciation sequence corresponding to the pronunciation label of each word based on a preset coding rule, to obtain a text sequence corresponding to the first voice sample; wherein the first voice sample includes the pronunciation labels of each word of the dialect; the preset coding rule uses digital coding to divide the data range according to the types of initials, finals, tones, and pronunciation methods, and the values ​​in different data ranges represent different initials, finals, tones, and pronunciation methods respectively; The step of constructing a first data set according to the first voice sample includes: annotating the first voice sample by pronunciation based on a self-defined initial consonant table, final consonant table, tone table, and pronunciation mode table to construct the first data set; Inputting the first speech sample and the text sequence of the first speech sample into a preset speech synthesis model in pairs to obtain output audio corresponding to the text sequence of the first speech sample and audio features of the first speech sample; According to the audio features of the first speech sample and the output audio, adjusting the parameters of the speech synthesis model to obtain a trained speech synthesis model; Constructing a second data set according to a second speech sample and the first data set; the second speech sample is a variant of the same dialect of the first speech sample in different regions; Performing pronunciation annotation on the second voice sample according to the second data set to obtain a text sequence of the second voice sample; The preset speech synthesis model is trained based on the audio features of the second speech sample and the text sequence of the second speech sample, and the parameters of the speech synthesis model are adjusted to obtain a trained speech synthesis model.

2. The method for training a speech synthesis model according to claim 1, wherein: The speech synthesis model includes an audio processing module, a text encoding module, a decoding module and a synthesis module; The step of inputting the first speech sample and the text sequence of the first speech sample into a preset speech synthesis model in pairs to obtain an output audio corresponding to the text sequence of the first speech sample and an audio feature of the first speech sample includes: Inputting the first speech sample into the audio processing module for processing to obtain audio features of the first speech sample; Inputting the text sequence of the first speech sample into the text encoding module for processing to obtain a feature vector corresponding to the text sequence; Inputting the feature vector corresponding to the text sequence into the decoding module for processing to obtain a frequency spectrum corresponding to the feature vector; The frequency spectrum is input to the synthesis module for processing to obtain the output audio.

3. The method for training a speech synthesis model according to claim 1, wherein: The step of constructing a first data set according to the first speech sample includes: Pronunciation annotation is performed on the text corresponding to the first voice sample according to the first voice sample, and a pronunciation label of each word in the first data set is determined.

4. The method for training a speech synthesis model according to claim 1, wherein: The step of constructing a second data set according to the second speech sample and the first data set includes: Performing pronunciation annotation on the text corresponding to the second voice sample according to the second voice sample and determining a pronunciation label for each word in the text corresponding to the second voice sample; The pronunciation label of each word in the text corresponding to the second voice sample is added to the first data set to obtain a second data set.

5. The method for training a speech synthesis model according to claim 1, wherein: The training of the preset speech synthesis model based on the second speech sample and the text sequence of the second speech sample includes: Inputting the text sequence of the second voice sample into a preset speech synthesis model for processing to obtain corresponding output audio; Processing the second speech sample by using the preset speech synthesis model to obtain a frequency spectrum feature corresponding to the second speech sample; According to the frequency spectrum features corresponding to the second speech sample and the output audio, the parameters of the speech synthesis model are adjusted to obtain a trained speech synthesis model.

6. A speech synthesis method, characterized in that: include: Get the text to be synthesized; The text to be synthesized is input into a trained speech synthesis model for processing, and the output audio output by the speech synthesis model is obtained. The speech synthesis model is a speech synthesis model obtained based on the training method of the speech synthesis model described in any one of claims 1-5.

7. A speech synthesis method as claimed in claim 6, characterized in that: The speech synthesis model includes a text encoding module, a decoding module and a synthesis module; The step of inputting the text to be synthesized into a trained speech synthesis model for processing, and obtaining the output audio output by the speech synthesis model, includes: Inputting the text to be synthesized into the text encoding module for processing to obtain a feature vector corresponding to the text to be synthesized; Inputting the feature vector corresponding to the text to be synthesized into the decoding module for processing to obtain a frequency spectrum corresponding to the feature vector; The frequency spectrum is input to the synthesis module for processing to obtain the output audio.

8. The speech synthesis method according to claim 7, characterized in that: The step of inputting the text to be synthesized into the text encoding module for processing to obtain a feature vector corresponding to the text to be synthesized includes: Generate a corresponding text sequence according to the text to be synthesized; Feature extraction is performed on the text sequence to obtain a feature vector corresponding to the text to be synthesized.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the training method of the speech synthesis model as described in any one of claims 1 to 5 or the speech synthesis method as described in any one of claims 6 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the training method of the speech synthesis model as described in any one of claims 1 to 5 or the speech synthesis method as described in any one of claims 6 to 7.

Images