Model training method and device, speech synthesis method and device, electronic equipment and storage medium

Abstract

The invention provides a model training method and device, a speech synthesis method and device, electronic equipment and a storage medium, and the model training method comprises the steps: obtaining a reconstructed acoustic feature outputted by a generative model, and extracting a reconstructed periodic feature of the reconstructed acoustic feature; inputting the reconstructed periodic features into a discrimination model to obtain an authenticity discrimination result of the reconstructed periodic features output by the discrimination model; and performing confrontation training on the generative model and the discrimination model based on the authenticity discrimination result, and taking the generative model obtained by training as an acoustic model. According to the method, the device, the electronic equipment and the storage medium provided by the invention, the periodic characteristics capable of reflecting the timbre are explicitly taken as an optimization target, so that the timbre naturalness and fidelity of the synthesized voice are higher when the generative model is applied to voice synthesis through adversarial training.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model training, speech synthesis method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, speech synthesis has evolved from the initial simple text-to-speech (TTS) conversion to a synthesis technology that focuses on complex emotional expression and personalized timbre generation. How to accurately reconstruct and imitate a specific timbre has become an important challenge for speech synthesis technology.

[0003] Since everyone's pronunciation habits and voice characteristics are different, it is difficult to find a universal method to simulate all timbres. Even if a specific timbre can be simulated, it is difficult to ensure its naturalness and realism. Current speech synthesis schemes usually mix timbre, rhythm, content and other information together for constrained training, which also results in the naturalness of timbre reconstruction being less than expected. Summary of the invention

[0004] The present invention provides a model training, speech synthesis method, device, electronic device and storage medium, which are used to solve the defect that the naturalness of the timbre of speech synthesis in the related art does not meet the expectation.

[0005] The present invention provides a model training method, comprising: Acquiring reconstructed acoustic features output by the generation model, and extracting reconstructed periodic features of the reconstructed acoustic features; Inputting the reconstructed periodic feature into a discrimination model to obtain a true or false discrimination result of the reconstructed periodic feature output by the discrimination model; Based on the authenticity discrimination result, adversarial training is performed on the generative model and the discriminative model, and the generative model obtained through the training is used as the acoustic model.

[0006] According to a model training method provided by the present invention, the step of extracting the reconstructed periodic features of the reconstructed acoustic features comprises: Extracting reconstructed periodic features of the reconstructed acoustic features under multiple periodic components; The step of inputting the reconstructed periodic feature into a discrimination model to obtain a true or false discrimination result of the reconstructed periodic feature output by the discrimination model comprises: The reconstructed periodic features under each periodic component are respectively input into the discriminant models corresponding to each periodic component, and the true and false discrimination results output by the discriminant models corresponding to each periodic component are obtained.

[0007] According to a model training method provided by the present invention, the multiple periodic components are multiple periodic components based on prime numbers.

[0008] According to a model training method provided by the present invention, the step of extracting the reconstructed periodic features of the reconstructed acoustic features under multiple periodic components includes: converting the reconstructed acoustic features into a linear magnitude spectrogram; Reconstructed periodic features under multiple periodic components are extracted from the linear amplitude spectrum.

[0009] According to a model training method provided by the present invention, the adversarial training of the generation model and the discrimination model based on the authenticity discrimination result includes: Based on the authenticity discrimination result of the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature, performing adversarial training on the generation model and the discrimination model; The authenticity discrimination result of the real periodic feature is output by the discrimination model based on the real periodic feature, and the real periodic feature is extracted from the real acoustic feature.

[0010] According to a model training method provided by the present invention, the authenticity discrimination result based on the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature is used to perform adversarial training on the generation model and the discrimination model, including: Determine a discrimination loss based on the authenticity discrimination result of the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature, and perform parameter iteration on the discrimination model based on the discrimination loss; Based on the authenticity discrimination result of the reconstructed periodic feature, a generation loss is determined, and parameters of the generation model are iterated based on the generation loss.

[0011] The present invention also provides a speech synthesis method, comprising: Inputting the text into a speech synthesis model to obtain synthesized speech output by the speech synthesis model; The speech synthesis model includes an acoustic model trained by a model training method.

[0012] The present invention also provides a model training device, comprising: A feature extraction unit, used to obtain the reconstructed acoustic features output by the generation model, and extract the reconstructed periodic features of the reconstructed acoustic features; A true-false discrimination unit, used for inputting the reconstructed periodic feature into a discrimination model to obtain a true-false discrimination result of the reconstructed periodic feature output by the discrimination model; The adversarial training unit is used to perform adversarial training on the generative model and the discriminative model based on the authenticity discrimination result, and use the generative model obtained by the training as the acoustic model.

[0013] The present invention also provides a speech synthesis device, comprising: A speech synthesis unit, used to input text into a speech synthesis model to obtain synthesized speech output by the speech synthesis model; The speech synthesis model includes an acoustic model trained by a model training method.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the model training method or speech synthesis method described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the model training methods or speech synthesis methods described above.

[0016] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned model training methods or speech synthesis methods.

[0017] The model training, speech synthesis method, device, electronic device and storage medium provided by the present invention input the reconstructed periodic features of the reconstructed acoustic features output by the generation model into the discrimination model for authenticity discrimination, thereby performing adversarial training on the generation model and the discrimination model. In this process, the periodic features that can reflect the timbre are explicitly used as optimization targets, so that when the generation model obtained by the adversarial training is applied to speech synthesis, the timbre of the synthesized speech is more natural and realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is one of the flow charts of the model training method provided by the present invention.

[0020] Figure 2 It is the acoustic characteristic spectrum provided by the present invention.

[0021] Figure 3It is a schematic diagram of extracting periodic features provided by the present invention.

[0022] Figure 4 This is the second flow chart of the model training method provided by the present invention.

[0023] Figure 5 It is a flow chart of the speech synthesis method provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the model training device provided by the present invention.

[0025] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Speech synthesis is a technology that converts text information into audible speech signals, allowing machines to "speak" like humans. With the development of artificial intelligence technology, speech synthesis has evolved from the initial simple text-to-speech conversion to a synthesis technology that focuses on complex emotional expression and personalized timbre generation. As a very important attribute of speech, timbre determines the personality and recognizability of synthesized speech. Therefore, how to accurately reconstruct and imitate a specific timbre has become a key challenge in speech synthesis technology.

[0028] In the field of speech synthesis technology, traditional speech synthesis methods rely on a large number of pre-recorded speech samples and complex signal processing techniques to simulate and reconstruct specific timbres. The inherent limitations of this method restrict its promotion in practical applications.

[0029] First, traditional speech synthesis methods require a lot of storage space to save pre-recorded speech samples. As the number of speech samples increases, the required storage space also increases dramatically, which is undoubtedly a huge challenge for devices or application scenarios with limited resources.

[0030] Secondly, since everyone's pronunciation habits and voice characteristics are unique, it is difficult to find a universal method to simulate all timbres. Even if it is possible to simulate the timbre of a specific individual or group, it is difficult to ensure the naturalness and realism of the generated voice, which directly affects the overall effect of speech synthesis.

[0031] In order to solve the above problems, speech synthesis methods based on machine learning and deep learning have emerged. Such methods learn the characteristics of speech signals by training models and try to reconstruct signals similar to the original speech. However, although such methods have achieved remarkable results in the field of speech synthesis, when training models, they usually mix information such as timbre, rhythm, and content as constraints, and do not explicitly design optimization targets for timbre, which results in the trained speech synthesis models being less natural than expected in timbre reconstruction.

[0032] Moreover, such models usually use acoustic features such as Mel-Frequency Cepstral Coefficients (MFCCs) as modeling targets, and use Mel-Frequency Cepstral Coefficients (MFCCs) to construct losses to constrain the reconstruction process. Speech is a strong periodic signal, and the resonance peaks in its spectrogram are a strong periodic feature that contributes greatly to the timbre. However, the Mel-Frequency Cepstral Coefficients (MFCCs) compress some high-frequency information based on the auditory characteristics of the human ear, simplifying the reconstruction task and destroying the periodicity of the audio spectrogram, which directly leads to the loss of features that contribute to the timbre and affects the timbre reconstruction effect.

[0033] In view of the above problems, the present invention provides a model training method. Figure 1 It is one of the flow charts of the model training method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, obtaining the reconstructed acoustic features output by the generation model, and extracting the reconstructed periodic features of the reconstructed acoustic features.

[0034] Here, the generated model is the model to be trained, and specifically may be the initial model of the acoustic model to be trained.

[0035] For example, when the generative model is an initial model of an acoustic model for speech synthesis, text can be input into the generative model, and the generative model performs speech synthesis based on the input text, thereby outputting reconstructed acoustic features. It can be understood that the reconstructed acoustic features here are the acoustic features output by the generative model, and the reconstructed acoustic features are used to reflect the acoustic features of the synthesized speech generated based on the generative model. The acoustic features here can specifically be Fbank (FilterBank) features, Mel-spectrograms, perceptual linear prediction (PLP) features, etc., and the embodiments of the present invention do not specifically limit this.

[0036] After the reconstructed acoustic features are obtained, periodic features can be extracted for the reconstructed acoustic features. In the embodiment of the present invention, the periodic features extracted for the reconstructed acoustic features are recorded as reconstructed periodic features.

[0037] It is understandable that the periodicity of speech has an important influence on the perception of timbre. As a type of feature that reflects the periodicity of speech, periodicity features can characterize the periodicity information in the speech frequency domain, especially the resonance peak information that determines the timbre, and have a great influence and contribution to the timbre. By extracting the reconstructed periodic features of the reconstructed acoustic features, a basis can be provided for the subsequent evaluation of the reconstruction effect of the reconstructed acoustic features in terms of timbre.

[0038] Here, there are many ways to extract periodic features. For example, for a mel-spectrogram, the mel-spectrogram can be inversely transformed into a linear amplitude spectrum, and then the periodic features can be extracted for the linear amplitude spectrum; for another example, a model for extracting periodic features from acoustic features can be pre-trained, and the extraction of periodic features can be implemented based on the trained model, which is not specifically limited in the embodiments of the present invention.

[0039] Step 120: input the reconstructed periodic feature into a discrimination model to obtain a authenticity discrimination result of the reconstructed periodic feature output by the discrimination model.

[0040] Here, the discriminant model is a model used to discriminate the authenticity of the input periodic feature. It can be understood that if the periodic feature is true, it means that the acoustic feature corresponding to the periodic feature comes from real speech, and if the periodic feature is false, it means that the acoustic feature corresponding to the periodic feature comes from synthesized false speech. That is, the discrimination of the authenticity of the periodic feature can also be understood as the discrimination of whether the speech from which the acoustic feature comes is real speech or synthesized false speech.

[0041] After obtaining the reconstructed periodic features, the reconstructed periodic features can be input into the discriminant model, and the discriminant model performs authenticity discrimination on the input reconstructed periodic features, thereby obtaining authenticity discrimination results of the reconstructed periodic features output by the discriminant model. The authenticity discrimination results here can be from real speech, or from synthesized false speech, or from the probability of real speech or synthesized false speech, which is not specifically limited in the embodiment of the present invention.

[0042] Step 130: Based on the authenticity discrimination result, the generative model and the discriminative model are subjected to adversarial training, and the generative model obtained through the training is used as the acoustic model.

[0043] Specifically, after obtaining the authenticity discrimination result, adversarial training can be performed on the above-mentioned generation model and discrimination model based on the authenticity discrimination result.

[0044] In adversarial training, the generative model can be regarded as a generator, and the discriminative model can be regarded as a discriminator. The generator and the discriminator compete with each other. The generator uses the reconstructed periodic features of the output reconstructed acoustic features to be as close as possible to the real periodic features, in order to achieve the purpose of making it difficult for the discriminator to distinguish between the reconstructed periodic features and the real periodic features; the discriminator uses the output authenticity discrimination results to be consistent with the actual situation of the input periodic features, in order to achieve a more accurate and reliable authenticity discrimination effect.

[0045] During the adversarial training process, the true-false discrimination result can reflect the difference between the reconstructed periodic features of the reconstructed acoustic features output by the generation model and the true periodic features, thereby determining the generation loss for the generation model to guide the generation model to update and iterate in the direction of outputting the reconstructed acoustic features with periodic features closer to the true periodic features; and the true-false discrimination result can also reflect the accuracy of the discrimination model in distinguishing the true from the false of the input periodic features, thereby determining the discrimination loss for the discrimination model to guide the discrimination model to update and iterate in the direction of outputting the true-false discrimination results closer to the actual situation.

[0046] After completing the adversarial training for the generative model and the discriminative model, the trained generative model and the trained discriminative model can be obtained. The trained generative model not only has the ability to map text features to acoustic features, but also the periodic features of the reconstructed acoustic features output by the generative model are very close to the real periodic features, that is, the timbre of the synthesized speech based on the reconstructed acoustic features output by the generative model is very close to the real timbre, which can ensure the naturalness and realism of the synthesized speech timbre.

[0047] In the method provided in an embodiment of the present invention, the reconstructed periodic features of the reconstructed acoustic features output by the generative model are input into the discriminative model for authenticity discrimination, thereby performing adversarial training on the generative model and the discriminative model. In this process, the periodic features that can reflect the timbre are explicitly taken as optimization targets, so that when the generative model obtained through adversarial training is applied to speech synthesis, the timbre of the synthesized speech is more natural and realistic.

[0048] Moreover, the above training method involves two types of models, namely, generative models and discriminative models. It has a simple structure and is easy to transplant. It can be applied to any acoustic model that targets Mel-spectrogram or linear amplitude spectrogram to improve the timbre reconstruction capability of such acoustic models.

[0049] Based on the above embodiment, the discriminant model is composed of one or more two-dimensional convolutional layers.

[0050] Specifically, in the discriminative model, the two-dimensional convolutional layer can effectively extract the local features of the periodic features in the form of the input image, including the edges and energy changes of the periodic feature images.

[0051] Furthermore, the two-dimensional convolution layer can slide on the periodic feature image through the convolution kernel, thereby realizing the operation on the local area of ​​the periodic feature image. Due to the characteristics of the convolution operation, the discriminant model can have position invariance, that is, no matter where the feature used for authenticity discrimination appears in the periodic feature image, the discriminant model can correctly identify it.

[0052] Moreover, in the discriminant model, the application of multiple two-dimensional convolutional layers can realize hierarchical feature extraction, that is, the shallow two-dimensional convolutional layer extracts low-level features, while the subsequent two-dimensional convolutional layers gradually extract higher-level features. As a result, the discriminant model can gradually build a complex feature hierarchy under multiple layers of two-dimensional convolution, and finally realize the authenticity discrimination of periodic features based on the hierarchical features.

[0053] Based on any of the above embodiments, in step 110, extracting the reconstructed periodic features of the reconstructed acoustic features includes: Reconstructed periodic features of the reconstructed acoustic features under multiple periodic components are extracted.

[0054] Specifically, different people have different timbres, and the corresponding formants in speech are also different, so the periodic components of harmonics in speech are also different. In order to enable the reconstructed periodic features to cover more information related to timbre, thereby bringing better timbre generalization, when extracting periodic features for reconstructed acoustic features, corresponding periodic feature extraction can be performed for multiple periodic components respectively.

[0055] Thus, the reconstructed periodic features of the reconstructed acoustic features under multiple periodic components can be obtained, that is, the reconstructed acoustic features can have multiple reconstructed periodic features, and each reconstructed periodic feature corresponds to a periodic component. For example, the reconstructed periodic features of the reconstructed acoustic features under 2, 3, 5, 7, 11 and other periodic components can be extracted.

[0056] It can be understood that the reconstructed periodic features under multiple periodic components can relatively completely and comprehensively cover the timbre-related information in the speech corresponding to the reconstructed acoustic features.

[0057] Accordingly, in step 120, the step of inputting the reconstructed periodic feature into the discrimination model to obtain the authenticity discrimination result of the reconstructed periodic feature output by the discrimination model includes: The reconstructed periodic features under each periodic component are respectively input into the discriminant models corresponding to each periodic component, and the true and false discrimination results output by the discriminant models corresponding to each periodic component are obtained.

[0058] Specifically, for the reconstructed periodic features under multiple periodic components, the discriminant model corresponding to each periodic component can be pre-trained. It can be understood that for each periodic component, there is a corresponding discriminant model, and the discriminant model is used to distinguish the authenticity of the periodic features under the periodic component. That is, the number of periodic components is consistent with the number of discriminant models.

[0059] After obtaining the reconstructed periodic features of the reconstructed acoustic features under multiple periodic components, the reconstructed periodic features under each periodic component can be input into the discriminant model under the corresponding periodic component, thereby obtaining the true and false discrimination results output by the discriminant model under each periodic component. It can be understood that each discriminant model outputs a true and false discrimination result for the reconstructed periodic features under the periodic component, and the number of true and false discrimination results obtained is consistent with the number of periodic components.

[0060] In the subsequent adversarial training combining the generative model and the discriminative model, the true and false discrimination results under each periodic component can be used as the basis for guiding the parameter iteration of the generative model and the corresponding discriminative model. That is, the parameter iteration of the generative model can refer to the true and false discrimination results under all periodic components, and the parameter iteration of each discriminative model refers to the true and false discrimination results under the periodic component corresponding to the discriminative model.

[0061] In the method provided in the embodiment of the present invention, by extracting periodic features from multiple periodic components and distinguishing the authenticity thereof respectively, the coverage of the periodic features in the periodic harmonic range can be ensured, thereby ensuring the generalization of the model training in timbre reconstruction optimization.

[0062] Based on any of the above embodiments, the multiple periodic components are multiple periodic components based on prime numbers.

[0063] Here, a prime number refers to a natural number greater than 1 that cannot be divided by other natural numbers except 1 and itself, such as 2, 3, 5, 7, etc. The unique divisibility property of prime numbers makes the distribution of periodic components based on prime numbers more uniform on the spectrum, which can reduce the overlap and interference between different periodic components, which is convenient for maintaining the independence and clarity of the periodic features under different periodic components.

[0064] Furthermore, multiple periodic components based on prime numbers can cover a wider harmonic range, so that the periodic characteristics of the multiple periodic components obtained can more comprehensively reflect the relevant information of the timbre.

[0065] Based on any of the above embodiments, in step 110, extracting the reconstructed periodic features of the reconstructed acoustic features under multiple periodic components includes: converting the reconstructed acoustic features into a linear magnitude spectrogram; Reconstructed periodic features under multiple periodic components are extracted from the linear amplitude spectrum.

[0066] Specifically, for acoustic features, such as reconstructing acoustic features, a method for extracting periodic features can be expressed as the following steps: First, the reconstructed acoustic features can be converted into linear amplitude spectrograms. It can be understood that compared with the Mel spectrum Figure 1 Compared with the reconstructed acoustic features, the linear amplitude spectrogram can retain most of the periodic features of speech in the frequency domain, including the resonance peak information that determines the timbre. Therefore, the linear amplitude spectrogram is more suitable for periodic feature extraction than the reconstructed acoustic features.

[0067] Taking the reconstruction of acoustic features as Mel spectrogram as an example, generally speaking, speech needs to be converted into a frequency domain signal through a short-time Fourier transform, so that a linear amplitude spectrogram can be obtained. Since the human ear's perception of frequency is nonlinear, it is necessary to convert the frequency axis of the linear scale in the linear amplitude spectrogram into a Mel Frequency scale. The Mel scale simulates the way the human ear perceives different frequencies, that is, the human ear is more sensitive to low-frequency components and relatively less sensitive to high-frequency components. The spectrum is weighted and summed using a Mel filter bank to calculate the energy or amplitude of each Mel frequency band to form a Mel spectrogram. In actual operation, the operation of converting from a linear amplitude spectrogram to a Mel spectrogram is usually implemented by multiplying a Mel matrix by a linear amplitude spectrogram. Therefore, after obtaining the reconstructed acoustic features in the form of a Mel spectrogram, the Mel spectrum can be multiplied by the pseudo-inverse matrix of the Mel matrix to achieve an inverse transformation and obtain a rough linear amplitude spectrogram, that is, the linear amplitude spectrogram obtained by conversion in the embodiment of the present invention.

[0068] For example, Figure 2 It is the acoustic characteristic spectrum provided by the present invention. Figure 2 In the figure, (a) is the original linear amplitude spectrogram, (b) is the Mel spectrum converted from the linear amplitude spectrogram, and (c) is the rough linear amplitude spectrogram obtained by inverse transforming the Mel spectrum. Comparing (a) and (c), it can be seen that the rough linear amplitude spectrogram has some loss of detail information compared to the original linear amplitude spectrogram, but the periodic features, especially the resonance peak information that determines the timbre, are still retained. Therefore, the linear amplitude spectrogram converted from the Mel spectrum can be used to extract periodic features.

[0069] After obtaining the linear amplitude spectrum, the reconstructed periodic features under multiple periodic components can be extracted for the linear amplitude spectrum. For example, for a frame of linear amplitude spectrum, the reconstructed periodic features can be constructed according to 2, 3, 5, 7, 11 and other periodic components. Figure 3 is a schematic diagram of extracting periodic features provided by the present invention, Figure 3 In the figure, (a) is a frame of linear amplitude spectrum, (b), (c), and (d) are feature maps of periodic features with periods of 2, 3, and 5 extracted based on (a), respectively.

[0070] Based on any of the above embodiments, in step 130, the adversarial training of the generative model and the discriminative model based on the authenticity discrimination result includes: Based on the authenticity discrimination result of the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature, performing adversarial training on the generation model and the discrimination model; The authenticity discrimination result of the real periodic feature is output by the discrimination model based on the real periodic feature, and the real periodic feature is extracted from the real acoustic feature.

[0071] Specifically, in the adversarial training combining the generative model and the discriminative model, in addition to applying the true and false discrimination results for the reconstructed periodic features, the true and false discrimination results for the real periodic features can also be applied.

[0072] That is, the acoustic features of the real speech can be obtained, which are recorded as real acoustic features here. On this basis, the periodic features can be extracted from the real acoustic features, which are recorded as real periodic features here. It can be understood that the method of extracting periodic features from the real acoustic features is the same as the method of extracting periodic features from the reconstructed acoustic features, which will not be repeated here.

[0073] After obtaining the true periodic features, the true periodic features can also be input into the discrimination model, and the discrimination model performs authenticity discrimination on the input true periodic features, thereby obtaining the authenticity discrimination result of the true periodic features output by the discrimination model.

[0074] On this basis, the authenticity discrimination results of the real periodic features and the authenticity discrimination results of the reconstructed periodic features can be combined to conduct adversarial training on the generation model and the discriminant model. In this process, the authenticity discrimination results of the reconstructed periodic features can reflect the difference between the reconstructed periodic features of the reconstructed acoustic features output by the generation model and the real periodic features, thereby determining the generation loss for the generation model to guide the generation model to update and iterate in the direction of outputting the reconstructed acoustic features that are closer to the real periodic features; and the authenticity discrimination results of the real periodic features and the authenticity discrimination results of the reconstructed periodic features can also reflect the accuracy of the discriminant model in distinguishing the authenticity of the input periodic features, thereby determining the discrimination loss for the discriminant model to guide the discriminant model to update and iterate in the direction of outputting the authenticity discrimination results that are closer to the real situation.

[0075] Based on any of the above embodiments, in step 130, the authenticity discrimination result based on the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature is used to perform adversarial training on the generation model and the discrimination model, including: Determine a discrimination loss based on the authenticity discrimination result of the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature, and perform parameter iteration on the discrimination model based on the discrimination loss; Based on the authenticity discrimination result of the reconstructed periodic feature, a generation loss is determined, and parameters of the generation model are iterated based on the generation loss.

[0076] Specifically, for the discriminant model, the authenticity discrimination result of the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature can both characterize the accuracy of the discriminant model in distinguishing the authenticity of the input periodic feature. Therefore, the discrimination loss for the discriminant model can be determined by combining the authenticity discrimination result of the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature.

[0077] Here, the discriminant loss can be the difference between the authenticity discrimination result of the real periodic feature and the real label, and the difference between the authenticity discrimination result of the reconstructed periodic feature and the synthetic label, the sum of the two or the result of weighted summation. After obtaining the discriminant loss, the discriminant model can be parameterized based on the discriminant loss, thereby gradually improving the discriminant model's ability to discriminate the authenticity of the input periodic feature.

[0078] For the generative model, the authenticity discrimination result of the reconstructed periodic feature can characterize the fidelity of the reconstructed acoustic feature output by the generative model in the timbre performance. Therefore, the generative loss for the generative model can be determined based on the authenticity discrimination result of the reconstructed periodic feature.

[0079] Here, the generation loss can be the difference between the authenticity discrimination result of the reconstructed periodic feature and the real label, that is, the higher the probability that the reconstructed periodic feature is judged by the discrimination model as coming from the real audio, the smaller the generation loss, and the lower the probability that the reconstructed periodic feature is judged by the discrimination model as coming from the real audio, the larger the generation loss. After obtaining the generation loss, the generation model can be iterated based on the generation loss, thereby gradually improving the ability of the generation model to generate more natural and realistic reconstructed acoustic features.

[0080] Based on any of the above embodiments, Figure 4 This is the second flow chart of the model training method provided by the present invention. Figure 4 As shown, the method may include the following steps: Step 410: Obtain reconstructed acoustic features and real acoustic features.

[0081] Here, both the reconstructed acoustic features and the real acoustic features can be mel-spectrograms. The real acoustic features can be obtained first, and then the text features corresponding to the real acoustic features are input into the generative model to obtain the reconstructed acoustic features output by the generative model. The generative model here is the acoustic model to be trained.

[0082] Step 420: Convert the acoustic features into periodic features.

[0083] Periodic feature conversion is performed for the reconstructed acoustic features and the real acoustic features respectively. For both the reconstructed acoustic features and the real acoustic features, the Mel-spectrogram needs to be inversely transformed into a linear amplitude spectrogram, and then the linear amplitude spectrogram needs to be converted into periodic features under different periodic components.

[0084] Step 430, using the discrimination models corresponding to different periodic components to discriminate the authenticity of the periodic features.

[0085] The discriminant models corresponding to different periodic components can be pre-set, so that the periodic features under different periodic components can be input into the corresponding discriminant models respectively, and the discriminant models will distinguish the authenticity of the input periodic features and output the authenticity distinction results.

[0086] Step 440, calculate the adversarial loss and perform adversarial training.

[0087] After obtaining the authenticity discrimination result, the adversarial loss can be calculated based on the authenticity discrimination result. The adversarial loss here specifically includes generation loss and discrimination loss, where the generation loss is obtained based on the authenticity discrimination result of the reconstructed periodic feature, and the discrimination loss is obtained based on the authenticity discrimination result of the real periodic feature and the reconstructed periodic feature. After obtaining the adversarial loss, adversarial training can be performed on the generation model and the discrimination model based on it.

[0088] Based on any of the above embodiments, Figure 5 Schematic diagram of the speech synthesis method provided by the present invention. Figure 5 As shown, the method includes: Step 510, inputting the text into a speech synthesis model to obtain synthesized speech output by the speech synthesis model; The speech synthesis model includes the acoustic model trained by the above-mentioned model training method.

[0089] Specifically, the trained acoustic model can be obtained based on the model training method provided in the above embodiments. Based on this, a speech synthesis model including the acoustic model can be constructed. For example, within speech synthesis, the input text can be first encoded into a text vector through the front-end text and prosody analysis model, and then the acoustic features are reconstructed for the text vector through the back-end acoustic model, and then the acoustic features are restored into a speech waveform through the vocoder, thereby outputting the synthesized speech.

[0090] Thus, after obtaining the speech synthesis model, the text can be input into the speech synthesis model, and the speech synthesis model outputs a synthesized speech with content consistent with the text. Since the speech synthesis model includes the acoustic model obtained based on the above embodiment, the naturalness and fidelity of the timbre in the synthesized speech are guaranteed.

[0091] In the method provided in the embodiment of the present invention, speech synthesis is performed based on a speech synthesis model including an acoustic model with timbre reconstruction as an optimization goal, which can ensure the naturalness and fidelity of the timbre of the synthesized speech.

[0092] The model training device provided by the present invention is described below. The model training device described below and the model training method described above can be referenced to each other.

[0093] Figure 6 Schematic diagram of the structure of the model training device provided by the present invention. Figure 6 As shown, the device comprises: A feature extraction unit 610, configured to obtain the reconstructed acoustic features output by the generation model, and extract the reconstructed periodic features of the reconstructed acoustic features; The authenticity discrimination unit 620 is used to input the reconstructed periodic feature into a discrimination model to obtain an authenticity discrimination result of the reconstructed periodic feature output by the discrimination model; The adversarial training unit 630 is used to perform adversarial training on the generation model and the discrimination model based on the authenticity discrimination result, and use the trained generation model as the acoustic model.

[0094] In the device provided by the embodiment of the present invention, the reconstructed periodic features of the reconstructed acoustic features output by the generative model are input into the discriminative model for authenticity discrimination, thereby performing adversarial training on the generative model and the discriminative model. In this process, the periodic features that can reflect the timbre are explicitly taken as optimization targets, so that when the generative model obtained through adversarial training is applied to speech synthesis, the timbre of the synthesized speech is more natural and realistic.

[0095] Moreover, the above-mentioned training device involves two types of models, namely, a generative model and a discriminative model. It has a simple structure and is easy to transplant. It can be applied to any acoustic model targeting a Mel-spectrogram or a linear amplitude spectrogram to enhance the timbre reconstruction capability of such acoustic models.

[0096] Based on any of the above embodiments, the feature extraction unit is specifically used for: Extracting reconstructed periodic features of the reconstructed acoustic features under multiple periodic components; The authenticity determination unit is specifically used for: The reconstructed periodic features under each periodic component are respectively input into the discriminant models corresponding to each periodic component, and the true and false discrimination results output by the discriminant models corresponding to each periodic component are obtained.

[0097] Based on any of the above embodiments, the multiple periodic components are multiple periodic components based on prime numbers.

[0098] Based on any of the above embodiments, the feature extraction unit is specifically used for: converting the reconstructed acoustic features into a linear magnitude spectrogram; Reconstructed periodic features under multiple periodic components are extracted from the linear amplitude spectrum.

[0099] Based on any of the above embodiments, the adversarial training unit is specifically used for: Based on the authenticity discrimination result of the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature, performing adversarial training on the generation model and the discrimination model; The authenticity discrimination result of the real periodic feature is output by the discrimination model based on the real periodic feature, and the real periodic feature is extracted from the real acoustic feature.

[0100] Based on any of the above embodiments, the adversarial training unit is specifically used for: Determine a discrimination loss based on the authenticity discrimination result of the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature, and perform parameter iteration on the discrimination model based on the discrimination loss; Based on the authenticity discrimination result of the reconstructed periodic feature, a generation loss is determined, and parameters of the generation model are iterated based on the generation loss.

[0101] The speech synthesis device provided by the present invention is described below. The speech synthesis device described below and the speech synthesis method described above can be referred to each other.

[0102] A speech synthesis device, comprising: A speech synthesis unit, used to input text into a speech synthesis model to obtain synthesized speech output by the speech synthesis model; The speech synthesis model includes the acoustic model trained by the model training method provided in the above embodiment.

[0103] In the device provided in the embodiment of the present invention, speech synthesis is performed based on a speech synthesis model including an acoustic model with timbre reconstruction as an optimization goal, which can ensure the naturalness and fidelity of the timbre of the synthesized speech.

[0104] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the model training method, which includes: Acquiring reconstructed acoustic features output by the generation model, and extracting reconstructed periodic features of the reconstructed acoustic features; Inputting the reconstructed periodic feature into a discrimination model to obtain a true or false discrimination result of the reconstructed periodic feature output by the discrimination model; Based on the authenticity discrimination result, adversarial training is performed on the generative model and the discriminative model, and the generative model obtained through the training is used as the acoustic model.

[0105] Alternatively, the processor 710 may call the logic instructions in the memory 730 to execute the speech synthesis method, which includes: Inputting the text into a speech synthesis model to obtain synthesized speech output by the speech synthesis model; The speech synthesis model includes an acoustic model trained by a model training method.

[0106] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0107] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the model training method provided by the above methods, the method includes: Acquiring reconstructed acoustic features output by the generation model, and extracting reconstructed periodic features of the reconstructed acoustic features; Inputting the reconstructed periodic feature into a discrimination model to obtain a true or false discrimination result of the reconstructed periodic feature output by the discrimination model; Based on the authenticity discrimination result, adversarial training is performed on the generative model and the discriminative model, and the generative model obtained through the training is used as the acoustic model.

[0108] Alternatively, a computer can execute the speech synthesis method provided by the above methods, the method comprising: Inputting the text into a speech synthesis model to obtain synthesized speech output by the speech synthesis model; The speech synthesis model includes an acoustic model trained by a model training method.

[0109] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the model training method provided by the above methods, the method comprising: Acquiring reconstructed acoustic features output by the generation model, and extracting reconstructed periodic features of the reconstructed acoustic features; Inputting the reconstructed periodic feature into a discrimination model to obtain a true or false discrimination result of the reconstructed periodic feature output by the discrimination model; Based on the authenticity discrimination result, adversarial training is performed on the generative model and the discriminative model, and the generative model obtained through the training is used as the acoustic model.

[0110] Alternatively, when the computer program is executed by a processor, the computer program is implemented to perform the speech synthesis method provided by the above methods, the method comprising: Inputting the text into a speech synthesis model to obtain synthesized speech output by the speech synthesis model; The speech synthesis model includes an acoustic model trained by a model training method.

[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method, characterized in that: include: Acquiring reconstructed acoustic features output by the generation model, and extracting reconstructed periodic features of the reconstructed acoustic features; Inputting the reconstructed periodic feature into a discrimination model to obtain a true or false discrimination result of the reconstructed periodic feature output by the discrimination model; Based on the authenticity discrimination result, adversarial training is performed on the generative model and the discriminative model, and the generative model obtained through the training is used as the acoustic model.

2. The model training method according to claim 1, characterized in that: The extracting the reconstructed periodic feature of the reconstructed acoustic feature comprises: Extracting reconstructed periodic features of the reconstructed acoustic features under multiple periodic components; The step of inputting the reconstructed periodic feature into a discrimination model to obtain a true or false discrimination result of the reconstructed periodic feature output by the discrimination model comprises: The reconstructed periodic features under each periodic component are respectively input into the discriminant models corresponding to each periodic component, and the true and false discrimination results output by the discriminant models corresponding to each periodic component are obtained.

3. The model training method according to claim 2, characterized in that: The multiple periodic components are multiple periodic components based on prime numbers.

4. The model training method according to claim 2, characterized in that: The extracting of the reconstructed periodic features of the reconstructed acoustic features under multiple periodic components comprises: converting the reconstructed acoustic features into a linear magnitude spectrogram; Reconstructed periodic features under multiple periodic components are extracted from the linear amplitude spectrum.

5. The model training method according to any one of claims 1 to 4, characterized in that: The performing adversarial training on the generation model and the discrimination model based on the authenticity discrimination result includes: Based on the authenticity discrimination result of the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature, performing adversarial training on the generation model and the discrimination model; The authenticity discrimination result of the real periodic feature is output by the discrimination model based on the real periodic feature, and the real periodic feature is extracted from the real acoustic feature.

6. The model training method according to claim 5, characterized in that: The method further comprises: performing adversarial training on the generation model and the discrimination model based on the authenticity discrimination result of the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature, including: Determine a discrimination loss based on the authenticity discrimination result of the real periodic feature and the authenticity discrimination result of the reconstructed periodic feature, and perform parameter iteration on the discrimination model based on the discrimination loss; Based on the authenticity discrimination result of the reconstructed periodic feature, a generation loss is determined, and parameters of the generation model are iterated based on the generation loss.

7. A speech synthesis method, characterized in that: include: Inputting the text into a speech synthesis model to obtain synthesized speech output by the speech synthesis model; The speech synthesis model includes an acoustic model trained by the model training method according to any one of claims 1 to 6.

8. A model training device, characterized in that: include: A feature extraction unit, used to obtain the reconstructed acoustic features output by the generation model, and extract the reconstructed periodic features of the reconstructed acoustic features; A true-false discrimination unit, used for inputting the reconstructed periodic feature into a discrimination model to obtain a true-false discrimination result of the reconstructed periodic feature output by the discrimination model; The adversarial training unit is used to perform adversarial training on the generative model and the discriminative model based on the authenticity discrimination result, and use the generative model obtained by the training as the acoustic model.

9. A speech synthesis device, characterized in that: include: A speech synthesis unit, used to input text into a speech synthesis model to obtain synthesized speech output by the speech synthesis model; The speech synthesis model includes an acoustic model trained by the model training method according to any one of claims 1 to 6.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, it implements the model training method as described in any one of claims 1 to 6, or the speech synthesis method as described in claim 7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 6, or the speech synthesis method as described in claim 7.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 6, or the speech synthesis method as described in claim 7.

Images