Speech synthesis method, device, terminal equipment and storage medium

Abstract

The present invention discloses a speech synthesis method, apparatus, device, and storage medium. The method comprises obtaining speech and text data to be processed and inputting them into a target speech synthesis model to obtain target synthesized speech data corresponding to acoustic features and content. The target speech synthesis model is generated by obtaining and adjusting the model parameters of the student speech synthesis model based on the relative entropy between the output distributions of a teacher speech synthesis model and a student speech synthesis model during model training. When the relative entropy reaches a first preset threshold, the corresponding student speech synthesis model is used as the target speech synthesis model. By implementing the present invention, the expressive power and generation quality of the target speech synthesis model can be ensured while also meeting the real-time requirements of users.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis, and in particular to a speech synthesis method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI), speech synthesis technology has been widely applied in various fields. Existing speech synthesis models, such as the CosyVoice model, support multiple languages, voice sentiment analysis, event recognition, and cross-language synthesis. To ensure the model's expressiveness and generation quality, the model requires more decoding layers to process and optimize the data. This means that during the inference process, each decoding layer takes up a significant amount of computational time. This can hinder the timely response to user commands in certain application scenarios with high real-time requirements, leading to choppy conversations or feedback between users, and thus impacting the user experience. Therefore, how to meet user real-time requirements while ensuring the model's expressiveness and generation quality in specific scenarios with high real-time requirements is a pressing technical challenge. Summary of the Invention

[0003] The present invention provides a speech synthesis method, apparatus, device and storage medium, which can ensure the expressiveness and generation quality of the model while meeting the real-time needs of users in certain specific scenarios with high real-time requirements.

[0004] The present invention provides a speech synthesis method, comprising: obtaining speech data to be processed and text data to be processed corresponding to a target language;

[0005] Inputting the speech data to be processed and the text data to be processed into a target speech synthesis model, so that the target speech synthesis model generates target synthesized speech data having the same acoustic features as the speech data to be processed and the same speech content as the text data to be processed based on the speech data to be processed and the text data to be processed;

[0006] The generation of the target speech synthesis model includes:

[0007] Obtaining a preset teacher speech synthesis model, a preset student speech synthesis model, a speech dataset to be trained corresponding to the target language, and a text dataset to be trained; wherein the number of decoding layers in the teacher speech synthesis model is greater than the number of decoding layers in the student speech synthesis model;

[0008] Repeat the following steps until the relative entropy between the output distribution of the student model and the output distribution of the teacher model reaches a first preset threshold, and use the student speech synthesis model corresponding to the time when the relative entropy reaches the first preset threshold as the target speech synthesis model;

[0009] Randomly select a speech data set to be trained and a text data set to be trained from the current speech data set to be trained and the text data set to be trained respectively;

[0010] Inputting the speech data to be trained and the text data to be trained into the teacher speech synthesis model and the student speech synthesis model, respectively, to obtain the teacher model output distribution of the teacher speech synthesis model and the student model output distribution of the student speech synthesis model;

[0011] Obtaining the relative entropy between the teacher model output distribution and the student model output distribution;

[0012] When it is determined that the relative entropy does not reach the first preset threshold, the model parameters of the student speech synthesis model are adjusted according to the relative entropy, and the current speech data to be trained and the text data to be trained are respectively removed from the speech data set to be trained and the text data set to be trained.

[0013] Furthermore, the target speech synthesis model includes: an input layer, a decoding layer set, a stream matching layer, and an output layer;

[0014] The input layer is used to collect data features of the voice data to be processed and the text data to be processed;

[0015] The decoding layer set is used to decode the data features of the collected voice data to be processed and the collected text data to be processed, so as to fuse the data features of the voice data to be processed and the text data to be processed to generate corresponding target voice features;

[0016] The stream matching layer is used to generate a mel-spectrogram corresponding to the target speech feature according to the target speech feature;

[0017] The output layer is used to convert the mel-spectrogram into target synthesized speech data having a sound wave signal.

[0018] Furthermore, the input layer includes: a text encoder and a speech segmenter;

[0019] The text encoder is used to extract text semantic features of the text data to be processed;

[0020] The speech segmenter is used to extract acoustic features of the speech data to be processed; wherein the acoustic features include: pitch, timbre, volume and speaking speed.

[0021] Furthermore, the number of decoding layers of the student speech synthesis model is 6.

[0022] Furthermore, the relative entropy includes: inverse KL divergence.

[0023] Based on the above method embodiment, the present invention provides a corresponding device embodiment;

[0024] The present invention provides a speech synthesis device, comprising: a data acquisition module and a speech synthesis module;

[0025] The data acquisition module is used to acquire the speech data to be processed and the text data to be processed corresponding to the target language;

[0026] The speech synthesis module is used to input the speech data to be processed and the text data to be processed into a target speech synthesis model, so that the target speech synthesis model generates target synthesized speech data that is consistent with the acoustic features of the speech data to be processed and has speech content consistent with the content of the text data to be processed based on the speech data to be processed and the text data to be processed;

[0027] The generation of the target speech synthesis model includes:

[0028] Obtaining a preset teacher speech synthesis model, a preset student speech synthesis model, a speech dataset to be trained corresponding to the target language, and a text dataset to be trained; wherein the number of decoding layers in the teacher speech synthesis model is greater than the number of decoding layers in the student speech synthesis model;

[0029] Repeat the following steps until the relative entropy between the output distribution of the student model and the output distribution of the teacher model reaches a first preset threshold, and use the student speech synthesis model corresponding to the time when the relative entropy reaches the first preset threshold as the target speech synthesis model;

[0030] Randomly select a speech data set to be trained and a text data set to be trained from the current speech data set to be trained and the text data set to be trained respectively;

[0031] Inputting the speech data to be trained and the text data to be trained into the teacher speech synthesis model and the student speech synthesis model, respectively, to obtain the teacher model output distribution of the teacher speech synthesis model and the student model output distribution of the student speech synthesis model;

[0032] Obtaining the relative entropy between the teacher model output distribution and the student model output distribution;

[0033] When it is determined that the relative entropy does not reach the first preset threshold, the model parameters of the student speech synthesis model are adjusted according to the relative entropy, and the current speech data to be trained and the text data to be trained are respectively removed from the speech data set to be trained and the text data set to be trained.

[0034] Furthermore, the target speech synthesis model includes: an input layer, a decoding layer set, a stream matching layer, and an output layer;

[0035] The input layer is used to collect data features of the voice data to be processed and the text data to be processed;

[0036] The decoding layer set is used to decode the data features of the collected voice data to be processed and the collected text data to be processed, so as to fuse the data features of the voice data to be processed and the text data to be processed to generate corresponding target voice features;

[0037] The stream matching layer is used to generate a mel-spectrogram corresponding to the target speech feature according to the target speech feature;

[0038] The output layer is used to convert the mel-spectrogram into target synthesized speech data having a sound wave signal.

[0039] Furthermore, the input layer includes: a text encoder and a speech segmenter;

[0040] The text encoder is used to extract text semantic features of the text data to be processed;

[0041] The speech segmenter is used to extract acoustic features of the speech data to be processed; wherein the acoustic features include: pitch, timbre, volume and speaking speed.

[0042] Based on the above method embodiment, the present invention provides a corresponding device embodiment;

[0043] The present invention provides a device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements any one of the speech synthesis methods of the present invention when executing the computer program.

[0044] Based on the above method embodiment, the present invention provides a storage medium embodiment;

[0045] The present invention provides a storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute any one of the speech synthesis methods described in the present invention.

[0046] The embodiments of the present invention have the following beneficial effects:

[0047] The present invention provides a speech synthesis method, apparatus, device and storage medium; the method,

[0048] By inputting the speech data to be trained and the text data to be processed corresponding to the target language into the teacher speech synthesis model and the student speech synthesis model respectively, the corresponding teacher model output distribution and student model output distribution are obtained, and then the model parameters of the student speech synthesis model are adjusted according to the relative entropy between the teacher model output distribution and the student model output distribution. When the relative entropy reaches a first preset threshold, the student model output distribution can be made to be similar to the teacher model output distribution. In this way, the student speech synthesis model has achieved almost the same effect as the teacher speech synthesis model in processing the target language data, and then the student speech synthesis model after adjusting the model parameters can be used as the target speech synthesis model. After inputting the speech data to be processed and the text data to be processed into the target speech synthesis model, the target synthesized speech data can be generated by the target speech synthesis model. By implementing the present invention, the target speech synthesis model can achieve the same effect as the existing teacher speech synthesis model with a more complex decoding layer structure when processing speech data corresponding to the target language. In addition, since the target speech synthesis model is only for the target language in a specific scenario, fewer decoding layers are required when setting up the student speech synthesis model, making the model's data processing more targeted and efficient, thereby improving the model's reasoning speed. In summary, the target speech synthesis model of the present invention not only retains the teacher speech synthesis model's ability to process target language data, but also speeds up the reasoning speed of target language data. It can ensure the model's expressiveness and generation quality in certain specific scenarios with high real-time requirements while also meeting the user's real-time needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of a speech synthesis method provided by one embodiment of the present invention;

[0050] Figure 2 is a schematic structural diagram of a target speech synthesis model provided by one embodiment of the present invention;

[0051] Figure 3 1 is a schematic diagram of a method for simplifying the decoding layer of a student speech synthesis model provided by one embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of a distillation learning process provided by one embodiment of the present invention;

[0053] Figure 5 It is a structural diagram of a speech synthesis device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0055] like Figure 1 As shown, an embodiment provides a speech synthesis method, including:

[0056] Step S101: Acquire speech data and text data to be processed corresponding to the target language;

[0057] Step S102: Inputting the speech data to be processed and the text data to be processed into a target speech synthesis model, so that the target speech synthesis model generates target synthesized speech data having the same acoustic features as the speech data to be processed and the same speech content as the text data to be processed based on the speech data to be processed and the text data to be processed;

[0058] The generation of the target speech synthesis model includes:

[0059] Obtaining a preset teacher speech synthesis model, a preset student speech synthesis model, a speech dataset to be trained corresponding to the target language, and a text dataset to be trained; wherein the number of decoding layers in the teacher speech synthesis model is greater than the number of decoding layers in the student speech synthesis model;

[0060] Repeat the following steps until the relative entropy between the output distribution of the student model and the output distribution of the teacher model reaches a first preset threshold, and use the student speech synthesis model corresponding to the time when the relative entropy reaches the first preset threshold as the target speech synthesis model;

[0061] Randomly select a speech data set to be trained and a text data set to be trained from the current speech data set to be trained and the text data set to be trained respectively;

[0062] Inputting the speech data to be trained and the text data to be trained into the teacher speech synthesis model and the student speech synthesis model, respectively, to obtain the teacher model output distribution of the teacher speech synthesis model and the student model output distribution of the student speech synthesis model;

[0063] Obtaining the relative entropy between the teacher model output distribution and the student model output distribution;

[0064] When it is determined that the relative entropy does not reach the first preset threshold, the model parameters of the student speech synthesis model are adjusted according to the relative entropy, and the current speech data to be trained and the text data to be trained are respectively removed from the speech data set to be trained and the text data set to be trained.

[0065] For step S101, in a preferred embodiment, it is first necessary to obtain the corresponding speech data to be processed and text data to be processed in the target language; the target language can be any language with a separate spoken and written language; the speech data to be processed refers to speech data with a specific speaker style; the text data to be processed refers to the text data corresponding to the target language.

[0066] Taking Chinese as the target language as an example, the speech data to be processed refers to speech data with a specific speaker style and pronounced in Chinese; then the text data to be processed refers to Chinese text data.

[0067] For step S102, in a preferred embodiment, the speech data to be processed and the text data to be processed are input into a trained target speech synthesis model. The target speech synthesis model can generate a target synthesized speech data that is consistent with the acoustic features of the speech data to be processed and the speech content is consistent with the content of the text data to be processed based on the acoustic features of the speech data to be processed and the content of the text data to be processed.

[0068] Taking Chinese as the target language as an example, after the target speech synthesis model obtains user A's to-be-processed voice data "go eat" and the to-be-processed text data "go travel", the target speech synthesis model can extract the corresponding acoustic features from user A's to-be-processed voice data "go eat", including pitch, timbre, volume, and speaking speed, etc., and then combine the extracted acoustic features with the to-be-processed text data "go travel" according to the content of the to-be-processed text data "go travel" to generate a target speech synthesis data that reads "go travel" with user A's acoustic features.

[0069] For the generation of the target speech synthesis model, first obtain a preset teacher speech synthesis model, which can be a cosyvoice model or any model in the existing technology that can achieve the same function; then obtain a preset student speech synthesis model. Compared with the structure of the teacher speech synthesis model, the structure of the student speech synthesis model has fewer decoding layers because the student speech synthesis model is only targeted at specific application scenarios or target languages ​​(such as Mandarin speech synthesis tasks). Too many decoding layers will only make the parameters of the student speech synthesis model redundant and increase delays; finally, obtain the speech dataset to be trained and the text dataset to be trained for model training. Both datasets belong to the same target language type (such as Chinese).

[0070] Then enter into a cyclic iterative process, which uses the teacher speech synthesis model as a "guide" to continuously train and optimize the student speech synthesis model until its performance in processing the speech data set to be trained and the text data set to be trained is close to the teacher speech synthesis model. The specific method is to first randomly select a speech data to be trained and a text data to be trained from the current speech data set to be trained and the text data set to be trained, respectively, and then input these two data into the teacher speech synthesis model and the student speech synthesis model respectively. This step is equivalent to the teacher speech synthesis model and the student speech synthesis model inputting the same data, and then use this to obtain the teacher model output distribution of the teacher speech synthesis model and the student model output distribution of the student speech synthesis model respectively. In order to check whether the teacher model output distribution and the student model output distribution are approximately equal, the method of the present invention is to calculate the output distribution of the teacher model and the output distribution of the student model. The relative entropy of the student model is determined by presetting a first preset threshold, and the calculated relative entropy is compared with the first preset threshold. When it is determined that the relative entropy has not reached the first preset threshold, it means that the output distribution of the teacher model and the output distribution of the student model have not yet reached the required approximately equal condition, and it also means that under certain conditions, the student speech synthesis model has not yet achieved a capability similar to that of the teacher model. At this time, the model parameters of the student speech synthesis model need to be adjusted, and then the current speech data to be trained and the text data to be trained that have been used for training are removed from the speech data set to be trained and the text data to be trained, respectively, to prevent invalid repeated training.

[0071] It should be noted that the first preset threshold can be set according to actual needs, and the specific calculation method of relative entropy also belongs to the existing technology, and the present invention does not limit it specifically; the method of adjusting the model parameters of the student speech synthesis model of the present invention can be adjusting the weights and biases in the decoding layer, etc. The specific adjustment method can also belong to any similar adjustment method in the existing technology, and the present invention does not make specific limitations.

[0072] In a preferred embodiment, as shown in 2, the target speech synthesis model mainly consists of several parts, including: an input layer, a decoding layer set, a stream matching layer, and an output layer;

[0073] The input layer refers to the portion of the model that receives the speech and text data to be processed and collects relevant data features. The decoding layer set, comprising several decoding layers, primarily receives data features from the input layer and begins decoding them, effectively fusing the relevant speech and text features to generate a target speech feature that integrates both speech and text information. The stream matching layer primarily converts the target speech features into a format that is easy to understand and process, namely a mel-spectrogram. A mel-spectrogram is a spectral representation based on the mel-scale that better reflects the characteristics of the human auditory system. By generating a mel-spectrogram, the stream matching layer provides a more intuitive and user-friendly data format for subsequent speech synthesis. The output layer is primarily responsible for converting the mel-spectrogram into target synthesized speech data containing acoustic signals. This method can utilize inverse transformation and signal processing techniques to convert the mel-spectrogram into actual acoustic signals. These acoustic signals are then further processed and optimized to ultimately generate high-quality, realistic target synthesized speech data. The specific method for generating target synthesized speech data can be any equivalent method in the prior art and is not specifically limited in this disclosure.

[0074] In a preferred embodiment, the input layer of the target speech synthesis model consists of two parts, including: a text encoder and a speech segmenter;

[0075] The text encoder is primarily responsible for parsing and processing text data. Its primary function is to extract key semantic features from the input text data. These features can capture and express the meaning, context, and relationships between words in the text, providing the necessary semantic information for the subsequent speech synthesis process.

[0076] The speech segmenter is primarily responsible for analyzing and processing speech data. Its main function is to extract acoustic features from the speech signal to be processed. These features determine the naturalness and fluency of the target synthesized speech data ultimately generated. Specifically, acoustic features encompass multiple aspects, including pitch, timbre, volume, and speaking rate. Pitch refers to the high and low pitch of a voice, and it plays a decisive role in expressing different emotions, intonation, and the changes in tone within a language. Timbre refers to the quality or characteristics of a voice, such as sweetness or depth, which primarily influences the user's perception of the sound. Volume refers to the size or strength of a sound, reflecting its loudness and significantly impacting the clarity and audibility of speech. Speaking rate refers to the speed of speech, that is, the number of syllables uttered per unit time, which influences the rhythm and fluency of speech.

[0077] In a preferred embodiment, the number of decoding layers of the student speech synthesis model in the present invention is 6 layers, that is, the decoding layer set in the target speech synthesis model has 6 decoding layers.

[0078] Alternatively, the present invention uses the existing CosyVoice model as the teacher speech synthesis model. While the CosyVoice model can generate speech with a real-time factor (RTF) less than 1, it still takes 3 to 4 seconds to generate short audio clips (e.g., 5 seconds). This can negatively impact user experience in certain applications requiring high real-time performance. Furthermore, the CosyVoice model supports multiple languages, speech sentiment analysis, event recognition, and cross-language synthesis, which can be redundant in certain scenarios.

[0079] To address this issue, this paper employs a distillation method for speech synthesis models. This method aims to simplify the model structure and improve inference speed while retaining the core functionality of the original model, thereby meeting the higher real-time requirements in specific scenarios. This paper primarily targets scenarios requiring only Chinese speech cloning. Through distillation technology, this method optimizes the CosyVoice model, enabling faster inference while maintaining the original speech synthesis quality.

[0080] Therefore, in the design of the student speech synthesis model structure, the present invention performs refined compression and simplification on the basis of the teacher speech synthesis model structure, performs deep optimization through the decoding layer of the model, reduces the original 14-layer decoding layer set of the cosyvoice model to 6 layers, and removes the middle 8 layers. The reason for retaining the front and back 3 decoding layers is that these 6 layers play a key role in the encoding and decoding process. The front 3 retained decoding layers are crucial for receiving and processing signals from the text encoder and speech segmenter, and they ensure the accurate transmission of information. The last 3 decoding layers play a key role in converting the feature vector into a standard audio signal format. Such adjustments not only ensure the effectiveness of the model, but also improve its operating efficiency.

[0081] The reason for reducing the decoding layer of the CosyVoice model is that the CosyVoice model performs well in the field of multilingual speech synthesis and has versatility such as sentiment analysis and voice event recognition. However, when the CosyVoice model focuses on a single Chinese speech synthesis application, the decoding layer accounts for 90% of the model parameters and consumes 80% of the time during inference. Therefore, reducing the decoding layer is crucial.

[0082] Optionally, compared with the teacher speech synthesis model, the structure of the simplified student speech synthesis model is as shown in 3. When simplifying the decoding layer structure, it is necessary to retain the front and back decoding layers and simplify the middle part.

[0083] In the figure, the text encoder extracts semantic features from the text, while the speech tokenizer focuses on capturing the timbre (acoustic) characteristics of the speech. Based on this, this process fuses the extracted text semantic features with the timbre features to form a new token sequence. This sequence is then input into multiple decoding layers for decoding, followed by the text encoder to generate the corresponding speech token sequence. A stream matching model is then used to convert the resulting speech token sequence into a mel-spectrogram. Finally, sound wave synthesis is performed to achieve speech cloning.

[0084] Optionally, during the distillation learning process, the parameters of all modules in the student speech synthesis model, except for the decoding layer, remain fixed. This strategy was chosen based on two considerations: first, the number of parameters in these modules is relatively small, so compression is unnecessary; second, the parameters of these modules have been trained on a large amount of data and have reached a high quality standard. Allowing these parameters to adjust during the distillation process could result in a decrease in model performance. Therefore, fixing the parameters of these modules not only helps reduce the number of parameters during training, thereby saving computing resources, but also effectively shortens training time.

[0085] Implementing the distillation strategy of this invention to optimize the student speech synthesis model inevitably reduces the model's generalization ability for speech synthesis in non-target languages. However, while maintaining the target speech synthesis quality, the size of the student speech synthesis model was successfully reduced to 40% of the original size of the teacher speech synthesis model. At the same time, the speech synthesis processing speed was doubled, significantly improving the computational efficiency of the student speech synthesis model.

[0086] In a preferred embodiment, the present invention calculates the relative entropy between the teacher model output distribution and the student model output distribution, which means calculating the inverse KL divergence between the teacher model output distribution and the student model output distribution. The specific calculation method belongs to the prior art and is not specifically limited by the present invention.

[0087] Alternatively, as Figure 4 As shown, the present invention adopts inverse KL divergence to achieve model distillation by minimizing the KL divergence between the output distribution q of the student speech synthesis model and the output distribution p of the teacher speech synthesis model.

[0088] In the knowledge distillation process, the forward KL divergence (denoted as KL(p||q)) measures the difference between the teacher model's output distribution p and the student model's output distribution q, ensuring that the student model's output distribution q fully covers the important feature regions of the teacher model's output distribution p. However, in the actual application of large-scale language models, due to the vast output space and complex distribution patterns, the forward KL divergence may cause the student speech synthesis model to overemphasize samples with extremely low probability of occurrence in real applications, thereby affecting the overall performance of the target speech synthesis model.

[0089] In contrast, the inverse KL divergence (denoted as KL(q||p)) of our invention exhibits a different mechanism of action in knowledge distillation. It forces the student model's output distribution q to focus on fitting the dominant distribution patterns of the teacher model's output distribution p. This strategy helps the student speech synthesis model more effectively capture key distributional features, especially when faced with multimodal distributions. Therefore, in the distillation learning of large language models, the application of inverse KL divergence helps improve the learning efficiency and generalization ability of the student speech synthesis model.

[0090] Therefore, the choice of divergence metric is crucial for distillation learning of large models. Inverse KL divergence demonstrates superior performance in tasks with complex output spaces and multimodal distributions, while forward KL divergence's advantage in traditional classification tasks stems primarily from its adaptability to small output spaces and single distributions. This paper uses inverse KL divergence instead of forward KL divergence during distillation learning training, a strategy that significantly improves training efficiency and reduces training time.

[0091] The implementation of the above embodiments of the present invention has the following effects:

[0092] 1. The present invention uses distillation training to enable the student speech synthesis model to achieve almost the same effect as the teacher speech synthesis model in processing target language data. In addition, since the target speech synthesis model is only for the target language in a specific scenario, fewer decoding layers are required when setting the student speech synthesis model, making the model's data processing more targeted and efficient, thereby improving the model's reasoning speed. Therefore, the target speech synthesis model of the present invention not only retains the teacher speech synthesis model's ability to process target language data, but also speeds up the reasoning speed of target language data. In certain specific scenarios with high real-time requirements, it can ensure the model's expressiveness and generation quality while meeting the user's real-time needs.

[0093] 2. During the distillation training process, the present invention uses the reverse KL divergence (Reverse KL) to replace the forward KL divergence (Forward KL). Reverse KL divergence exhibits superior performance when dealing with data with complex output spaces and multimodal distributions. In comparison, the advantage of forward KL divergence in traditional classification problems is mainly reflected in its good adaptability to small-scale output spaces and unimodal distributions. During the distillation training process of the present invention, the reverse KL divergence is adopted as our divergence metric, replacing the traditional forward KL divergence, which significantly improves training efficiency and shortens the required time investment, thereby optimizing the entire distillation learning process.

[0094] Based on the above method embodiments, the present invention provides corresponding device embodiments.

[0095] like Figure 5 As shown, an embodiment of the present invention provides a speech synthesis device, comprising: a data acquisition module and a speech synthesis module;

[0096] The data acquisition module is used to acquire the speech data to be processed and the text data to be processed corresponding to the target language;

[0097] The speech synthesis module is used to input the speech data to be processed and the text data to be processed into a target speech synthesis model, so that the target speech synthesis model generates target synthesized speech data that is consistent with the acoustic features of the speech data to be processed and has speech content consistent with the content of the text data to be processed based on the speech data to be processed and the text data to be processed;

[0098] The generation of the target speech synthesis model includes:

[0099] Obtaining a preset teacher speech synthesis model, a preset student speech synthesis model, a speech dataset to be trained corresponding to the target language, and a text dataset to be trained; wherein the number of decoding layers in the teacher speech synthesis model is greater than the number of decoding layers in the student speech synthesis model;

[0100] Repeat the following steps until the relative entropy between the output distribution of the student model and the output distribution of the teacher model reaches a first preset threshold, and use the student speech synthesis model corresponding to the time when the relative entropy reaches the first preset threshold as the target speech synthesis model;

[0101] Randomly select a speech data set to be trained and a text data set to be trained from the current speech data set to be trained and the text data set to be trained respectively;

[0102] Inputting the speech data to be trained and the text data to be trained into the teacher speech synthesis model and the student speech synthesis model, respectively, to obtain the teacher model output distribution of the teacher speech synthesis model and the student model output distribution of the student speech synthesis model;

[0103] Obtaining the relative entropy between the teacher model output distribution and the student model output distribution;

[0104] When it is determined that the relative entropy does not reach the first preset threshold, the model parameters of the student speech synthesis model are adjusted according to the relative entropy, and the current speech data to be trained and the text data to be trained are respectively removed from the speech data set to be trained and the text data set to be trained.

[0105] In a preferred embodiment, the target speech synthesis model includes: an input layer, a decoding layer set, a stream matching layer, and an output layer;

[0106] The input layer is used to collect data features of the voice data to be processed and the text data to be processed;

[0107] The decoding layer set is used to decode the data features of the collected voice data to be processed and the collected text data to be processed, so as to fuse the data features of the voice data to be processed and the text data to be processed to generate corresponding target voice features;

[0108] The stream matching layer is used to generate a mel-spectrogram corresponding to the target speech feature according to the target speech feature;

[0109] The output layer is used to convert the mel-spectrogram into target synthesized speech data having a sound wave signal.

[0110] In a preferred embodiment, the input layer includes: a text encoder and a speech segmenter;

[0111] The text encoder is used to extract text semantic features of the text data to be processed;

[0112] The speech segmenter is used to extract acoustic features of the speech data to be processed; wherein the acoustic features include: pitch, timbre, volume and speaking speed.

[0113] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0115] Based on the above method embodiments, the present invention provides corresponding device embodiments.

[0116] Another embodiment of the present invention provides a device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; when the processor executes the computer program, the speech synthesis method of any embodiment of the present invention is implemented.

[0117] For example, in this embodiment, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more module elements may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the device;

[0118] The device may be a computing device such as a desktop computer, a laptop, a PDA, a cloud server, etc. The device may include, but is not limited to, a processor, a memory;

[0119] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the device, connecting various parts of the entire device using various interfaces and lines;

[0120] The memory can be used to store the computer programs and / or modules. The processor implements the various functions of the device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application required for a function, etc.; in addition, the memory may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0121] Based on the above method embodiment, the present invention provides a corresponding storage medium embodiment.

[0122] Another embodiment of the present invention provides a storage medium, which includes a stored computer program. When the computer program is executed, the device where the storage medium is located is controlled to execute the speech synthesis method of any embodiment of the present invention.

[0123] In this embodiment, the storage medium is a computer-readable storage medium, and the computer program includes computer program code, which may be in source code form, object code form, an executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium.

[0124] By implementing the above-mentioned embodiments of the present invention, it is possible to ensure the expressiveness and generation quality of the target speech synthesis model while also meeting the real-time requirements of users.

[0125] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also regarded as protection of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Obtaining the speech data and text data to be processed corresponding to the target language; Inputting the speech data to be processed and the text data to be processed into a target speech synthesis model, so that the target speech synthesis model generates target synthesized speech data having the same acoustic features as the speech data to be processed and the same speech content as the text data to be processed based on the speech data to be processed and the text data to be processed; The generation of the target speech synthesis model includes: Obtaining a preset teacher speech synthesis model, a preset student speech synthesis model, a speech dataset to be trained corresponding to the target language, and a text dataset to be trained; wherein the number of decoding layers in the teacher speech synthesis model is greater than the number of decoding layers in the student speech synthesis model; Repeat the following steps until the relative entropy between the output distribution of the student model and the output distribution of the teacher model reaches a first preset threshold, and use the student speech synthesis model corresponding to the time when the relative entropy reaches the first preset threshold as the target speech synthesis model; Randomly select a speech data set to be trained and a text data set to be trained from the current speech data set to be trained and the text data set to be trained respectively; Inputting the speech data to be trained and the text data to be trained into the teacher speech synthesis model and the student speech synthesis model, respectively, to obtain the teacher model output distribution of the teacher speech synthesis model and the student model output distribution of the student speech synthesis model; Obtaining the relative entropy between the teacher model output distribution and the student model output distribution; When it is determined that the relative entropy does not reach the first preset threshold, the model parameters of the student speech synthesis model are adjusted according to the relative entropy, and the current speech data to be trained and the text data to be trained are respectively removed from the speech data set to be trained and the text data set to be trained.

2. A speech synthesis method according to claim 1, characterized in that: The target speech synthesis model includes: an input layer, a decoding layer set, a stream matching layer and an output layer; The input layer is used to collect data features of the voice data to be processed and the text data to be processed; The decoding layer set is used to decode the data features of the collected voice data to be processed and the collected text data to be processed, so as to fuse the data features of the voice data to be processed and the text data to be processed to generate corresponding target voice features; The stream matching layer is used to generate a mel-spectrogram corresponding to the target speech feature according to the target speech feature; The output layer is used to convert the mel-spectrogram into target synthesized speech data having a sound wave signal.

3. A speech synthesis method according to claim 2, characterized in that: The input layer includes: a text encoder and a speech segmenter; The text encoder is used to extract text semantic features of the text data to be processed; The speech segmenter is used to extract acoustic features of the speech data to be processed; wherein the acoustic features include: pitch, timbre, volume and speaking speed.

4. A speech synthesis method according to claim 3, characterized in that: The student speech synthesis model has 6 decoding layers.

5. A speech synthesis method according to claim 4, characterized in that: The relative entropy includes: inverse KL divergence.

6. A speech synthesis device, characterized in that: include: Data acquisition module and speech synthesis module; The data acquisition module is used to acquire the speech data to be processed and the text data to be processed corresponding to the target language; The speech synthesis module is used to input the speech data to be processed and the text data to be processed into a target speech synthesis model, so that the target speech synthesis model generates target synthesized speech data that is consistent with the acoustic features of the speech data to be processed and has speech content consistent with the content of the text data to be processed based on the speech data to be processed and the text data to be processed; The generation of the target speech synthesis model includes: Obtaining a preset teacher speech synthesis model, a preset student speech synthesis model, a speech dataset to be trained corresponding to the target language, and a text dataset to be trained; wherein the number of decoding layers in the teacher speech synthesis model is greater than the number of decoding layers in the student speech synthesis model; Repeat the following steps until the relative entropy between the output distribution of the student model and the output distribution of the teacher model reaches a first preset threshold, and use the student speech synthesis model corresponding to the time when the relative entropy reaches the first preset threshold as the target speech synthesis model; Randomly select a speech data set to be trained and a text data set to be trained from the current speech data set to be trained and the text data set to be trained respectively; Inputting the speech data to be trained and the text data to be trained into the teacher speech synthesis model and the student speech synthesis model, respectively, to obtain the teacher model output distribution of the teacher speech synthesis model and the student model output distribution of the student speech synthesis model; Obtaining the relative entropy between the teacher model output distribution and the student model output distribution; When it is determined that the relative entropy does not reach the first preset threshold, the model parameters of the student speech synthesis model are adjusted according to the relative entropy, and the current speech data to be trained and the text data to be trained are respectively removed from the speech data set to be trained and the text data set to be trained.

7. The speech synthesis device according to claim 6, wherein: The target speech synthesis model includes: an input layer, a decoding layer set, a stream matching layer and an output layer; The input layer is used to collect data features of the voice data to be processed and the text data to be processed; The decoding layer set is used to decode the data features of the collected voice data to be processed and the collected text data to be processed, so as to fuse the data features of the voice data to be processed and the text data to be processed to generate corresponding target voice features; The stream matching layer is used to generate a mel-spectrogram corresponding to the target speech feature according to the target speech feature; The output layer is used to convert the mel-spectrogram into target synthesized speech data having a sound wave signal.

8. The speech synthesis device according to claim 7, wherein: The input layer includes: a text encoder and a speech segmenter; The text encoder is used to extract text semantic features of the text data to be processed; The speech segmenter is used to extract acoustic features of the speech data to be processed; wherein the acoustic features include: pitch, timbre, volume and speaking speed.

9. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the speech synthesis method according to any one of claims 1 to 5 when executing the computer program.

10. A storage medium, characterized in that: The storage medium includes a stored computer program, wherein when the computer program is executed, the device where the storage medium is located is controlled to execute the speech synthesis method according to any one of claims 1 to 5.

Images