Speech synthesis model training method, speech synthesis method and related equipment

Abstract

An embodiment of the present application discloses a training method, device and electronic device for a speech synthesis model, the method comprising: obtaining a training sample data pair and a speech synthesis model, the speech synthesis model comprising a phoneme processing module and a text processing module; performing feature extraction on the phoneme data by a phoneme feature extraction module in the phoneme processing module to obtain phoneme features, and performing feature extraction on the training text by the text processing module to obtain text features; fusing the phoneme features and the text features by the text processing module, and performing speech synthesis processing based on the fused features by the speech synthesis module to obtain training speech corresponding to the training text; and training the speech synthesis model based on the training speech corresponding to the training text and the synthesized speech label corresponding to the training text.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a training method for a speech synthesis model, a speech synthesis method, and related equipment. Background Art

[0002] With the rapid development of artificial intelligence technology, end-to-end text-to-speech (TTS) technology is becoming increasingly popular. Currently, TTS models typically extract phoneme features from phoneme data and then perform speech synthesis based on these phoneme features. Training TTS models typically requires a large amount of annotated data to achieve smooth and clear synthesized speech. Due to the limited number of phoneme features, the synthesized speech based on these phoneme features can easily have low accuracy. Therefore, further solutions are still needed to obtain a highly accurate speech synthesis model to improve the quality of speech synthesis. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a training method for a speech synthesis model, a speech synthesis method, and related equipment to improve the accuracy of the speech synthesis model.

[0004] To solve the above technical problems, the embodiments of the present application are implemented as follows:

[0005] First, a training method for a speech synthesis model is proposed, including:

[0006] Obtaining a training sample data pair and a speech synthesis model, wherein the speech synthesis model includes a phoneme processing module and a text processing module, the phoneme processing module includes a phoneme feature extraction module and a speech synthesis module, and the training sample data pair includes a training text, phoneme data corresponding to the training text, and a synthesized speech label corresponding to the training text;

[0007] Performing feature extraction on the phoneme data by the phoneme feature extraction module in the phoneme processing module to obtain phoneme features, and performing feature extraction on the training text by the text processing module to obtain text features;

[0008] The text processing module fuses the phoneme features and the text features, and the speech synthesis module performs speech synthesis processing based on the fused features to obtain training speech corresponding to the training text;

[0009] The speech synthesis model is trained based on the training speech corresponding to the training text and the synthesized speech label corresponding to the training text.

[0010] Secondly, a speech synthesis method is proposed, comprising:

[0011] Obtaining a text to be synthesized and phoneme data corresponding to the text to be synthesized;

[0012] Inputting the phoneme data into a phoneme feature extraction module included in a phoneme processing module in a speech synthesis model to perform feature processing to obtain phoneme features, and inputting the text to be synthesized into a text processing module in the speech synthesis module to perform feature extraction to obtain text features; wherein the speech synthesis model is trained based on the speech synthesis model training method described in the first aspect;

[0013] The phoneme features and the text features are fused by the text processing module, and the speech synthesis module included in the phoneme processing module performs speech synthesis processing based on the fused features to obtain the synthesized speech corresponding to the text to be synthesized.

[0014] Thirdly, a training device for a speech synthesis model is proposed, comprising:

[0015] an acquisition unit, configured to acquire a training sample data pair and a speech synthesis model, wherein the speech synthesis model includes a phoneme processing module and a text processing module, the phoneme processing module includes a phoneme feature extraction module and a speech synthesis module, and the training sample data pair includes a training text, phoneme data corresponding to the training text, and a synthesized speech label corresponding to the training text;

[0016] a first extraction unit, configured to perform feature extraction on the phoneme data to obtain phoneme features by using a phoneme feature extraction module in the phoneme processing module, and to perform feature extraction on the training text to obtain text features by using the text processing module;

[0017] A second extraction unit is configured to fuse the phoneme features and the text features through the text processing module, and perform speech synthesis processing based on the fused features through the speech synthesis module to obtain a training speech corresponding to the training text;

[0018] The model training unit is used to train the speech synthesis model based on the training speech corresponding to the training text and the synthesized speech label corresponding to the training text.

[0019] In a fourth aspect, a speech synthesis device is proposed, comprising:

[0020] An acquisition unit, configured to acquire a text to be synthesized and phoneme data corresponding to the text to be synthesized;

[0021] an extraction unit, configured to input the phoneme data into a phoneme feature extraction module included in a phoneme processing module in a speech synthesis model to perform feature processing to obtain phoneme features, and input the text to be synthesized into a text processing module in the speech synthesis module to perform feature extraction to obtain text features; wherein the speech synthesis model is trained based on the speech synthesis model training method described in the first aspect;

[0022] The synthesis unit fuses the phoneme features and the text features through the text processing module, and performs speech synthesis processing based on the fused features through the speech synthesis module included in the phoneme processing module to obtain the synthesized speech corresponding to the text to be synthesized.

[0023] In the fifth aspect, an electronic device is proposed, characterized in that it includes: a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein when the computer program is executed by the processor, the steps of the method described in the first aspect are implemented; or when the computer program is executed by the processor, the steps of the method described in the second aspect are implemented.

[0024] In the sixth aspect, a computer-readable storage medium is proposed, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the steps of the method described in the first aspect are implemented; or, when the computer program is executed by the processor, the steps of the method described in the second aspect are implemented.

[0025] It can be seen from the technical solutions provided by the above embodiments of the present application that the embodiments of the present application have at least one of the following technical effects:

[0026] In one or more embodiments provided herein, a speech synthesis model includes a phoneme processing module and a text processing module. The phoneme processing module in the speech synthesis model can be used not only to process the phoneme data corresponding to the training text to obtain phoneme features, but also to process the training text to obtain text features. Since the text features also include prosodic features, the text processing module can be used to fuse the phoneme features and text features to obtain richer prosodic features. Finally, the speech synthesis module performs speech synthesis processing based on the richer features obtained through the fusion. Since the trained speech synthesis model is not only based on phoneme features, but can also perform speech synthesis based on the prosodic features in the text data, the speech synthesis model trained using this speech synthesis model training method has a higher speech synthesis accuracy and can synthesize speech with better sound quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0028] Figure 1 This is a schematic diagram of the implementation flow of a method for training a speech synthesis model provided in an embodiment of the present application.

[0029] Figure 2 Schematic diagram of synthesized speech labels in training sample pairs of a speech synthesis model training method provided according to one embodiment of the present application.

[0030] Figure 3 2 is a schematic diagram of the network architecture of a speech synthesis model provided according to an embodiment of the present application.

[0031] Figure 4 2 is a schematic diagram of the network architecture of the encoder in the FastSpeech2 model provided according to an embodiment of the present application.

[0032] Figure 5 This is a schematic diagram of the network structure of a multi-head attention mechanism provided according to an embodiment of the present application.

[0033] Figure 6 It is a schematic diagram of an implementation flow of a speech synthesis method provided according to an embodiment of the present application.

[0034] Figure 7 This is a schematic diagram of the application of a speech synthesis method provided in an embodiment of the present application in an actual scenario.

[0035] Figure 8 It is a structural diagram of a training device for a speech synthesis model provided according to an embodiment of the present application.

[0036] Figure 9 This is a structural diagram of a speech synthesis device provided according to an embodiment of the present application.

[0037] Figure 10 This is a schematic structural diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of this application more apparent, the following exemplary embodiments of this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this application, rather than all the embodiments of this application, and it should be understood that this application is not limited to the exemplary embodiments described herein.

[0039] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application.

[0040] It should be understood that the various steps described in the method embodiments of the present application can be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0041] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Definitions of other terms are provided below.

[0042] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0043] It should be noted that the modifications of "one" and "multiple" mentioned in this application are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0044] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0045] When studying speech synthesis technology, the present applicant discovered that speech synthesis technology includes autoregressive models and non-autoregressive models. Autoregressive models generally use attention technology to learn the alignment between phonemes and speech frames, such as tacotron2. Commonly used non-regressive models include FastSpeech1, which is constructed based on the Teacher-Student knowledge distillation technology, and the FastSpeech2 model. Among them, the FastSpeech2 model is a TTS model jointly proposed by Microsoft Research Asia and Zhejiang University. Based on FastSpeech1, it abandons the Teacher-Student knowledge distillation framework to reduce training complexity, and directly uses real speech data as the training target to avoid information loss. At the same time, it introduces more accurate duration information and other variable information in speech, such as pitch, volume energy, etc. to improve the quality of synthesized speech.

[0046] The FastSpeech2 model incorporates a multi-head attention mechanism in both the encoder and decoder structures. Although the FastSpeech2 model performs well in speech synthesis, training is typically based on the phoneme features of the training text. This training method may require more labeled data for model convergence. Furthermore, the relatively limited features used during training may result in low speech synthesis accuracy after the converged model.

[0047] On this basis, the present application proposes an improved speech synthesis model and a training scheme for the speech synthesis model. In general, the improved speech synthesis model includes a phoneme processing module and a text processing module. When training the speech synthesis model, firstly, a training sample data pair is obtained, wherein the training sample data pair includes a training text, phoneme data corresponding to the training text, and a synthesized speech label corresponding to the training text; further, the phoneme data is subjected to feature extraction processing by the phoneme processing module in the speech synthesis model to obtain phoneme features, and the training text is subjected to feature extraction by the text processing module in the speech synthesis model to obtain text features; finally, the phoneme features and text features are fused and input into the text synthesis module for speech synthesis processing, and the speech synthesis model is trained based on the training speech obtained by the speech synthesis processing and the synthesized speech label corresponding to the training text.

[0048] The speech synthesis model after training can be applied to any application scenario that synthesizes text into speech. For example, the intelligent voice navigation application scenario can synthesize navigation speech based on navigation text and the phoneme data of a specified person; the intelligent video commentary application scenario can synthesize video content commentary based on the subtitle content of the video or the introduction of the video content and the phoneme data of a specified person; the intelligent text reading assistant application scenario can synthesize speech based on the text content of a book or chat text and the phoneme data of a specified person. The specified person can be a robot, a well-known public figure, or a specified friend in the chat list.

[0049] It should be understood that the speech synthesis model training scheme provided in the embodiments of the present application can be executed by an electronic device, specifically by a processor of the electronic device. The electronic device can be a terminal device, such as a smartphone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device, smart home appliance, smart watch, vehicle terminal, aircraft, etc.; or the electronic device can also be a server, such as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0050] Based on the above-mentioned training scheme of the speech synthesis model, the present application embodiment provides a training method for the speech synthesis model. Figure 1 , is a flow chart of a method for training a speech synthesis model provided by an embodiment of the present application. Figure 1 The training method may include the following steps:

[0051] S110, obtaining a training sample data pair and a speech synthesis model, the speech synthesis model includes a phoneme processing module and a text processing module, the phoneme processing module includes a phoneme feature extraction module and a speech synthesis module, the training sample data pair includes a training text, the phoneme data corresponding to the training text, and the synthesized speech label corresponding to the training text.

[0052] It should be understood that the synthesized speech label corresponding to the training text in each training sample data pair should correspond to the training text and the phoneme data corresponding to the training text, that is, the text corresponding to the synthesized speech label should be consistent with the training text, and the phoneme corresponding to the synthesized speech label should be consistent with the phoneme data corresponding to the training text. The phoneme data corresponding to the training text includes pinyin, tone, and rhythm (i.e., pause time), where tone includes first, second, third, and fourth tones. First tone can be represented as 1, second tone can be represented as 2, third tone can be represented as 3, fourth tone can be represented as 4, light tone as 5, and variable tone as 6. Rhythm can be represented by 7, 8, or 9, with 7 indicating a short pause time, 9 indicating a long pause time, and 8 indicating a medium pause time.

[0053] Optionally, the corresponding synthetic speech labels and training texts in the training sample data pair can be determined by any of the following methods: given a training text, the training text is manually converted into speech, and the converted speech is the synthetic speech label corresponding to the training text; given a speech, the speech is converted into text manually or by artificial intelligence technology, and the converted text is the training text, and the given speech is the synthetic speech label corresponding to the training text. It should be noted that the embodiment of the present application only lists two feasible ways to determine the corresponding training text and synthetic speech labels. In actual applications, the corresponding training text and speech synthesis labels can be obtained according to the actual situation. For example, a speech synthesis model that has been trained usually performs speech synthesis on a training text to obtain the synthetic speech label corresponding to the training text; or, from a video with audio and subtitles, corresponding subtitles and audio are clipped as corresponding training text and synthetic speech labels.

[0054] As an example, a training sample data pair can be represented as<phonemes,text,auido> , where phonemes is the phoneme data corresponding to the training text, text is the training text, and auido is the synthesized speech label corresponding to the training text text. For example, the training text in a training sample data pair can be represented as: "Jeju Island, the largest island in South Korea", and the phoneme data phonemes corresponding to the training text is represented as: "han2 guo2 7zui4 da4 de5 7dao6 yu6 7ji3zhou1 dao3", where 2 in han2 indicates that the tone is the second tone, 2 in guo2 7 indicates that the tone of the pinyin guo is the second tone, and 7 indicates that there is a short pause after the pinyin guo. The synthesized speech label in this training sample pair can be represented as Figure 2 As shown, Figure 2 What is shown is a voice, and the playback result of the voice is: Jeju Island, the largest island in South Korea.

[0055] In one embodiment, the speech synthesis model is used to perform speech synthesis, see Figure 3 , which is a structural diagram of a speech synthesis model provided in an embodiment of the present application. Figure 3In the speech synthesis model shown, 301 represents a phoneme processing module, and 302 represents a text processing module. As the names imply, the phoneme processing module 301 is mainly used to process phoneme data, and the text processing module 302 is mainly used to process text. The phoneme processing module 301 can be a FastSpeech2 model, and the phoneme processing module 301 can also include a phoneme feature extraction module 3011 and a speech synthesis module 3012, wherein the phoneme feature extraction module can refer to the phoneme vectorization layer Phoneme Embedding in the FastSpeech2 model. Figure 3 As shown in 30111 and the encoding layer Encoder is as follows Figure 3 As shown in 30112, the speech synthesis module 3012 may include a variable information adapter in the FastSpeech2 model as shown in 30121, a Mel spectrum decoder module as shown in Figure 3 30122 and vocoder as Figure 3 30123. The text processing module 302 may be a Bert model, and the text processing module 302 may further include a text vector extraction module 3021 and a feature fusion module 3022. The text vector extraction module 3021 may be the text vectorization layer LMEmbeddings layer in the Bert model, and the feature fusion module 3022 may be the self-attention mechanism network SelfAttention in the Bert model.

[0056] Bert stands for Bidirectional Encoder Representation from Transformers, meaning the encoder of a bidirectional Transformer, as the encoder does not have access to the information being predicted. The Bert model's main innovation lies in its pre-training method, using Masked LM and NextSentence Prediction to capture representations at the word and sentence levels, respectively. The Transformer is a time series model based on a self-attention mechanism. Its encoder effectively encodes time series information, typically processing it far better than LSTM and at a much faster speed. Transformers are widely used in fields such as natural language processing, computer vision, machine translation, and speech recognition.

[0057] S120 , performing feature extraction on the phoneme data by the phoneme feature extraction module in the phoneme processing module to obtain phoneme features, and performing feature extraction on the training text by the text processing module to obtain text features.

[0058] Specifically, the phoneme feature extraction module in the phoneme processing module extracts the phoneme data to obtain the phoneme feature, which can be obtained by Figure 3 The phoneme feature extraction module 3011 in the FastSpeech2 model shown, that is, the phoneme vectorization layer Phoneme Embedding is as shown in FIG. Figure 3 As shown in 30111 and the encoding layer Encoder is as follows Figure 3 As shown in 30112, feature extraction is performed on the phoneme data to obtain phoneme features.

[0059] Figure 4 This is a diagram of the network architecture of the encoder in the FastSpeech2 model, that is, Figure 3 The network architecture diagram of Encoder 30112 is shown in the figure. The Encoder in the FastSpeech2 model uses a transformer network structure, and the core of the transformer is the multi-head attention mechanism. Figure 4 In the figure, phoneme data is input to the input embedding layer as input data (input), and then input to the multi-head attention mechanism network through positional encoding. The input and output results of each layer of the multi-head attention mechanism network are added and normalized through the Add&Norm layer. Finally, the output result of the encoder is obtained through the forward propagation Feed Forward layer and the vector addition and layer normalization Add&Norm layer.

[0060] Figure 5 The left figure is a schematic diagram of the network structure of the multi-head attention mechanism Multi-headattention attention mechanism, where the right figure is a schematic diagram of the network structure of the scaled dot product attention mechanism Scaled Dot-Product Attention. Specifically, the input of the multi-head attention mechanism network Multi-head Attention is Q, K and V. Q and K can be obtained by matrix multiplication to obtain an attention matrix, that is, sequentially through Figure 5 In the left figure, the vector machine MatMul, scale, mask, and activation function SoftMax can obtain an attention matrix, and the attention matrix can be multiplied by the vector machine MatMul and V to obtain a feature with attention information. Figure 5The Concat layer in the right figure performs splicing and fusion, as well as linear mapping.

[0061] While retaining the original FastSpeech2 model's network structure, the present embodiment integrates the Bert model's text vector extraction module (LMEmbeddings) and feature fusion module, namely, the Self-Attention module, based on the FastSpeech2 model's network structure. The Bert model is trained with a large amount of text data, allowing it to learn the rich prosodic features of the training text even with less annotated sample data, thereby improving the quality of speech synthesized by the native FastSpeech2 model.

[0062] Taking the training text "Jeju Island, the largest island in South Korea" as an example, the rhythm in the training text, for example, "South Korea" is a phrase, and "the largest" is another phrase, then when reading, there is usually a pause between "South Korea" and "the largest". In order to improve the sound quality of the synthesized speech and make the synthesized speech closer to the speech of a real person, the embodiment of the present application, based on the phoneme features, also considers the text features corresponding to the training text, and the text features also include rhythmic features.

[0063] Optionally, to achieve deep fusion of phoneme features and text features in each training sample data pair, embodiments of the present application can utilize the self-attention mechanism network of the Bert model to fuse the first text feature K and the second text feature V of each training sample data pair with its phoneme features, where K=V. Based on this, the training text can be first subjected to feature extraction through a text vector extraction module. Specifically, the text processing module includes a text vector extraction module and a feature fusion module. Feature extraction of the training text by the text processing module obtains text features, including:

[0064] Perform feature extraction on the training text by using a text vector extraction module in the text processing module to obtain a first text feature;

[0065] A second text feature is determined based on the first text feature.

[0066] The first text feature and the second text feature are the same.

[0067] The text vector extraction module in the text processing module performs feature extraction on the training text. Specifically, the text vector extraction module (LM Embeddings) in the Bert model extracts the first text feature of the training text in each training sample data pair, and then determines the second text feature based on the first text feature. The first text feature and the second text feature can be LM Embedding vectors.

[0068] As an example, the text vector extraction module in the text processing module performs feature extraction on the training text "The weather is nice today" to obtain sentence vectors of specific dimensions, namely the first text vector and the second text vector, as follows:

[0069] [[1.21984698e-01 7.84057677e-02-1.06496774e-01-3.25891018e-014.94978607e-01-4.69692767e-01 2.54333645e-01-8.82656407e-03...]]

[0070] Because the text vector extraction module in the text processing module in the speech synthesis model uses the text vector extraction module (LM Embeddings) in the Bert model and uses the Encoder in the Transformer model, which contains a self-attention mechanism, the text vector extraction module used in the embodiment of the present application can learn the prosodic information before and after specific words in the training text, that is, obtain deeper semantic features in the training text. This makes the sound quality of the speech synthesized by the speech synthesis model provided by the embodiment of the present application better.

[0071] S130: The text processing module fuses the phoneme features and the text features, and the speech synthesis module performs speech synthesis processing based on the fused features to obtain training speech corresponding to the training text.

[0072] Optionally, the text processing module also includes a feature fusion module; the phoneme feature and the text feature are fused through the text processing module, including: using the feature fusion module to fuse the phoneme feature, the first text feature and the second text feature based on the attention mechanism.

[0073] exist Figure 3 In order to fuse the output result Q of the encoder and the output results K and V of the text vector extraction module, the architecture of the speech synthesis model in the embodiment of the present application integrates the self-attention mechanism network structure of the Bert model, where K=V, both are the output results of the text vector extraction module, namely the first text feature and the second text feature, and Q is the output result of the encoder in the native FastSpeech2, namely the phoneme feature. While retaining the length of the text vector, the output result phoneme feature Q of the encoder in FastSpeech2 and the first and second text features K and V extracted by the text vector extraction module can be deeply fused.

[0074] Optionally, the speech synthesis module further includes a variable information adapter, a mel spectrum decoder module, and a vocoder. The speech synthesis module performs speech synthesis processing based on the fused features to obtain training speech corresponding to the training text, including:

[0075] Performing an addition operation on the fused features and the phoneme features, and inputting the addition operation result into a variable information adapter for processing to obtain a variable information sequence;

[0076] The variable information sequence is processed by the Mel spectrum decoder module to obtain the Mel spectrum features of the training speech corresponding to the training text;

[0077] The mel spectrum features of the training speech corresponding to the training text are processed by the vocoder to perform speech synthesis processing to obtain the training speech corresponding to the training text.

[0078] Mel-frequency Cepstral Coefficients (MFCCs) are widely used in speaker segmentation, voiceprint recognition, speech recognition, and speech synthesis. Mel-frequency is based on the characteristics of human hearing and has a nonlinear relationship with hertz frequency. Mel-frequency features utilize this relationship to calculate hertz frequency features. Mel-frequency features are primarily used for feature extraction in speech data.

[0079] Figure 3 The network architecture of the speech synthesis model shown in the figure incorporates a variable information adapter (Variance Adapter) between the encoder and the mel-spectrogram decoder. This allows for the introduction of more speech-varying information, such as duration, pitch, and volume (spectral energy), into the trained speech synthesis model. This addresses the one-to-many mapping problem in speech synthesis. Specifically, in text-to-speech synthesis, a single text can correspond to multiple variable speech sounds, including duration, pitch, and volume.

[0080] Specifically, the training text in each training sample data pair can be extracted by the text vector extraction module to obtain a sentence vector for the text. The phoneme data corresponding to the training text can be extracted by the phoneme feature extraction module of the speech synthesis model to obtain a feature vector for the phoneme data. Positional encoding is then used to obtain contextual information about the position of each phoneme in each phoneme data. The encoder can then learn a deeper phoneme feature Q. The first and second text features K and V output by the text vector extraction module are then fused with the deeper phoneme feature Q output by the encoder. Specifically, the self-attention mechanism network of the speech synthesis model is used to combine the feature information obtained by fusion, output, with the encoder output Q. The result is then input into the mel-spectrogram decoder of the speech synthesis model for training, resulting in the speech synthesis model. This training process preserves the original information of the phonemes in each training sample data while integrating the feature information of the training text.

[0081] Optionally, the text vector extraction module in the text processing module in the speech synthesis model can be obtained based on not only the Bert model but also other pre-trained models, such as the AlBert model, the Bart model, and the like.

[0082] S140 , training a speech synthesis model based on the training speech corresponding to the training text and the synthesized speech label corresponding to the training text.

[0083] Optionally, in order to enable the trained speech synthesis model to synthesize speech that is close to real speech, the embodiment of the present application can use the mel-spectrogram features of the synthesized speech labels in each training sample data pair as a target, and continuously reduce the difference between the mel-spectrogram features of the training speech and the target spectrogram features, that is, adjust the model parameters of the speech synthesis model so that the mel-spectrogram features of the training speech synthesized by the speech synthesis model continuously approach the target spectrogram features. Specifically, based on the training speech corresponding to the training text and the synthesized speech label corresponding to the training text, the speech synthesis model is trained, including: obtaining the target spectrogram features of the synthesized speech label corresponding to the training text; adjusting the model parameters of the speech synthesis model in the direction of reducing the difference between the mel-spectrogram features of the training speech and the target spectrogram features. Among them, obtaining the target spectrogram features of the synthesized speech label corresponding to the training text can be obtained by methods such as Fourier transform.

[0084] One or more embodiments provided by the present application, the speech synthesis model includes a phoneme processing module and a text processing module, which can not only use the phoneme processing module in the speech synthesis model to process the phoneme data corresponding to the training text to obtain phoneme features, but also use the text processing module in the speech synthesis model to process the training text to obtain text features, and then use the text processing module to fuse the phoneme features and text features to obtain richer rhythmic features, and finally use the speech synthesis module to perform speech synthesis processing based on the richer features obtained by fusion. Since the speech synthesized based on the speech synthesis model is not only based on phoneme features, but also can be based on text features, the speech synthesis model trained using the speech synthesis model training method can synthesize speech with better sound quality.

[0085] Based on the above-mentioned speech synthesis model training method embodiment, the present application embodiment provides a speech synthesis method, see Figure 6 , is a schematic diagram of an implementation flow of a speech synthesis method provided by an embodiment of the present application. Figure 6 The speech synthesis method shown can be executed by an electronic device, specifically by a processor of the electronic device. The electronic device can be a terminal device, such as a smart phone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device, smart home appliance, smart watch, vehicle terminal, aircraft, etc.; or, the electronic device can also be a server, such as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; or, the speech synthesis method can also be executed by a client that can implement the speech synthesis method, such as a chat application client, a navigation application client, a reading application client, etc. Specifically, Figure 6 The method may include the following steps:

[0086] S610: Obtain the text to be synthesized and the phoneme data corresponding to the text to be synthesized.

[0087] The text to be synthesized can be any text that needs to be synthesized into speech. The text to be synthesized varies in different application scenarios. For example, in the intelligent text reading assistant scenario, the text to be synthesized can be text chat messages in a chat application page or book text in a reading application page. In the intelligent video commentary scenario, the text to be synthesized can be the introduction of the video content or the subtitles in the video.

[0088] Taking the intelligent text reading assistant scenario as an example, when the text to be synthesized is a text chat message in a chat application page, the phoneme data corresponding to the text to be synthesized can be the phoneme data of a robot configured in the chat application, or the phoneme data of a contact corresponding to the text chat message in the chat application. When the phoneme data corresponding to the text to be synthesized is the phoneme data of a robot configured in the chat application, the phoneme data can be obtained based on the voice features of the robot. When the phoneme data corresponding to the text to be synthesized is the phoneme data of a contact corresponding to the text chat message in the chat application, the phoneme data can be obtained based on the voice information of the contact in the historical chat record (the voice information comes from the contact, that is, the voice information is recorded and sent by the contact).

[0089] S620, input the phoneme data into the phoneme feature extraction module included in the phoneme processing module in the speech synthesis model, perform feature processing to obtain phoneme features, and input the text to be synthesized into the text processing module in the speech synthesis module to perform feature extraction to obtain text features.

[0090] Among them, the speech synthesis model is based on Figure 1 The training method of the speech synthesis model is used for training. Figure 1 The relevant steps in the embodiment and the specific implementation of step S602 are the same as Figure 1 The implementation of step S120 in the embodiments is the same and will not be repeated here.

[0091] S630: The phoneme features and the text features are fused through the text processing module, and a speech synthesis module included in the phoneme processing module performs speech synthesis processing based on the fused features to obtain a synthesized speech corresponding to the text to be synthesized.

[0092] It should be understood that the implementation method included in step S630 is the same as Figure 1 The implementation of step S130 in the embodiment is the same, and the details can be found in the above description, which will not be repeated here.

[0093] In one embodiment, after obtaining the synthesized speech corresponding to the text to be synthesized, the electronic device may output the synthesized speech corresponding to the text to be synthesized.

[0094] As a feasible implementation method, outputting the synthesized speech corresponding to the text to be synthesized may include: outputting identification information corresponding to the synthesized speech corresponding to the text to be synthesized; and playing the synthesized speech corresponding to the text to be synthesized when the identification information is triggered.

[0095] As another feasible implementation, outputting the synthesized speech corresponding to the text to be synthesized may also include: directly playing the synthesized speech corresponding to the text to be synthesized.

[0096] It should be understood that the method of outputting the synthesized speech corresponding to the text to be synthesized may vary in different application scenarios, as well as in the same application scenario. For example, in the intelligent voice navigation application scenario, direct output can be used; in the intelligent text assistant reading application scenario, identification information can be output first, and when the identification information is detected to be triggered, the synthesized speech corresponding to the text to be synthesized can be played.

[0097] For example, in the chat application scenario of the intelligent text assistant reading application scenario, the user may not be able to listen to the voice at any time. Therefore, when obtaining the synthesized voice corresponding to the text to be synthesized, the current time can be detected. If the current time is working hours, the synthesized voice will not be played directly, but the identification information corresponding to the synthesized voice will be output first; when the identification information is triggered, the synthesized voice will be played; conversely, if the current time is not working hours, the synthesized voice can be played directly.

[0098] Figure 7 This is a schematic diagram of the speech synthesis method provided in the embodiment of the present application being applied to a practical scenario. Figure 7 In the chat page of group chat 1, a user can click on a text chat message from user A to trigger a speech synthesis operation for the text chat message. After the user clicks on the text chat message, in response to the click operation, the phoneme data of the robot configured in the chat application can be obtained based on the text chat message, and the obtained phoneme data corresponds to the text chat message; alternatively, the voice messages previously sent by user A in group chat 1 can be obtained, and based on the text chat message and the previously sent voice messages of user A, the phoneme data of user A can be obtained, and the phoneme data corresponding to the text chat message.

[0099] After obtaining the phoneme data corresponding to the text chat message, a speech synthesis operation can be performed based on the text chat message and the corresponding phoneme data to obtain the synthesized speech corresponding to the text chat message. After obtaining the synthesized speech corresponding to the text chat message, the synthesized speech can be played directly, or the current time can be detected. If the current time is working hours, the synthesized speech will not be played directly. Instead, identification information corresponding to the synthesized speech will be output first, such as a pop-up prompt box "Do you want to play the synthesized speech?"; when the identification information is triggered, such as when the user clicks the "Yes" button, the synthesized speech will be played.

[0100] One or more embodiments provided by the present application, when performing speech synthesis operations on a text to be synthesized and phoneme data corresponding to the text to be synthesized, not only can the phoneme feature extraction module included in the phoneme processing module in the speech synthesis model be used to process the phoneme data corresponding to the text to be synthesized to obtain phoneme features, but the text processing module in the speech synthesis model included in the phoneme processing module can also be used to process the text to be synthesized to obtain text features, and then the phoneme features and text features are fused through the text processing module to obtain richer rhythmic features, and finally the speech synthesis module is used to perform speech synthesis processing based on the richer features obtained by the fusion. Since the synthesized speech is not only based on phoneme features, but can also be based on rhythmic information in text features, the synthesized speech is more accurate and has better sound quality.

[0101] Based on the above-mentioned speech synthesis model training method embodiment, the present application embodiment provides a speech synthesis model training device. Figure 8 This is a structural diagram of a speech synthesis model training device 800 provided by an embodiment of the present application. Figure 8 In a software implementation, the speech synthesis model training device 800 may include:

[0102] An acquisition unit 801 is configured to acquire a training sample data pair and a speech synthesis model, wherein the speech synthesis model includes a phoneme processing module and a text processing module, wherein the phoneme processing module includes a phoneme feature extraction module and a speech synthesis module, and wherein the training sample data pair includes a training text, phoneme data corresponding to the training text, and a synthesized speech label corresponding to the training text;

[0103] A first extraction unit 802 is configured to perform feature extraction on the phoneme data to obtain phoneme features using a phoneme feature extraction module in the phoneme processing module, and to perform feature extraction on the training text to obtain text features using the text processing module;

[0104] A second extraction unit 803 is configured to fuse the phoneme features and the text features through the text processing module, and perform speech synthesis processing based on the fused features through the speech synthesis module to obtain a training speech corresponding to the training text;

[0105] The model training unit 804 is used to train the speech synthesis model based on the training speech corresponding to the training text and the synthesized speech label corresponding to the training text.

[0106] Optionally, the text processing module includes a text vector extraction module and a feature fusion module. When the first extraction unit 802 extracts features from the phoneme data through the phoneme feature extraction module in the phoneme processing module to obtain phoneme features, the first extraction unit 802 specifically performs the following steps:

[0107] Using the text vector extraction module in the text processing module to perform feature extraction on the training text to obtain a first text feature;

[0108] A second text feature is determined based on the first text feature.

[0109] Optionally, the text processing module further includes a feature fusion module; when the second extraction unit 803 fuses the phoneme feature and the text feature through the text processing module, specifically performs:

[0110] The feature fusion module is used to fuse the phoneme feature, the first text feature and the second text feature based on the attention mechanism.

[0111] Optionally, in one embodiment, the speech synthesis module includes a variable information adapter, a mel spectrum decoder module, and a vocoder. The second extraction unit 803 performs speech synthesis processing based on the fused features through the speech synthesis module, specifically performing:

[0112] Performing an addition operation on the fused feature and the phoneme feature, and inputting the addition operation result into the variable information adapter for processing to obtain a variable information sequence;

[0113] Processing the variable information sequence through the Mel spectrum decoder module to obtain Mel spectrum features of the training speech corresponding to the training text;

[0114] The vocoder performs speech synthesis processing on the Mel spectrum features of the training speech corresponding to the training text to obtain the training speech corresponding to the training text.

[0115] Optionally, in one embodiment, when training the speech synthesis model based on the training speech corresponding to the training text and the synthesized speech label corresponding to the training text, the model training unit 804 specifically performs:

[0116] Obtaining target spectrum features of the synthesized speech tag corresponding to the training text;

[0117] The model parameters of the speech synthesis model are adjusted in a direction of reducing the difference between the Mel spectrum features of the training speech and the target spectrum features.

[0118] The above-mentioned various units in the training device of the speech synthesis model provided in the embodiment of the present application can realize the above-mentioned Figure 1 The method steps of the embodiment of the training method of the speech synthesis model shown in FIG. For example, step S110 can be performed by Figure 8 The acquisition unit 801 in the training device shown in FIG. 1 is executed, and step S120 can be performed by Figure 8 The first extraction unit 802 in the training device shown in FIG. 1 is executed, and step S130 can be performed by Figure 8 The second extraction unit 803 in the training device shown in FIG. 1 is used to perform step S140. Figure 8 The training is performed by the model training unit 804 in the training device shown.

[0119] According to another embodiment of the present application, Figure 8 The various units in the training device shown can be individually or all combined into one or several other units to form a whole, or one (or more) of the units can be further divided into multiple functionally smaller units to form a whole, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, other units can also be included based on the training device. In actual applications, these functions can also be implemented with the assistance of other units and can be implemented by the collaboration of multiple units.

[0120] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM) and other processing elements and storage elements. Figure 1 A computer program (including program code) for each step of the corresponding method shown in FIG. Figure 8 The computer program can be recorded on a computer-readable storage medium, for example, and loaded into an electronic device through the computer-readable storage medium and run therein.

[0121] The speech synthesis model trained by the training device of the speech synthesis model in the embodiment of the present application includes a phoneme processing module and a text processing module. Not only can the phoneme processing module in the speech synthesis model be used to process the phoneme data corresponding to the training text to obtain phoneme features, but the text processing module in the speech synthesis model can also be used to process the training text to obtain text features. Since the text features also include prosodic features, the phoneme features and text features are fused through the text processing module to obtain richer prosodic features. Finally, the speech synthesis processing is performed based on the richer features obtained by fusion through the speech synthesis module. Since the trained speech synthesis model is not only based on phoneme features, but can also perform speech synthesis based on the prosodic features in the text data, the speech synthesis model trained using the speech synthesis model training method has a higher speech synthesis accuracy and can synthesize speech with better sound quality.

[0122] Based on the embodiment of the above-mentioned speech synthesis method, the present application proposes a speech synthesis device. Figure 9 , is a structural diagram of a speech synthesis device 900 provided by an embodiment of the present application. Figure 9 In a software implementation, the speech synthesis device 900 may include:

[0123] An acquisition unit 901 acquires phoneme data and text data corresponding to the text to be synthesized;

[0124] Extraction unit 902, inputs the phoneme data into the phoneme feature extraction module included in the phoneme processing module in the speech synthesis model, performs feature processing to obtain phoneme features, and inputs the text data into the text processing module in the speech synthesis module to perform feature extraction to obtain text features; wherein, the speech synthesis model is based on Figure 1 The speech synthesis model is trained using the training method shown;

[0125] The synthesis unit 903 is used to fuse the phoneme features and the text features through the text processing module, and perform speech synthesis processing based on the fused features through the speech synthesis module included in the phoneme processing module to obtain the synthesized speech corresponding to the text to be synthesized.

[0126] The above-mentioned units in the speech synthesis device provided in the embodiment of the present application can realize the above-mentioned Figure 6 The method steps provided in the embodiment of the speech synthesis method shown in FIG. For example, step S610 may be performed by Figure 9 The acquisition unit 901 in the speech synthesis device shown in FIG. 1 is executed, and step S620 can be performed by Figure 9 The extraction unit 902 in the speech synthesis device shown in FIG. 1 is executed, and step S630 can be performed by Figure 9The extraction unit 903 in the speech synthesis device shown is used for execution.

[0127] According to another embodiment of the present application, Figure 9 The various units in the training device shown can be individually or all combined into one or several other units to form a whole, or one (or more) of the units can be further divided into multiple functionally smaller units to form a whole, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, other units can also be included based on the training device. In actual applications, these functions can also be implemented with the assistance of other units and can be implemented by the collaboration of multiple units.

[0128] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM) and other processing elements and storage elements. Figure 1 A computer program (including program code) for each step of the corresponding method shown in FIG. Figure 9 The speech synthesis device shown in and the training method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into an electronic device through the computer-readable storage medium and run therein.

[0129] One or more embodiments provided by the present application, when performing speech synthesis operations on a text to be synthesized and phoneme data corresponding to the text to be synthesized, not only can the phoneme feature extraction module included in the phoneme processing module in the speech synthesis model be used to process the phoneme data corresponding to the text to be synthesized to obtain phoneme features, but the text processing module in the speech synthesis model included in the phoneme processing module can also be used to process the text to be synthesized to obtain text features, and then the phoneme features and text features are fused through the text processing module to obtain richer rhythmic features, and finally the speech synthesis module is used to perform speech synthesis processing based on the richer features obtained by the fusion. Since the synthesized speech is not only based on phoneme features, but can also be based on rhythmic information in text features, the synthesized speech is more accurate and has better sound quality.

[0130] Based on the above-mentioned speech synthesis model training method embodiment, the present application also provides an electronic device, see Figure 10 , is a structural diagram of an electronic device provided in an embodiment of the present application. Figure 10The electronic device shown may include a processor 1001 and a computer storage medium 1004 . The processor 1001 and the computer storage medium 1004 may be connected via a bus or other means.

[0131] The computer storage medium 1004 can be stored in the memory of the electronic device. The computer storage medium 1004 is used to store computer programs, and the processor 1001 is used to execute the computer programs stored in the computer storage medium 1004. The processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device. It is suitable for implementing one or more computer programs, specifically for loading and executing:

[0132] Obtaining a training sample data pair and a speech synthesis model, wherein the speech synthesis model includes a phoneme processing module and a text processing module, the phoneme processing module includes a phoneme feature extraction module and a speech synthesis module, and the training sample data pair includes a training text, phoneme data corresponding to the training text, and a synthesized speech label corresponding to the training text;

[0133] Performing feature extraction on the phoneme data by the phoneme feature extraction module in the phoneme processing module to obtain phoneme features, and performing feature extraction on the training text by the text processing module to obtain text features;

[0134] The text processing module fuses the phoneme features and the text features, and the speech synthesis module performs speech synthesis processing based on the fused features to obtain training speech corresponding to the training text;

[0135] The speech synthesis model is trained based on the training speech corresponding to the training text and the synthesized speech label corresponding to the training text.

[0136] In one embodiment, the text processing module includes a text vector extraction module and a feature fusion module. When the processor 1001 extracts features from the training text through the text processing module to obtain text features, the processor 1001 performs the following steps:

[0137] The text vector extraction module in the text processing module is used to perform feature extraction on the training text to obtain a first text feature; and a second text feature is determined based on the first text feature.

[0138] In one embodiment, the text processing module further includes a feature fusion module; when the processor 1001 fuses the phoneme feature and the text feature through the text processing module, the processor 1001 performs the following steps:

[0139] The feature fusion module fuses the phoneme feature, the first text feature, and the second text feature based on an attention mechanism.

[0140] In one embodiment, the speech synthesis module includes a variable information adapter, a mel-spectrogram decoder module, and a vocoder. When the speech synthesis module performs speech synthesis processing based on the fused features to obtain training speech corresponding to the training text, the processor 1001 executes the following steps:

[0141] Performing an addition operation on the fused feature and the phoneme feature, and inputting the addition operation result into the variable information adapter for processing to obtain a variable information sequence;

[0142] Processing the variable information sequence through the Mel spectrum decoder module to obtain Mel spectrum features of the training speech corresponding to the training text;

[0143] The vocoder performs speech synthesis processing on the Mel spectrum features of the training speech corresponding to the training text to obtain the training speech corresponding to the training text.

[0144] In one embodiment, the processor 1001 performs the following steps when training the speech synthesis model based on the training speech corresponding to the training text and the synthesized speech label corresponding to the training text:

[0145] Obtaining target spectrum features of the synthesized speech tag corresponding to the training text;

[0146] The model parameters of the speech synthesis model are adjusted in a direction of reducing the difference between the Mel spectrum features of the training speech and the target spectrum features.

[0147] In one or more embodiments provided herein, a speech synthesis model includes a phoneme processing module and a text processing module. The phoneme processing module in the speech synthesis model can be used not only to process the phoneme data corresponding to the training text to obtain phoneme features, but also to process the training text to obtain text features. Since the text features also include prosodic features, the text processing module can be used to fuse the phoneme features and text features to obtain richer prosodic features. Finally, the speech synthesis module is used to perform speech synthesis based on the richer features obtained through the fusion. Since the trained speech synthesis model is not only based on phoneme features, but can also perform speech synthesis based on the prosodic features in the text data, the speech synthesis model trained using this speech synthesis model training method has a higher speech synthesis accuracy and can synthesize speech with better sound quality.

[0148] Alternatively, the processor 1001 is further adapted to implement one or more computer programs, specifically adapted to load and execute:

[0149] Obtaining a text to be synthesized and phoneme data corresponding to the text to be synthesized;

[0150] The phoneme data is input into the phoneme feature extraction module included in the phoneme processing module of the speech synthesis model to perform feature processing to obtain phoneme features, and the text to be synthesized is input into the text processing module of the speech synthesis module to perform feature extraction to obtain text features; wherein, the speech synthesis model is based on Figure 1 The speech synthesis model is trained using the training method;

[0151] The phoneme features and the text features are fused by the text processing module, and the speech synthesis module included in the phoneme processing module performs speech synthesis processing based on the fused features to obtain the synthesized speech corresponding to the text to be synthesized.

[0152] One or more embodiments provided by the present application, when performing speech synthesis operations on the text to be synthesized and the phoneme data corresponding to the text to be synthesized, not only can the phoneme processing module in the speech synthesis model be used to process the phoneme data corresponding to the text to be synthesized to obtain phoneme features, but the text processing module in the speech synthesis model can also be used to process the text to be synthesized to obtain text features, and then the phoneme features and text features are fused through the text processing module to obtain richer rhythmic features, and finally the speech synthesis module is used to perform speech synthesis processing based on the richer features obtained by fusion. Since the synthesized speech is not only based on phoneme features, but can also be based on rhythmic information in text features, the synthesized speech is more accurate and its sound quality is better.

[0153] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the various processes of the aforementioned embodiment of the speech synthesis method and achieves the same technical effects. To avoid repetition, the details are not described here. The computer-readable storage medium may be, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0154] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0158] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0159] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0160] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0161] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0162] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for training a speech synthesis model, characterized in that: include: Obtaining a training sample data pair and a speech synthesis model, wherein the speech synthesis model includes a phoneme processing module and a text processing module, the phoneme processing module includes a phoneme feature extraction module and a speech synthesis module, and the training sample data pair includes a training text, phoneme data corresponding to the training text, and a synthesized speech label corresponding to the training text; Performing feature extraction on the phoneme data using a phoneme feature extraction module in the phoneme processing module to obtain a phoneme feature, and performing feature extraction on the training text using a text vector extraction module in the text processing module to obtain a first text feature, and determining a second text feature based on the first text feature; The feature fusion module in the text processing module fuses the input phoneme feature, the first text feature, and the second text feature based on an attention mechanism, and performs speech synthesis processing based on the fused features by the speech synthesis module to obtain training speech corresponding to the training text; wherein the first text feature and the second text feature are the same; The speech synthesis model is trained based on the training speech corresponding to the training text and the synthesized speech label corresponding to the training text.

2. The method according to claim 1, wherein The speech synthesis module includes a variable information adapter, a mel spectrum decoder module and a vocoder. The speech synthesis module performs speech synthesis processing based on the fused features to obtain training speech corresponding to the training text, including: Performing an addition operation on the fused feature and the phoneme feature, and inputting the addition operation result into the variable information adapter for processing to obtain a variable information sequence; Processing the variable information sequence through the Mel spectrum decoder module to obtain Mel spectrum features of the training speech corresponding to the training text; The vocoder performs speech synthesis processing on the Mel spectrum features of the training speech corresponding to the training text to obtain the training speech corresponding to the training text.

3. The method according to claim 2, wherein The training of the speech synthesis model based on the training speech corresponding to the training text and the synthesized speech label corresponding to the training text includes: Obtaining target spectrum features of the synthesized speech tag corresponding to the training text; The model parameters of the speech synthesis model are adjusted in a direction of reducing the difference between the Mel spectrum features of the training speech and the target spectrum features.

4. A speech synthesis method, characterized in that: include: Obtaining a text to be synthesized and phoneme data corresponding to the text to be synthesized; Inputting the phoneme data into a phoneme feature extraction module included in a phoneme processing module in a speech synthesis model to perform feature processing to obtain phoneme features, and inputting the text to be synthesized into a text processing module in the speech synthesis module to perform feature extraction to obtain text features; wherein the speech synthesis model is trained based on the speech synthesis model training method according to any one of claims 1 to 3; The phoneme features and the text features are fused by the text processing module, and the speech synthesis module included in the phoneme processing module performs speech synthesis processing based on the fused features to obtain the synthesized speech corresponding to the text to be synthesized.

5. A training device for a speech synthesis model, characterized in that: include: an acquisition unit, configured to acquire a training sample data pair and a speech synthesis model, wherein the speech synthesis model includes a phoneme processing module and a text processing module, the phoneme processing module includes a phoneme feature extraction module and a speech synthesis module, and the training sample data pair includes a training text, phoneme data corresponding to the training text, and a synthesized speech label corresponding to the training text; a first extraction unit, configured to perform feature extraction on the phoneme data using a phoneme feature extraction module in the phoneme processing module to obtain a phoneme feature, and to perform feature extraction on the training text using a text vector extraction module in the text processing module to obtain a first text feature, and to determine a second text feature based on the first text feature; a second extraction unit, configured to utilize a feature fusion module in the text processing module to fuse the input phoneme feature, the first text feature, and the second text feature based on an attention mechanism, and to perform speech synthesis processing based on the fused feature by the speech synthesis module to obtain training speech corresponding to the training text; wherein the first text feature and the second text feature are the same; The model training unit is used to train the speech synthesis model based on the training speech corresponding to the training text and the synthesized speech label corresponding to the training text.

6. A speech synthesis device, characterized in that: include: An acquisition unit, configured to acquire a text to be synthesized and phoneme data corresponding to the text to be synthesized; An extraction unit, configured to input the phoneme data into a phoneme feature extraction module included in a phoneme processing module in a speech synthesis model to perform feature processing to obtain phoneme features, and input the text to be synthesized into a text processing module in the speech synthesis module to perform feature extraction to obtain text features; wherein the speech synthesis model is trained based on the speech synthesis model training method according to any one of claims 1 to 3; The synthesis unit fuses the phoneme features and the text features through the text processing module, and performs speech synthesis processing based on the fused features through the speech synthesis module included in the phoneme processing module to obtain the synthesized speech corresponding to the text to be synthesized.

7. An electronic device, characterized in that: include: A computer storage medium, a processor, and a computer program stored on the computer storage medium and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method according to any one of claims 1 to 3; or, when executed by the processor, implements the steps of the method according to claim 4.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 3; or, when the computer program is executed by a processor, the computer program implements the steps of the method according to claim 4.

Images