A speech synthesis method and device, electronic equipment and storage medium

Abstract

The application provides a speech synthesis method and device, electronic equipment and storage medium, wherein the speech synthesis method comprises: processing emotion sample data by using an emotion classification model to obtain an emotion feature vector with a first emotion label; training a speech synthesis model by using the emotion feature vector and synthesis model training data with a second emotion label; and performing a speech synthesis task by using the trained speech synthesis model. The speech synthesis model uses the emotion feature vector obtained by processing the emotion sample data by using the emotion classification model to perform a speech synthesis task of superimposed emotion, thereby improving the humanization effect of the synthesized speech of the speech synthesis method.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and more specifically, to a speech synthesis method, apparatus, electronic device, and storage medium. Background Technology

[0002] Human-like TTS speech synthesis has always been a very challenging area in speech synthesis tasks. Human-like TTS not only requires the synthesized speech to be full and clear in tone, but also requires the synthesized speech to accurately reflect the speaker's emotions.

[0003] Currently, humanoid TTS speech synthesis tasks typically employ corpus-based annotation methods when adding emotions. This approach sets tone tags in the text corpus to simulate human tone during speech synthesis. However, this method requires predicting the tone represented by each phoneme individually, demanding high prediction accuracy. Prediction errors can affect the tone judgment of the entire sentence, resulting in poor humanoid quality in the synthesized speech. Summary of the Invention

[0004] The purpose of this application is to provide a speech synthesis method, apparatus, electronic device, and storage medium to improve the anthropomorphic effect of synthesized speech.

[0005] In a first aspect, embodiments of this application provide a speech synthesis method, comprising: processing emotion sample data using an emotion classification model to obtain an emotion feature vector with a first emotion label; training a speech synthesis model using the emotion feature vector and synthesis model training data with a second emotion label; and performing a speech synthesis task using the trained speech synthesis model.

[0006] In the implementation of the above scheme, the speech synthesis model uses the emotion feature vector obtained by processing the emotion sample data through the emotion classification model to superimpose the emotion into the speech synthesis task, which makes the human-like effect of the synthesized speech produced by the above speech synthesis method better.

[0007] In one implementation of the first aspect, an emotion classification model is used to process the emotion sample data to obtain an emotion feature vector with a first emotion label, including: obtaining a first Mel spectrum of the emotion sample data; obtaining an emotion feature vector of the emotion sample data based on the first Mel spectrum; classifying the emotion sample data based on the emotion feature vector to obtain a first emotion label of the emotion sample data.

[0008] In the implementation of the above scheme, the emotion classification model extracts the emotion feature vectors from the emotion sample data and classifies the emotion sample data according to the emotion feature vectors to obtain the corresponding emotion labels. This allows the corresponding emotion feature vectors to be obtained simply by querying the corresponding emotion labels when performing speech synthesis tasks, which greatly improves the speech synthesis efficiency of the above speech synthesis method. At the same time, the emotion classification model extracts emotion feature vectors with emotion labels and uses the emotion feature vectors to train the speech synthesis model and perform speech synthesis tasks, which improves the quality of the speech synthesized by the above speech synthesis method and makes the human-like effect of the synthesized speech better.

[0009] In one implementation of the first aspect, obtaining the emotion feature vector of the emotion sample data based on the first Mel spectrum includes: obtaining the emotion feature vector of the emotion sample data using a Long Short-Term Memory (LSTM) network based on the first Mel spectrum.

[0010] In the implementation of the above scheme, the Long Short-Term Memory (LSTM) network is used to extract the emotion feature vector of the emotion sample data. The extracted emotion feature vector can more accurately represent the emotion of the emotion sample data. The emotion feature vector is used to train the speech synthesis model and perform speech synthesis tasks, which improves the quality of the speech synthesized by the above speech synthesis method and makes the human-like effect of the speech synthesized by the above speech synthesis method better.

[0011] In one implementation of the first aspect, before processing the emotion sample data using the emotion classification model, the method further includes: pre-training the emotion classification model using classification model training data with a third emotion label.

[0012] In the implementation of the above scheme, the emotion classification model is pre-trained using training data with emotion labels, which avoids the need for large-scale annotation of the corpus when performing speech synthesis tasks, and greatly improves the speech synthesis efficiency of the above speech synthesis method.

[0013] In one implementation of the first aspect, after processing the emotion sample data using an emotion classification model to obtain emotion feature vectors with first emotion labels, the method further includes: classifying the emotion feature vectors according to the label content of the first emotion labels to obtain emotion feature vector sets for each type of first emotion label; fusing the emotion feature vectors in each of the emotion feature vector sets to obtain fused emotion feature vectors corresponding to each type of first emotion label; and training the speech synthesis model using the emotion feature vectors and synthesis model training data with second emotion labels, which includes: training the speech synthesis model using the fused emotion feature vectors and synthesis model training data with second emotion labels.

[0014] In the implementation of the above scheme, by fusing the emotion feature vectors corresponding to the first emotion tags of each type, a fused emotion feature that can more accurately represent the corresponding emotion type is obtained. The fused emotion feature is used to train the speech synthesis model, which improves the quality of the speech synthesized by the above speech synthesis method and makes the human-like effect of the speech synthesized by the above speech synthesis method better.

[0015] In one implementation of the first aspect, the step of fusing the emotional feature vectors in each of the emotional feature vector sets to obtain the fused emotional feature vector corresponding to each first emotion label type includes: fusing the emotional feature vectors in each of the emotional feature vector sets by averaging or weighted averaging to obtain the fused emotional feature vector corresponding to each first emotion label type.

[0016] In the implementation of the above scheme, the emotional feature vectors in each set of emotional feature vectors can be merged by averaging, or by weighted averaging, so that the above speech synthesis method can be applied to more application scenarios and improve the adaptability of the above speech synthesis method.

[0017] In one implementation of the first aspect, training the speech synthesis model using the emotion feature vector and synthesis model training data with a second emotion label includes: obtaining the second Mel spectrum of the audio data in the synthesis model training data; obtaining the intermediate vector of the speech to be synthesized based on the text data in the synthesis model training data; obtaining the emotion feature vector corresponding to the second emotion label based on the second emotion label in the synthesis model training data; embedding the emotion feature vector into the intermediate vector of the speech to be synthesized to obtain the speech feature vector to be synthesized; obtaining the third Mel spectrum of the speech to be synthesized based on the speech feature vector to be synthesized; calculating the loss function value of the speech synthesis model based on the second Mel spectrum and the third Mel spectrum; updating the parameters of the speech synthesis model based on the loss function value; and repeating the above steps iteratively until the iteration termination condition is reached.

[0018] In the implementation of the above scheme, the speech synthesis model is trained by using emotion feature vectors and synthesis model data. This allows the speech synthesis model to better embed emotion feature vectors into the intermediate vector of the speech to be synthesized, thereby improving the quality of the speech synthesized by the above speech synthesis method and making the human-like effect of the speech synthesized by the above speech synthesis method better.

[0019] Secondly, embodiments of this application provide a speech synthesis device, comprising:

[0020] The emotion feature vector acquisition module is used to process emotion sample data using an emotion classification model to obtain emotion feature vectors with a first emotion label.

[0021] The speech synthesis model training module is used to train the speech synthesis model using the emotion feature vector and the synthesis model training data with the second emotion label.

[0022] The speech synthesis task execution module is used to perform speech synthesis tasks using the trained speech synthesis model.

[0023] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.

[0024] Fourthly, embodiments of this application provide an electronic device, including: a memory, a processor, and a communication bus, wherein the processor and the memory communicate with each other through the bus, the memory stores computer program instructions, and the computer program instructions are read and executed by the processor to perform the method provided in the first aspect or any possible implementation of the first aspect.

[0025] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A schematic flowchart illustrating the speech synthesis method provided in this application embodiment;

[0028] Figure 2 This is a schematic diagram of the structure of the emotion classification model provided in the embodiments of this application;

[0029] Figure 3 This is a schematic diagram of the structure of the speech synthesis device provided in the embodiments of this application;

[0030] Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of this application, and are therefore merely examples and should not be used to limit the scope of protection of this application.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0033] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0034] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0035] Please see Figure 1 This application provides a speech synthesis method, including:

[0036] Step S110: Process the emotion sample data using an emotion classification model to obtain the emotion feature vector with the first emotion label;

[0037] Step S120: Train the speech synthesis model using emotion feature vectors and synthesis model training data with second emotion labels;

[0038] Step S130: Use the trained speech synthesis model to perform the speech synthesis task.

[0039] In the implementation of the above scheme, the speech synthesis model uses the emotion feature vector obtained by processing the emotion sample data through the emotion classification model to superimpose the emotion into the speech synthesis task, which makes the human-like effect of the synthesized speech better. At the same time, since the emotion sample data is processed directly by the emotion classification model to obtain the emotion feature vector with the first emotion label, the speech synthesis efficiency of the above speech synthesis method is improved.

[0040] The following details step S110:

[0041] The emotion sample data referred to in step S110 can be speech data used to extract emotion feature vectors.

[0042] As an optional implementation of the above-mentioned speech synthesis method, step S110 uses an emotion classification model to process the emotion sample data and obtain an emotion feature vector with a first emotion label, including: obtaining the first Mel spectrum of the emotion sample data; obtaining the emotion feature vector of the emotion sample data based on the first Mel spectrum; classifying the emotion sample data based on the emotion feature vector to obtain the first emotion label of the emotion sample data. For example, this implementation involves: obtaining the spectrogram of the emotion sample data, then processing the spectrogram using a Mel filter bank to obtain the first Mel spectrum; inputting the first Mel spectrum into a neural network such as a deep neural network (DNN) or a recurrent neural network (RNN) to extract the emotion feature vector of the emotion sample data; using a classification sub-model, for example, composed of a feedforward neural network and a softmax activation function, classifying the emotion sample data based on the emotion feature vector, obtaining the emotion type corresponding to the emotion feature vector, and determining the obtained emotion type as the first emotion label corresponding to the emotion feature vector, thereby obtaining the first emotion label corresponding to the emotion sample data.

[0043] It is understandable that after extracting the emotion feature vector and the first emotion label from the emotion sample data, an emotion feature vector library can be constructed, which stores the emotion feature vector and the first emotion label corresponding to the emotion feature vector.

[0044] As an optional implementation of the above speech synthesis method, step S110 obtains the emotion feature vector of the emotion sample data based on the first Mel spectrum, including: obtaining the emotion feature vector of the emotion sample data by using a Long Short-Term Memory (LSTM) network based on the first Mel spectrum.

[0045] Long Short-Term Memory (LSTM) networks add forget gates, information addition gates, and output gates to RNNs to control the information flow. Using LSTM networks can extract emotional feature vectors from emotional sample data more accurately.

[0046] Please see Figure 2 This application provides an emotion classification model 200, comprising:

[0047] Data preprocessing module 210 is used to extract the 40-dimensional first Mel spectrum of the emotion sample data;

[0048] LSTM network 220 is used to extract the emotion feature vector of the emotion sample data based on the first Mel spectrum;

[0049] The feedforward neural network 230, activated by the softmax activation function, is used to classify emotional sample data based on emotional feature vectors and obtain the first emotional label of the emotional sample data.

[0050] The LSTM network 220 used in this embodiment has a three-layer structure, with each layer containing 128 basic units, for a total of 384 basic units.

[0051] As an optional implementation of the above-mentioned speech synthesis method, before processing the emotion sample data using the emotion classification model in step S110, the method further includes: pre-training the emotion classification model using training data of the classification model with a third emotion label. For example, this implementation involves: inputting the classification model training data into the emotion classification model, obtaining the classification result of the emotion classification model on the training data, calculating the loss function value of the emotion classification model based on the classification result and the third emotion label; repeating the above steps iteratively until an iteration termination condition is reached, wherein the iteration termination condition includes reaching a preset number of iterations or the loss function value of the emotion classification model reaching a preset threshold.

[0052] It should be noted that, based on different business needs, the emotion labels that the emotion classification model can classify need to be specified in advance. For example, in telephone follow-up business, the emotional needs of the follow-up robot include emotions such as "greeting", "calm" and "excited". Therefore, it is necessary to set corresponding labels according to the emotional needs of the follow-up robot so that the emotion classification model can obtain the emotional feature vector of the corresponding label.

[0053] In addition, the training data for the classification model can include speech data in WAV format, text data in TXT format corresponding to the speech data, and pre-specified emotion labels. The emotion labels can also be text data in TXT format.

[0054] As an optional implementation of the above-mentioned speech synthesis method, after processing the emotion sample data using an emotion classification model in step S110 to obtain emotion feature vectors with first emotion labels, the method further includes: classifying the emotion feature vectors according to the label content of the first emotion labels to obtain emotion feature vector sets for each type of first emotion label; and fusing the emotion feature vectors in each emotion feature vector set to obtain fused emotion feature vectors corresponding to each type of first emotion label. Step S120 uses emotion feature vectors and synthesis model training data with second emotion labels to train the speech synthesis model, including: using fused emotion feature vectors and synthesis model training data with second emotion labels to train the speech synthesis model. For example, after processing the emotion sample data using an emotion classification model, each first emotion label may correspond to multiple emotion feature vectors. In this case, multiple emotion feature vectors can be fused to obtain a fused emotion feature vector, and then the fused emotion feature vector and synthesis model training data with second emotion labels can be used to train the speech synthesis model.

[0055] As an optional implementation of the aforementioned speech synthesis method, fusing emotional feature vectors from various emotional feature vector sets to obtain fused emotional feature vectors corresponding to each first emotion label type includes: fusing emotional feature vectors from various emotional feature vector sets by averaging or weighted averaging to obtain fused emotional feature vectors corresponding to each first emotion label type. For example, this implementation may use averaging or weighted averaging to fuse emotional feature vectors from various emotional feature vector sets. If weighted averaging is used, the weight of each speech can be determined by pre-assigning weights to the speech data in the emotional sample data, or by calculating the confidence level of the emotional sample data classification when classifying the emotional sample data, and using the classification confidence level as the weight of the emotional sample data.

[0056] Step S120 is described in detail below:

[0057] As an optional implementation of the above-mentioned speech synthesis method, step S120 uses emotion feature vectors and synthesis model training data with second emotion labels to train the speech synthesis model, including: obtaining the second Mel spectrum of the audio data in the synthesis model training data; obtaining the intermediate vector of the speech to be synthesized based on the text data in the synthesis model training data; obtaining the emotion feature vector corresponding to the second emotion label based on the second emotion label in the synthesis model training data; embedding the emotion feature vector into the intermediate vector of the speech to be synthesized to obtain the feature vector of the speech to be synthesized; obtaining the third Mel spectrum of the speech to be synthesized based on the feature vector of the speech to be synthesized; calculating the loss function value of the speech synthesis model based on the second Mel spectrum and the third Mel spectrum, and updating the parameters of the speech synthesis model based on the loss function value; repeating the above steps iteratively until the iteration termination condition is reached. This implementation method includes, for example: the synthesis model training data includes audio data and corresponding text data, with a second emotion label in the text data; obtaining the second Mel spectrum of the audio data in the synthesis model training data; using a text encoder, obtaining an intermediate vector of the speech to be synthesized based on the text data in the synthesis model training data; obtaining an emotion feature vector corresponding to the second emotion label based on the second emotion label in the synthesis model training dataset; then, embedding the emotion feature vector into the intermediate vector of the speech to be synthesized to obtain a feature vector of the speech to be synthesized, the method of embedding the emotion feature vector into the intermediate vector of the speech to be synthesized can refer to existing technology; using a decoder, obtaining the third Mel spectrum of the speech to be synthesized based on the feature vector of the speech to be synthesized; calculating the loss function value of the speech synthesis model based on the second Mel spectrum and the third Mel spectrum, the loss function value of the speech synthesis model can be used as the loss function of the speech synthesis model; repeating the above steps iteratively until the iteration termination condition is reached, wherein the iteration termination condition includes reaching a preset number of iterations or the loss function value reaching a preset threshold.

[0058] Please see Figure 3 This application provides a speech synthesis device 300, comprising:

[0059] The emotion feature vector acquisition module 310 is used to process the emotion sample data using an emotion classification model to obtain an emotion feature vector with a first emotion label.

[0060] The speech synthesis model training module 320 is used to train the speech synthesis model using the emotion feature vector and the synthesis model training data with the second emotion label.

[0061] The speech synthesis task execution module 330 is used to perform speech synthesis tasks using the trained speech synthesis model.

[0062] As an optional implementation of the above-mentioned speech synthesis device, the emotion feature vector acquisition module 310 includes:

[0063] The first Mel spectrum acquisition unit is used to acquire the first Mel spectrum of the emotion sample data.

[0064] An emotion feature vector extraction unit is used to obtain the emotion feature vector of the emotion sample data based on the first Mel spectrum.

[0065] The first emotion label acquisition unit is used to classify the emotion sample data according to the emotion feature vector and obtain the first emotion label of the emotion sample data.

[0066] As an optional implementation of the above-mentioned speech synthesis device, the emotion feature vector extraction unit obtains the emotion feature vector of the emotion sample data based on the first Mel spectrum, including: obtaining the emotion feature vector of the emotion sample data by using a Long Short-Term Memory (LSTM) network based on the first Mel spectrum.

[0067] As an optional embodiment of the above-mentioned speech synthesis device, the speech synthesis device 300 further includes:

[0068] The emotion classification model pre-training module is used to pre-train the emotion classification model using training data from the classification model with third emotion labels.

[0069] As an optional embodiment of the above-mentioned speech synthesis device, the speech synthesis device 300 further includes:

[0070] The emotion feature vector fusion module is used to classify the emotion feature vectors according to the tag content of the first emotion tag to obtain an emotion feature vector set for each type of first emotion tag; and to fuse the emotion feature vectors in each of the emotion feature vector sets to obtain the fused emotion feature vector corresponding to each type of first emotion tag.

[0071] At this time, the speech synthesis model training module 320 uses the fused emotion feature vector and the synthesis model training data with the second emotion label to train the speech synthesis model.

[0072] As an optional implementation of the above-mentioned speech synthesis device, the emotion feature vector fusion module fuses the emotion feature vectors in each of the emotion feature vector sets to obtain the fused emotion feature vector corresponding to each first emotion label type, including: fusing the emotion feature vectors in each of the emotion feature vector sets by averaging or weighted averaging to obtain the fused emotion feature vector corresponding to each first emotion label type.

[0073] As an optional implementation of the above-mentioned speech synthesis device, the speech synthesis model training module 320 includes:

[0074] The second Mel spectrum acquisition unit is used to acquire the second Mel spectrum of the audio data in the training data of the synthetic model.

[0075] The intermediate vector acquisition unit for speech to be synthesized is used to acquire the intermediate vector of speech to be synthesized based on the text data in the training data of the synthesis model.

[0076] The emotion feature vector query unit is used to obtain the emotion feature vector corresponding to the second emotion label based on the second emotion label carried in the training data of the synthetic model;

[0077] An emotion feature vector embedding unit is used to embed the emotion feature vector into the intermediate vector of the speech to be synthesized to obtain the speech feature vector to be synthesized.

[0078] The third Mel spectrum acquisition unit is used to acquire the third Mel spectrum of the speech to be synthesized based on the feature vector of the speech to be synthesized.

[0079] The parameter update unit is used to calculate the loss function value of the speech synthesis model based on the second Mel spectrum and the third Mel spectrum, and update the parameters of the speech synthesis model based on the loss function value.

[0080] The iterative training unit repeatedly calls the above modules until the iteration termination condition is met.

[0081] As an optional implementation of the above-mentioned speech synthesis device, the intermediate vector acquisition module for the speech to be synthesized can be a text encoder.

[0082] As an optional implementation of the aforementioned speech synthesis device, the third Mel spectrum acquisition unit can be a decoder.

[0083] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. (Refer to...) Figure 4 The electronic device 400 includes a processor 410, a memory 420, and a communication interface 430. These components are interconnected and communicate with each other via a communication bus 440 and / or other forms of connection mechanism (not shown).

[0084] The memory 420 includes one or more (only one is shown in the figure), which may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The processor 410 and other possible components may access the memory 420 to read and / or write data therein.

[0085] Processor 410 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The processor 410 can be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Network Processor (NP), or other conventional processors; it can also be a special-purpose processor, including a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0086] Communication interface 430 includes one or more (only one is shown in the figure) and can be used to communicate directly or indirectly with other devices to exchange data. For example, communication interface 430 can be an Ethernet interface; it can be a mobile communication network interface, such as an interface for 3G, 4G, or 5G networks; or it can be other types of interfaces with data transmission and reception functions.

[0087] One or more computer program instructions may be stored in the memory 420, and the processor 410 may read and run these computer program instructions to implement the speech synthesis method provided in the embodiments of this application and other desired functions.

[0088] Understandable. Figure 4The structure shown is for illustrative purposes only; the electronic device 400 may also include more than [other components]. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof. For example, electronic device 400 can be a single server (or other device with computing power), a combination of multiple servers, a cluster of a large number of servers, etc., and can be either a physical device or a virtual device.

[0089] This application also provides a computer-readable storage medium storing computer program instructions. These computer program instructions are read and executed by a computer's processor to perform the speech synthesis method provided in this application. For example, the computer-readable storage medium can be implemented as follows: Figure 4 The memory 420 in the electronic device 400.

[0090] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0091] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0093] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A speech synthesis method, characterized in that, include: An emotion classification model is used to process emotion sample data to obtain an emotion feature vector with a first emotion label. The emotion classification model includes a data preprocessing module, a Long Short-Term Memory (LSTM) network, and a feedforward neural network. The data preprocessing module extracts a 40-dimensional first Mel spectrum from the emotion sample data. The LSTM obtains the emotion feature vector based on the first Mel spectrum of the emotion sample data. The feedforward neural network classifies the emotion sample data based on the emotion feature vector to obtain the first emotion label of the emotion sample data. The speech synthesis model is trained using the aforementioned emotion feature vector and the synthesis model training data with the second emotion label. The trained speech synthesis model is used to perform the speech synthesis task. Before processing the emotion sample data using the emotion classification model, the following steps are also included: After processing the emotion sample data using an emotion classification model to obtain an emotion feature vector with a first emotion label, the process further includes: Based on the label content of the first emotion label, the emotion feature vector is classified to obtain an emotion feature vector set for each type of first emotion label; By fusing the emotional feature vectors from each of the aforementioned emotional feature vector sets, the fused emotional feature vectors corresponding to each type of first emotional label are obtained. The step of training the speech synthesis model using the emotion feature vector and the synthesis model training data with the second emotion label includes: The speech synthesis model is trained using the fused emotion feature vector and the synthesis model training data with the second emotion label.

2. The speech synthesis method according to claim 1, characterized in that, Before processing the emotion sample data using the emotion classification model, the following steps are also included: The emotion classification model is pre-trained using training data with a third emotion label.

3. The speech synthesis method according to claim 1, characterized in that, The process of fusing the emotion feature vectors from each of the emotion feature vector sets to obtain the fused emotion feature vector corresponding to each first emotion label type includes: By averaging or weighted averaging, the emotional feature vectors from each of the emotional feature vector sets are merged to obtain the fused emotional feature vector corresponding to each first emotional label type.

4. The speech synthesis method according to claim 1, characterized in that, The step of training the speech synthesis model using the emotion feature vector and the synthesis model training data with the second emotion label includes: Obtain the second Mel spectrum of the audio data in the training data of the synthetic model; Based on the text data in the training data of the synthesis model, obtain the intermediate vector of the speech to be synthesized; Based on the second emotion label carried in the training data of the synthetic model, obtain the emotion feature vector corresponding to the second emotion label; The emotion feature vector is embedded into the intermediate vector of the speech to be synthesized to obtain the speech feature vector to be synthesized. Based on the feature vector of the speech to be synthesized, obtain the third Mel spectrum of the speech to be synthesized; The loss function value of the speech synthesis model is calculated based on the second Mel spectrum and the third Mel spectrum, and the parameters of the speech synthesis model are updated based on the loss function value. Repeat the above steps until the iteration termination condition is met.

5. A speech synthesis device, characterized in that, include: An emotion feature vector acquisition module is used to process emotion sample data using an emotion classification model to obtain an emotion feature vector with a first emotion label. The emotion classification model includes a data preprocessing module, a Long Short-Term Memory (LSTM) network, and a feedforward neural network. The data preprocessing module extracts a 40-dimensional first Mel spectrum from the emotion sample data. The LSTM obtains the emotion feature vector based on the first Mel spectrum of the emotion sample data. The feedforward neural network classifies the emotion sample data based on the emotion feature vector to obtain the first emotion label of the emotion sample data. The speech synthesis model training module is used to train the speech synthesis model using the emotion feature vector and the synthesis model training data with the second emotion label. The speech synthesis task execution module is used to perform speech synthesis tasks using the trained speech synthesis model. The emotion feature vector fusion module is used to classify the emotion feature vectors according to the tag content of the first emotion tag to obtain an emotion feature vector set for each type of first emotion tag; and to fuse the emotion feature vectors in each of the emotion feature vector sets to obtain the fused emotion feature vector corresponding to each type of first emotion tag. The speech synthesis model training module uses the fused emotion feature vector and the synthesis model training data with the second emotion label to train the speech synthesis model.

6. An electronic device, characterized in that, include: A processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other via the communication bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1 to 4 by calling the program instructions.

7. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method as described in any one of claims 1 to 4.

Images