Emotional speech synthesis model training method and device, equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a training method, device and equipment of an emotional speech synthesis model and a storage medium, which can be applied to intelligent speech dialogue scenes of businesses such as finance, insurance and medical treatment. According to the invention, at least one layer of the pre-trained speech synthesis model is selected as the target layer, the VB-LoRA fine tuning module is loaded for the target layer, and fine tuning is carried out by using the emotional speech data, so that the model realizes emotional speech synthesis and output, the model does not add emotional information in the training stage, and only the emotional information is added in the fine tuning process. Therefore, fine adjustment can be carried out by adding emotion information of different emotion categories, expression ability of different emotion categories is given to the model, and expandability and flexibility of the model are enhanced; moreover, during fine tuning, only the parameters of the target layer loaded with the VB-LoRA fine tuning module are adjusted, full-parameter fine tuning does not need to be performed on the whole model, the workload of fine tuning of the model is reduced, and the calculation cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method, device, equipment and storage medium for an emotional speech synthesis model. Background Art

[0002] With the rapid development of artificial intelligence technology, speech synthesis technology has made significant progress and can produce high-quality speech that is almost indistinguishable from real human voices. For example, VITS (Variational Inference Transformer) uses complex neural network structures and training methods to make synthesized speech extremely similar to natural speech. In this field, adding emotional expression to speech is a focus of continuous attention. The integration of emotions not only enhances the realism of speech synthesis, but also has significant value in many practical applications such as movies, games, and dubbing.

[0003] Currently, most solutions for emotional text-to-speech synthesis directly fine-tune the speech synthesis algorithm model in whole or in part. The practice of adding emotional information during the training phase limits the use of a fixed number of emotional categories during the inference phase, which leads to insufficient scalability and flexibility. In addition, fine-tuning all parameters of large models requires a lot of computational costs. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a training method, device, equipment and storage medium for an emotional speech synthesis model to solve the problems that the existing speech synthesis model lacks scalability and flexibility, and fine-tuning requires a large amount of computing cost.

[0005] In a first aspect, a training method for an emotional speech synthesis model is provided, comprising: obtaining a pre-trained speech synthesis model; selecting at least one layer of the pre-trained speech synthesis model as a target layer, and loading a VB-LoRA fine-tuning module for the target layer; obtaining emotional speech data, and fine-tuning the parameters of the target layer based on the emotional speech data to obtain a trained emotional speech synthesis model, wherein the emotional speech data comprises emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond to each other one by one.

[0006] In a second aspect, a training device for an emotional speech synthesis model is provided, comprising:

[0007] An acquisition module is used to obtain a pre-trained speech synthesis model;

[0008] A loading module, used to select at least one layer of the pre-trained speech synthesis model as a target layer, and load the VB-LoRA fine-tuning module for the target layer;

[0009] The fine-tuning module is used to obtain emotional speech data and perform VB-LoRA fine-tuning on the parameters of the target layer based on the emotional speech data to obtain a trained emotional speech synthesis model, wherein the emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond one to one.

[0010] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the training method of the above-mentioned emotional speech synthesis model when executing the computer program.

[0011] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the training method of the above-mentioned emotional speech synthesis model are implemented.

[0012] The training method, device, equipment and storage medium of the above-mentioned emotional speech synthesis model select at least one layer of the pre-trained speech synthesis model as the target layer, load the VB-LoRA fine-tuning module for the target layer, and then use the emotional speech data for fine-tuning to obtain a trained emotional speech synthesis model, so that the pre-trained speech synthesis model that originally did not have the ability to express emotions can achieve emotional speech synthesis and output. The model does not add emotional information during the training stage, but only adds emotional information during the fine-tuning process, so that fine-tuning can be performed by adding emotional information of different emotional categories, giving the model the ability to express different emotional categories, thereby enhancing the scalability and flexibility of the model; moreover, when fine-tuning the pre-trained speech synthesis model, only the parameters of the target layer loaded with the VB-LoRA fine-tuning module are adjusted, and there is no need to fine-tune all parameters of the entire model, thereby reducing the workload of model fine-tuning and reducing the computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0014] Figure 1 is an exemplary system architecture diagram in which the present invention can be applied;

[0015] Figure 2 is a flow chart of a method for training an emotional speech synthesis model in one embodiment of the present invention;

[0016] Figure 3 is a schematic diagram of a framework of a pre-trained speech synthesis model in one embodiment of the present invention;

[0017] Figure 4 is a schematic diagram of the structure of the target layer after the VB-LoRA fine-tuning module is loaded in one embodiment of the present invention;

[0018] Figure 5 is another flow chart of a method for training an emotional speech synthesis model in one embodiment of the present invention;

[0019] Figure 6 is a structural schematic diagram of a training device for an emotional speech synthesis model in one embodiment of the present invention;

[0020] Figure 7 is another structural schematic diagram of a training device for an emotional speech synthesis model in one embodiment of the present invention;

[0021] Figure 8 It is a structural diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] like Figure 1 As shown, the system architecture includes a client, a network, and a server. The network is used to provide a medium for a communication link between the client and the server, that is, the client can communicate with the server through the network. The network includes various connection types, such as wired, wireless communication links or fiber optic cables, etc. Users can use the client to interact with the server through the network to receive or send messages, etc. Various communication client applications can be installed on the client, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc. The server can be a server that provides various services, such as a background server that supports the pages displayed on the client.

[0024] The embodiments of the present application provide a training method, apparatus, device and storage medium for an emotional speech synthesis model. Among them, the training method and apparatus for the emotional speech synthesis model can be applied to a server. The client can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0025] The present invention can be applied to intelligent voice dialogue scenarios in finance, insurance, medical and other businesses, wherein the emotional speech synthesis model trained by the present invention is embedded in the intelligent speech engine. During the dialogue, the server receives the user's voice question and answer data through the client, and the intelligent speech engine processes the user's voice question and answer data to generate a corresponding reply text. The reply text is input into the emotional speech synthesis model to realize emotional speech synthesis and output. Finally, the emotional speech is output to the user through the client, thereby improving the quality of the intelligent voice dialogue.

[0026] Figure 2 A flow chart of a method for training an emotional speech synthesis model provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the training method of the emotional speech synthesis model includes steps S10 to S30:

[0027] S10: Obtain a pre-trained speech synthesis model.

[0028] The pre-trained speech synthesis model is a pre-trained speech synthesis model. In this embodiment, the pre-trained speech synthesis model is a pre-trained VITS2 model; of course, in other embodiments of the present invention, the pre-trained speech synthesis model may also be a pre-trained VITS model, a Grad-TTS model (a text-to-speech model based on a diffusion probability model) or other speech synthesis models.

[0029] After completing the pre-training of the speech synthesis model to be trained, the obtained pre-trained speech synthesis model can be saved on the server or client. When fine-tuning the pre-trained speech synthesis model using emotional speech data, it is necessary to first retrieve the pre-trained speech synthesis model from the server or client, that is, to obtain the pre-trained speech synthesis model.

[0030] S20: Select at least one layer of the pre-trained speech synthesis model as a target layer, and load the VB-LoRA fine-tuning module for the target layer.

[0031] The pre-trained speech synthesis model obtained in step S10 can be used directly, that is, the pre-trained speech synthesis model can synthesize speech information. It is a general speech synthesis model that can only perform neutral speech synthesis output and does not have the ability to express emotions. Therefore, in order to achieve emotional speech synthesis and output, further training and processing are required.

[0032] In order to give the model the ability to express emotions, the present invention loads the VB-LoRA fine-tuning module onto at least one layer of a pre-trained speech synthesis model, and uses emotional speech data for fine-tuning to obtain a trained emotional speech synthesis model. The trained emotional speech synthesis model can realize emotional speech synthesis and output.

[0033] In this embodiment, the pre-trained speech synthesis model is a pre-trained VITS2 model, and its framework is as follows: Figure 3 As shown. The pre-trained speech synthesis model includes a text encoder, a random duration predictor, a projection layer, a flow layer and a decoder, wherein the text encoder is responsible for extracting key features from the text, and then the random duration predictor uses these key features to predict the duration of the speech, the projection layer is responsible for defining the sampling strategy, the flow layer converts the sampled data into acoustic features, and the decoder converts these acoustic features into the final speech signal. Through analysis, we can conclude that the text encoder and projection layer are mainly involved in the front-end modeling work, the flow layer and the decoder undertake the back-end modeling tasks, and at the same time, the random duration predictor is responsible for handling the feature alignment problem.

[0034] The selecting at least one layer of the pre-trained speech synthesis model as the target layer includes: selecting any one or any combination from a text encoder, a random duration predictor, a projection layer, a stream layer and a decoder as the target layer.

[0035] Loading the VB-LoRA fine-tuning module for the target layer means integrating the VB-LoRA fine-tuning module into the target layer selected in the pre-trained speech synthesis model. Specifically, by adding two low-rank matrices A and B to the target layer, and coordinating with the original fixed parameter W, the input x of the target layer is processed to generate a better output h, thereby improving the performance and adaptability of the model. Figure 4 As shown, the target layer can be expressed as: h=Wx+BAx after loading the VB-LoRA fine-tuning module, wherein h is the output of the target layer, x is the input of the target layer, W is the initial weight of the target layer, and B and A are two low-rank matrices obtained by the VB-LoRA fine-tuning module by disassembling the weight change of the target layer.

[0036] S30: Acquire emotional speech data, and fine-tune the parameters of the target layer based on the emotional speech data to obtain a trained emotional speech synthesis model, wherein the emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond to each other one by one.

[0037] When fine-tuning the pre-trained speech synthesis model, it is necessary to obtain emotional speech data, and then use the emotional speech data to train the pre-trained speech synthesis model, adjust and optimize the parameters of the target layer, and then obtain the emotional speech synthesis model when the training is completed. The emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond to each other.

[0038] During specific implementation, the emotional speech can be directly recorded by the speaker or sampled from an existing corpus. The ESD database is an emotional speech database for sound conversion research, which is suitable for multi-speaker and cross-language emotional sound conversion research. The ESD database contains 350 parallel utterances, pronounced by 10 English native speakers and 10 Chinese native speakers, covering 5 emotion categories (neutral, happy, angry, sad and surprised), and recorded for more than 29 hours in a controlled acoustic environment. In this embodiment, all speech is resampled to 16kHz from the English data of the ESD database to obtain the emotional speech, and then the emotional speech is annotated with speech content to obtain emotional text, the emotional speech is annotated with emotional embedding to obtain emotional embedding data, and the emotional speech is annotated with emotional classification to obtain emotional category data. The emotional text, emotional embedding data, and emotional category data obtained after annotation correspond to the emotional speech.

[0039] In step S30, fine-tuning the parameters of the target layer based on the emotional speech data specifically includes the following steps S31-S34:

[0040] S31: Freeze the parameters of the pre-trained speech synthesis model and initialize the parameters of the VB-LoRA fine-tuning module.

[0041] The target layer is expressed as: h=Wx+BAx after loading the VB-LoRA fine-tuning module. For step S31, the parameters of the pre-trained speech synthesis model are frozen, that is, the parameter W is fixed, and only two low-rank matrices A and B are trained during training. The parameters of the VB-LoRA fine-tuning module are initialized, that is, the two low-rank matrices A and B are initialized, wherein the low-rank matrix A can be initialized by a random Gaussian distribution, and the low-rank matrix B can be initialized by a 0 matrix.

[0042] S32: Perform forward propagation calculation on the target layer based on the emotional speech data to obtain the output of the target layer.

[0043] The target layer is expressed as: h=Wx+BAx after loading the VB-LoRA fine-tuning module. Taking the emotional text in the emotional speech data as input x, the output of the target layer is h=Wx+BAx. The output h of the target layer is the predicted emotional speech, predicted emotional embedding data, and predicted emotional category data obtained by inputting a certain sample (emotional text) into the target layer.

[0044] S33: Calculate model loss based on the output of the target layer.

[0045] The output of the target layer includes predicted emotional speech, predicted emotional embedding data, and predicted emotional category data. For step S33, the model loss can be calculated through the following steps S331-S334:

[0046] S331: Calculate the reconstruction loss according to the predicted emotional speech of the i-th sample in the emotional speech data and the emotional speech in the corresponding sample.

[0047] Specifically, using the formula Calculate the reconstruction loss L recon , where N is the number of samples in the emotional speech data, predicted i and target i are the predicted emotional speech of the i-th sample and the emotional speech in the corresponding sample (that is, the real emotional speech).

[0048] S332: Calculate the emotion embedding loss according to the predicted emotion embedding data of the i-th sample in the emotional speech data and the emotion embedding data in the corresponding sample.

[0049] Specifically, using the formula Calculate the sentiment embedding loss L emotion , where predicted_emotion i and target_emotion i They are respectively the predicted emotion embedding data of the i-th sample and the emotion embedding data in the corresponding sample (that is, the real emotion embedding data).

[0050] S333: Calculate the emotion classification loss according to the predicted emotion category data of the i-th sample in the emotional speech data and the emotion category data in the corresponding sample.

[0051] Specifically, using the formula Calculate the sentiment classification loss L classification , where M is the number of sentiment categories and y icis an indicator variable. If the true category of the i-th sample is c, then y ic =1, otherwise 0, is the probability that the model predicts that the i-th sample belongs to category c.

[0052] S334: Calculate model loss based on reconstruction loss, sentiment embedding loss and sentiment classification loss.

[0053] Specifically, using the formula L total =w1*L recon +w2*L emotion +w3*L classification Calculate the model loss L total , where w1, w2, and w3 are the weights of reconstruction loss, sentiment embedding loss, and sentiment classification loss, respectively.

[0054] S34: Back-propagation is performed according to the model loss to update the parameters of the VB-LoRA fine-tuning module.

[0055] In the fine-tuning process, since the parameters of the pre-trained speech synthesis model are frozen, in the fine-tuning training process, back propagation only updates the parameters of the VB-LoRA fine-tuning module, that is, updates the low-rank matrix A and the low-rank matrix B, and the parameters W of the pre-trained speech synthesis model remain unchanged. When the model loss is less than the loss threshold, the training is stopped to obtain a trained emotional speech synthesis model.

[0056] The training method of the emotional speech synthesis model in the embodiment of the present invention can be applied to intelligent voice dialogue scenarios in finance, insurance, medical and other businesses, wherein the emotional speech synthesis model trained by the training method of the emotional speech synthesis model is embedded in the intelligent voice engine. During the conversation, the server receives the user's voice question and answer data through the client, and the intelligent voice engine processes the user's voice question and answer data to generate a corresponding reply text. The reply text is input into the emotional speech synthesis model to realize emotional speech synthesis and output. Finally, the emotional speech is output to the user through the client, thereby improving the quality of the intelligent voice dialogue.

[0057] The training method of the emotional speech synthesis model provided in the embodiment of the present invention selects at least one layer of the pre-trained speech synthesis model as the target layer, loads the VB-LoRA fine-tuning module for the target layer, and then uses the emotional speech data for fine-tuning to obtain a trained emotional speech synthesis model, so that the pre-trained speech synthesis model that originally did not have the ability to express emotions can achieve emotional speech synthesis and output, promotes the development of adaptable emotional speech synthesis systems, and contributes to artificial intelligence content generation. The model does not add emotional information during the training stage, but only adds emotional information during the fine-tuning process, so that fine-tuning can be performed by adding emotional information of different emotional categories, giving the model the ability to express different emotional categories, thereby enhancing the scalability and flexibility of the model; moreover, when fine-tuning the pre-trained speech synthesis model, only the parameters of the target layer loaded with the VB-LoRA fine-tuning module are adjusted, and there is no need to fine-tune all parameters of the entire model, thereby reducing the workload of model fine-tuning and reducing the computational cost.

[0058] Figure 5 Another flow chart of the training method of the emotional speech synthesis model provided by the embodiment of the present invention. Figure 5 As shown, the training method of the emotional speech synthesis model includes steps S01 to S30, wherein steps S10 to S30 are Figure 2 Steps S10 to S30 in the training method of the emotional speech synthesis model in the illustrated embodiment are the same and will not be described in detail here.

[0059] Figure 5 The training method of the emotional speech synthesis model in the embodiment shown is Figure 2 Compared with the training method of the emotional speech synthesis model in the illustrated embodiment, the difference is that step S01 and step S02 are included before step S10.

[0060] S01: Load the speech synthesis model to be trained and obtain model training data.

[0061] S02: Based on the model training data, the speech synthesis model to be trained is trained to obtain a pre-trained speech synthesis model.

[0062] Step S01 and step S02 are mainly used to pre-train the speech synthesis model. When pre-training the speech synthesis model, first load the speech synthesis model that needs to be pre-trained, that is, load the speech synthesis model to be trained, and at the same time obtain the model training data for model pre-training, and then pre-train the loaded speech synthesis model to be trained according to the obtained model training data. Among them, the model training data includes pre-trained speech and pre-trained text, and each pre-trained text has a corresponding and unique pre-trained speech.

[0063] In specific implementation, the pre-trained speech can be obtained by directly recording the speaker or sampling an existing corpus. The VCTK corpus contains about 44 hours of speech data, which are spoken by 110 native English speakers with different accents. Each speaker reads about 400 sentences, which are derived from newspaper articles, rainbow paragraphs, and induced paragraphs for speech accent archives. In this embodiment, all speech data in the VCTK corpus are resampled to 16kHz to obtain the pre-trained speech, and then the pre-trained speech is content-annotated to obtain pre-trained text information corresponding to the pre-trained speech.

[0064] In actual applications, when performing speech synthesis output, it is necessary to input corresponding text information, and then use the trained speech synthesis model to process the text information accordingly to output the corresponding speech information. In this embodiment, the training speech synthesis model is pre-trained according to the acquired model training data to obtain a pre-trained speech synthesis model. At this time, the pre-trained speech synthesis model itself can be used directly, that is, the pre-trained speech synthesis model can synthesize speech information and is a general speech synthesis model, but it can only perform neutral speech synthesis output and does not have the ability to express emotions.

[0065] The training method of the emotional speech synthesis model in the embodiment of the present invention can be applied to intelligent voice dialogue scenarios in finance, insurance, medical and other businesses, wherein the emotional speech synthesis model trained by the training method of the emotional speech synthesis model is embedded in the intelligent voice engine. During the conversation, the server receives the user's voice question and answer data through the client, and the intelligent voice engine processes the user's voice question and answer data to generate a corresponding reply text. The reply text is input into the emotional speech synthesis model to realize emotional speech synthesis and output. Finally, the emotional speech is output to the user through the client, thereby improving the quality of the intelligent voice dialogue.

[0066] The training method of the emotional speech synthesis model provided in the embodiment of the present invention selects at least one layer of the pre-trained speech synthesis model as the target layer, loads the VB-LoRA fine-tuning module for the target layer, and then uses the emotional speech data for fine-tuning to obtain a trained emotional speech synthesis model, so that the pre-trained speech synthesis model that originally did not have the ability to express emotions can achieve emotional speech synthesis and output, promotes the development of adaptable emotional speech synthesis systems, and contributes to artificial intelligence content generation. The model does not add emotional information during the training stage, but only adds emotional information during the fine-tuning process, so that fine-tuning can be performed by adding emotional information of different emotional categories, giving the model the ability to express different emotional categories, thereby enhancing the scalability and flexibility of the model; moreover, when fine-tuning the pre-trained speech synthesis model, only the parameters of the target layer loaded with the VB-LoRA fine-tuning module are adjusted, and there is no need to fine-tune all parameters of the entire model, thereby reducing the workload of model fine-tuning and reducing the computational cost.

[0067] In one embodiment, a training device for an emotional speech synthesis model is provided. Figure 2 The training methods of the emotional speech synthesis models in the illustrated embodiments correspond one to one. Figure 6 As shown, the training device of the emotional speech synthesis model includes an acquisition module 10, a loading module 20 and a fine-tuning module. The functional modules are described in detail as follows:

[0068] Acquisition module 10, for executing Figure 2 Step S10 in the method for training an emotional speech synthesis model in the illustrated embodiment is used to obtain a pre-trained speech synthesis model.

[0069] Load module 20 for executing Figure 2 Step S20 in the training method of the emotional speech synthesis model in the illustrated embodiment is used to select at least one layer of the pre-trained speech synthesis model as a target layer and load the VB-LoRA fine-tuning module for the target layer.

[0070] The fine-tuning module 30 is used to perform Figure 2 Step S30 in the training method of the emotional speech synthesis model in the shown embodiment is used to obtain emotional speech data, and perform VB-LoRA fine-tuning on the parameters of the target layer based on the emotional speech data to obtain a trained emotional speech synthesis model, wherein the emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond one to one.

[0071] The training device for the emotional speech synthesis model in the embodiment of the present invention can be applied to intelligent voice dialogue scenarios in finance, insurance, medical and other businesses, wherein the emotional speech synthesis model trained by the training device for the emotional speech synthesis model is embedded in the intelligent voice engine. During the conversation, the server receives the user's voice question and answer data through the client, and the intelligent voice engine processes the user's voice question and answer data to generate a corresponding reply text. The reply text is input into the emotional speech synthesis model to realize emotional speech synthesis and output. Finally, the emotional speech is output to the user through the client, thereby improving the quality of the intelligent voice dialogue.

[0072] The training device of the emotional speech synthesis model provided in the embodiment of the present invention selects at least one layer of the pre-trained speech synthesis model as the target layer, loads the VB-LoRA fine-tuning module for the target layer, and then uses the emotional speech data for fine-tuning to obtain a trained emotional speech synthesis model, so that the pre-trained speech synthesis model that originally did not have the ability to express emotions can achieve emotional speech synthesis and output, promotes the development of adaptable emotional speech synthesis systems, and contributes to artificial intelligence content generation. The model does not add emotional information during the training stage, but only adds emotional information during the fine-tuning process, so that fine-tuning can be performed by adding emotional information of different emotional categories, giving the model the ability to express different emotional categories, thereby enhancing the scalability and flexibility of the model; moreover, when fine-tuning the pre-trained speech synthesis model, only the parameters of the target layer loaded with the VB-LoRA fine-tuning module are adjusted, and there is no need to fine-tune all parameters of the entire model, thereby reducing the workload of model fine-tuning and reducing the computational cost.

[0073] For the specific definition of the training device for the emotional speech synthesis model, please refer to the definition of the training method for the emotional speech synthesis model in the above text, which will not be repeated here. The various modules in the above-mentioned training device for the emotional speech synthesis model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0074] In one embodiment, a training device for an emotional speech synthesis model is provided. Figure 5 The training methods of the emotional speech synthesis models in the illustrated embodiments correspond one to one. Figure 7 As shown, the training device of the emotional speech synthesis model includes a model loading module 01, a pre-training module 02, an acquisition module 10, a loading module 20 and a fine-tuning module 30. The functional modules are described in detail as follows:

[0075] Model loading module 01, used to execute Figure 5Step S01 in the training method of the emotional speech synthesis model in the illustrated embodiment is used to load the speech synthesis model to be trained and obtain model training data.

[0076] Pre-training module 02, used to perform Figure 5 Step S02 in the training method of the emotional speech synthesis model in the illustrated embodiment is used to train the speech synthesis model to be trained based on the model training data to obtain a pre-trained speech synthesis model.

[0077] Acquisition module 10, for executing Figure 5 Step S10 in the method for training an emotional speech synthesis model in the illustrated embodiment is used to obtain a pre-trained speech synthesis model.

[0078] Load module 20 for executing Figure 5 Step S20 in the training method of the emotional speech synthesis model in the illustrated embodiment is used to select at least one layer of the pre-trained speech synthesis model as a target layer and load the VB-LoRA fine-tuning module for the target layer.

[0079] The fine-tuning module 30 is used to perform Figure 5 Step S30 in the training method of the emotional speech synthesis model in the shown embodiment is used to obtain emotional speech data, and perform VB-LoRA fine-tuning on the parameters of the target layer based on the emotional speech data to obtain a trained emotional speech synthesis model, wherein the emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond one to one.

[0080] The training device for the emotional speech synthesis model in the embodiment of the present invention can be applied to intelligent voice dialogue scenarios in finance, insurance, medical and other businesses, wherein the emotional speech synthesis model trained by the training device for the emotional speech synthesis model is embedded in the intelligent voice engine. During the conversation, the server receives the user's voice question and answer data through the client, and the intelligent voice engine processes the user's voice question and answer data to generate a corresponding reply text. The reply text is input into the emotional speech synthesis model to realize emotional speech synthesis and output. Finally, the emotional speech is output to the user through the client, thereby improving the quality of the intelligent voice dialogue.

[0081] The training device of the emotional speech synthesis model provided in the embodiment of the present invention selects at least one layer of the pre-trained speech synthesis model as the target layer, loads the VB-LoRA fine-tuning module for the target layer, and then uses the emotional speech data for fine-tuning to obtain a trained emotional speech synthesis model, so that the pre-trained speech synthesis model that originally did not have the ability to express emotions can achieve emotional speech synthesis and output, promotes the development of adaptable emotional speech synthesis systems, and contributes to artificial intelligence content generation. The model does not add emotional information during the training stage, but only adds emotional information during the fine-tuning process, so that fine-tuning can be performed by adding emotional information of different emotional categories, giving the model the ability to express different emotional categories, thereby enhancing the scalability and flexibility of the model; moreover, when fine-tuning the pre-trained speech synthesis model, only the parameters of the target layer loaded with the VB-LoRA fine-tuning module are adjusted, and there is no need to fine-tune all parameters of the entire model, thereby reducing the workload of model fine-tuning and reducing the computational cost.

[0082] In one embodiment, a computer device is provided, whose internal structure diagram can be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with other external electronic devices through a network connection. When the computer program is executed by the processor, it is implemented Figure 2 or Figure 5 Functions or steps of the method for training the emotional speech synthesis model in the illustrated embodiment.

[0083] In one embodiment, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, Figure 2 or Figure 5 Functions or steps of the method for training the emotional speech synthesis model in the illustrated embodiment.

[0084] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0085] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0086] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A training method for an emotional speech synthesis model, characterized in that: include: Get a pre-trained speech synthesis model; Select at least one layer of the pre-trained speech synthesis model as the target layer and load the VB-LoRA fine-tuning module for the target layer; Acquire emotional speech data, and fine-tune the parameters of the target layer based on the emotional speech data to obtain a trained emotional speech synthesis model, wherein the emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond one to one.

2. The training method of the emotional speech synthesis model as claimed in claim 1, characterized in that: After loading the VB-LoRA fine-tuning module, the target layer is expressed as: h=Wx+BAx, wherein h is the output of the target layer, x is the input of the target layer, W is the initial weight of the target layer, and B and A are two low-rank matrices obtained by the VB-LoRA fine-tuning module by disassembling the weight change of the target layer.

3. The training method of the emotional speech synthesis model as claimed in claim 2, characterized in that: The fine-tuning of the parameters of the target layer based on the emotional speech data comprises: Freeze the parameters of the pre-trained speech synthesis model and initialize the parameters of the VB-LoRA fine-tuning module; Perform forward propagation calculation on the target layer based on the emotional speech data to obtain the output of the target layer; Calculate the model loss based on the output of the target layer; Back propagation is performed based on the model loss to update the parameters of the VB-LoRA fine-tuning module.

4. The training method of the emotional speech synthesis model as claimed in claim 3, characterized in that: The output of the target layer includes predicted emotional speech, predicted emotional embedding data, and predicted emotional category data. The calculation of the model loss based on the output of the target layer includes: Calculating the reconstruction loss according to the predicted emotional speech of the i-th sample in the emotional speech data and the emotional speech in the corresponding sample; Calculating the emotion embedding loss according to the predicted emotion embedding data of the i-th sample in the emotion speech data and the emotion embedding data in the corresponding sample; Calculate the emotion classification loss according to the predicted emotion category data of the i-th sample in the emotion speech data and the emotion category data in the corresponding sample; The model loss is calculated based on the reconstruction loss, sentiment embedding loss, and sentiment classification loss.

5. The training method of the emotional speech synthesis model as claimed in claim 4, characterized in that: Using the formula Calculate the reconstruction loss L recon , where N is the number of samples in the emotional speech data, predicted i and target i are the predicted emotional speech of the i-th sample and the emotional speech in the corresponding sample; Using the formula Calculate the sentiment embedding loss L emotion , where predicted_emotion i and target_emotion i are the predicted sentiment embedding data of the i-th sample and the sentiment embedding data in the corresponding sample; Using the formula Calculate the sentiment classification loss L classification , where M is the number of sentiment categories and y ic is an indicator variable. If the true category of the i-th sample is c, then y ic =1, otherwise 0, is the probability that the model predicts that the i-th sample belongs to category c; Using the formula L total =w1*L recon +w2*L emotion +w3*L classification Calculate the model loss L total , where w1, w2, and w3 are the weights of reconstruction loss, sentiment embedding loss, and sentiment classification loss, respectively.

6. The training method of the emotional speech synthesis model according to claim 1, characterized in that: The pre-trained speech synthesis model includes a text encoder, a random duration predictor, a projection layer, a flow layer and a decoder. Selecting at least one layer of the pre-trained speech synthesis model as the target layer includes: selecting any one or any combination of the text encoder, random duration predictor, projection layer, flow layer and decoder as the target layer.

7. The training method of the emotional speech synthesis model as claimed in claim 1, characterized in that: Before obtaining the pre-trained speech synthesis model, the method includes: Load the speech synthesis model to be trained and obtain the model training data; Based on the model training data, the speech synthesis model to be trained is trained to obtain a pre-trained speech synthesis model.

8. A training device for an emotional speech synthesis model, characterized in that: include: An acquisition module is used to obtain a pre-trained speech synthesis model; A loading module, used to select at least one layer of the pre-trained speech synthesis model as a target layer, and load the VB-LoRA fine-tuning module for the target layer; The fine-tuning module is used to obtain emotional speech data and perform VB-LoRA fine-tuning on the parameters of the target layer based on the emotional speech data to obtain a trained emotional speech synthesis model, wherein the emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond one to one.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the training method of the emotional speech synthesis model as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the training method of the emotional speech synthesis model as described in any one of claims 1 to 7 are implemented.

Images