Method, device and equipment for training emotional speech synthesis model and storage medium

Abstract

This invention relates to the field of artificial intelligence technology and discloses a training method, apparatus, device, and storage medium for an emotional speech synthesis model, which can be applied to intelligent voice dialogue scenarios in finance, insurance, and medical fields. This invention selects at least one layer of a pre-trained speech synthesis model as the target layer, loads a VB-LoRA fine-tuning module onto the target layer, and then fine-tunes it using emotional speech data. This enables the model to achieve emotional speech synthesis and output. No emotional information is added during the training phase; emotional information is only added during fine-tuning. This allows for fine-tuning by adding emotional information of different emotion categories, giving the model the ability to express different emotion categories, thus enhancing the model's scalability and flexibility. Furthermore, during fine-tuning, only the parameters of the target layer with the VB-LoRA fine-tuning module are adjusted, eliminating the need for full parameter fine-tuning of the entire model, reducing the workload of model fine-tuning and lowering computational costs.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, device, and storage medium for an emotional speech synthesis model. Background Technology

[0002] With the rapid development of artificial intelligence technology, speech synthesis technology has made significant progress, capable of producing high-quality speech that is almost indistinguishable from real human voices. For example, VITS (Variational Inference Transformer), through complex neural network structures and training methods, makes synthesized speech extremely similar to natural speech. In this field, adding emotional expression to speech is a continuous focus of attention. The incorporation of emotion not only enhances the realism of speech synthesis but also has significant value in various practical applications such as film, games, and voice acting.

[0003] Currently, most solutions for emotional text-to-speech synthesis involve directly fine-tuning the entire or partial aspects of the speech synthesis algorithm model. The practice of adding emotional information during the training phase limits the use of only a fixed number of emotional categories during the inference phase, resulting in insufficient scalability and flexibility. Furthermore, fine-tuning all parameters of a large model requires significant computational costs. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a training method, apparatus, device and storage medium for an emotional speech synthesis model, so as to solve the problems of insufficient scalability and flexibility of existing speech synthesis models, and the large amount of computational cost required for fine-tuning.

[0005] Firstly, a training method for an emotional speech synthesis model is provided, comprising: acquiring a pre-trained speech synthesis model; selecting at least one layer of the pre-trained speech synthesis model as a target layer and loading a VB-LoRA fine-tuning module for the target layer; acquiring emotional speech data and fine-tuning the parameters of the target layer based on the emotional speech data to obtain a trained emotional speech synthesis model, wherein the emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond one-to-one.

[0006] Secondly, a training device for an emotional speech synthesis model is provided, comprising:

[0007] The acquisition module is used to acquire the pre-trained speech synthesis model;

[0008] The loading module is used to select at least one layer of the pre-trained speech synthesis model as the target layer and load the VB-LoRA fine-tuning module for the target layer;

[0009] The fine-tuning module is used to acquire emotional speech data and perform VB-LoRA fine-tuning on the parameters of the target layer based on the emotional speech data to obtain a trained emotional speech synthesis model. The emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and there is a one-to-one correspondence between emotional text, emotional speech, emotional embedding data, and emotional category data.

[0010] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described training method for the emotional speech synthesis model.

[0011] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, the steps of the above-described training method for the emotional speech synthesis model are implemented.

[0012] The training method, apparatus, device, and storage medium of the aforementioned emotional speech synthesis model select at least one layer of the pre-trained speech synthesis model as the target layer, load the VB-LoRA fine-tuning module onto the target layer, and then fine-tune it using emotional speech data to obtain a trained emotional speech synthesis model. This enables the pre-trained speech synthesis model, which originally lacked emotional expression capabilities, to achieve emotional speech synthesis and output. Emotional information is not added to the model during the training phase, but only during the fine-tuning process. Thus, by adding emotional information of different emotion categories, the model can be fine-tuned, giving it the ability to express different emotion categories, thereby enhancing the model's scalability and flexibility. Moreover, when fine-tuning the pre-trained speech synthesis model, only the parameters of the target layer with the VB-LoRA fine-tuning module are adjusted, without the need for full parameter fine-tuning of the entire model, reducing the workload of model fine-tuning and lowering computational costs. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is an exemplary system architecture diagram in which the present invention can be applied;

[0015] Figure 2 This is a flowchart illustrating a training method for an emotional speech synthesis model according to an embodiment of the present invention.

[0016] Figure 3 This is a schematic diagram of the framework of a pre-trained speech synthesis model in one embodiment of the present invention;

[0017] Figure 4 This is a schematic diagram of the target layer after loading the VB-LoRA fine-tuning module in one embodiment of the present invention;

[0018] Figure 5 This is another flowchart illustrating the training method of the emotional speech synthesis model in one embodiment of the present invention;

[0019] Figure 6 This is a schematic diagram of a training device for an emotional speech synthesis model according to an embodiment of the present invention.

[0020] Figure 7 This is another schematic diagram of the training device for the emotional speech synthesis model in one embodiment of the present invention;

[0021] Figure 8 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] like Figure 1 As shown, the system architecture includes a client, a network, and a server. The network serves as the medium providing a communication link between the client and the server; that is, the client can communicate with the server through the network. Networks include various connection types, such as wired, wireless communication links, or fiber optic cables. Users can use the client to interact with the server through the network to receive or send messages, etc. Various communication client applications can be installed on the client, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platforms, etc. The server can be a server that provides various services, such as a backend server that supports the pages displayed on the client.

[0024] This application provides a training method, apparatus, device, and storage medium for an emotional speech synthesis model. The training method and apparatus for the emotional speech synthesis model can be applied to a server. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention is described in detail below through specific embodiments. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0025] This invention can be applied to intelligent voice dialogue scenarios in finance, insurance, and healthcare. By embedding the emotional speech synthesis model trained using this invention into an intelligent speech engine, during dialogue, the server receives the user's voice question-and-answer data through the client. The intelligent speech engine processes the user's voice question-and-answer data to generate corresponding response text. The response text is then input into the emotional speech synthesis model to achieve emotional speech synthesis and output. Finally, the emotional speech is output to the user through the client, thus improving the quality of intelligent voice dialogue.

[0026] Figure 2 A flowchart illustrating the training method for the emotional speech synthesis model provided in this embodiment of the invention is shown below. Figure 2 As shown, the training method for the emotional speech synthesis model includes steps S10 to S30:

[0027] S10: Obtain the pre-trained speech synthesis model.

[0028] The pre-trained speech synthesis model is a pre-trained speech synthesis model. In this embodiment, the pre-trained speech synthesis model is a pre-trained VITS2 model; of course, in other embodiments of the present invention, the pre-trained speech synthesis model may also be a pre-trained VITS model, a Grad-TTS model (a text-to-speech model based on a diffusion probability model), or other speech synthesis models.

[0029] After the pre-training of the speech synthesis model to be trained is completed, the resulting pre-trained speech synthesis model can be stored on the server or client. When fine-tuning the pre-trained speech synthesis model using emotional speech data, the pre-trained speech synthesis model must first be retrieved from the server or client, that is, the pre-trained speech synthesis model must be obtained.

[0030] S20: Select at least one layer of the pre-trained speech synthesis model as the target layer, and load the VB-LoRA fine-tuning module for the target layer.

[0031] The pre-trained speech synthesis model obtained in step S10 can be used directly. That is to say, the pre-trained speech synthesis model can synthesize speech information and is a general speech synthesis model. It can only synthesize neutral speech and does not have the ability to express emotions. Therefore, in order to achieve emotional speech synthesis and output, further training and processing are required.

[0032] To endow the model with emotional expression capabilities, this invention loads the VB-LoRA fine-tuning module onto at least one layer of a pre-trained speech synthesis model and fine-tunes it using emotional speech data to obtain a trained emotional speech synthesis model. This trained emotional speech synthesis model can achieve emotional speech synthesis and output.

[0033] In this embodiment, the pre-trained speech synthesis model is a pre-trained VITS2 model, the framework of which is as follows: Figure 3 As shown, the pre-trained speech synthesis model includes a text encoder, a random duration predictor, a projection layer, a flow layer, and a decoder. The text encoder extracts key features from the text. The random duration predictor then uses these features to predict the duration of the speech. The projection layer defines the sampling strategy. The flow layer converts the sampled data into acoustic features, and the decoder converts these acoustic features into the final speech signal. Analysis reveals that the text encoder and projection layer primarily participate in the front-end modeling, while the flow layer and decoder handle the back-end modeling. The random duration predictor also addresses feature alignment.

[0034] Selecting at least one layer of the pre-trained speech synthesis model as the target layer includes selecting any one or any combination of text encoder, random duration predictor, projection layer, streaming layer and decoder as the target layer.

[0035] Loading the VB-LoRA fine-tuning module into the target layer means integrating the VB-LoRA fine-tuning module into the selected target layer in the pre-trained speech synthesis model. Specifically, by adding two low-rank matrices, A and B, to the target layer, in conjunction with the original fixed parameters W, the input x of the target layer is processed to generate a better output h, thereby improving the model's performance and adaptability. Figure 4 As shown, the target layer after loading the VB-LoRA fine-tuning module can be represented as: h = Wx + BAx, where h is the output of the target layer, x is the input of the target layer, W is the initial weight of the target layer, and B and A are two low-rank matrices obtained by the VB-LoRA fine-tuning module by decomposing the weight changes of the target layer.

[0036] S30: Acquire emotional speech data and fine-tune the parameters of the target layer based on the emotional speech data to obtain a trained emotional speech synthesis model. The emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond one-to-one.

[0037] When fine-tuning a pre-trained speech synthesis model, it is necessary to acquire emotional speech data. Then, the pre-trained speech synthesis model is trained using this emotional speech data to adjust and optimize the parameters of the target layer, thereby obtaining the emotional speech synthesis model upon completion of training. The emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, with each of these elements corresponding one-to-one.

[0038] In practice, the emotional speech can be obtained by directly recording the speaker or by sampling from an existing corpus. The ESD database is an emotional speech database for voice conversion research, suitable for multi-speaker and cross-lingual emotional speech conversion research. The ESD database contains 350 parallel utterances, spoken by 10 native English speakers and 10 native Chinese speakers, covering 5 emotion categories (neutral, happy, angry, sad, and surprised), and was recorded for over 29 hours under controlled acoustic conditions. In this embodiment, all speech is resampled to 16kHz from the English data in the ESD database to obtain the emotional speech. Then, the emotional speech is annotated with speech content to obtain emotional text, emotional embedding is annotated with emotional embedding to obtain emotional embedding data, and emotional classification is annotated with emotional category data. The annotated emotional text, emotional embedding data, and emotional category data correspond to the emotional speech.

[0039] In step S30, the fine-tuning of the parameters of the target layer based on emotional speech data specifically includes the following steps S31-S34:

[0040] S31: Freeze the parameters of the pre-trained speech synthesis model and initialize the parameters of the VB-LoRA fine-tuning module.

[0041] The target layer, after loading the VB-LoRA fine-tuning module, is represented as: h = Wx + BAx. For step S31, the parameters of the pre-trained speech synthesis model are frozen, i.e., the parameter W is fixed, and only two low-rank matrices A and B are trained during training. The parameters of the VB-LoRA fine-tuning module are initialized, i.e., the two low-rank matrices A and B are initialized. Low-rank matrix A can be initialized using a random Gaussian distribution, and low-rank matrix B can be initialized using a zero matrix.

[0042] S32: Perform forward propagation calculation on the target layer based on the emotional speech data to obtain the output of the target layer.

[0043] The target layer, after loading the VB-LoRA fine-tuning module, is represented as: h = Wx + BAx. Taking the emotional text from the emotional speech data as input x, the output h of the target layer is h = Wx + BAx. The output h of the target layer consists of the predicted emotional speech, predicted emotional embedding data, and predicted emotional category data obtained by inputting a sample (emotional text) into the target layer.

[0044] S33: Calculate the model loss based on the output of the target layer.

[0045] The output of the target layer includes predicted emotional speech, predicted emotional embedding data, and predicted emotional category data. For step S33, the model loss can be calculated through the following steps S331-S334:

[0046] S331: Calculate the reconstruction loss based on the predicted emotional speech of the i-th sample in the emotional speech data and the emotional speech in the corresponding sample.

[0047] Specifically, using the formula Calculate reconstruction loss L recon Where N is the number of samples in the emotional speech data, predicted i and target i These are the predicted emotional speech of the i-th sample and the emotional speech (i.e., the real emotional speech) in the corresponding sample, respectively.

[0048] S332: Calculate the emotional embedding loss based on the predicted emotional embedding data of the i-th sample in the emotional speech data and the emotional embedding data in the corresponding sample.

[0049] Specifically, using the formula Calculate the emotional embedding loss L emotion Among them, predicted_emotion i and target_emotion i These are the predicted sentiment embedding data for the i-th sample and the sentiment embedding data (i.e., the real sentiment embedding data) in the corresponding sample, respectively.

[0050] S333: Calculate the emotion classification loss based on the predicted emotion category data of the i-th sample in the emotion speech data and the emotion category data in the corresponding sample.

[0051] Specifically, using the formula Calculate the sentiment classification loss L classification Where M is the number of emotion categories, y icy is an indicator variable; if the true class of the i-th sample is c, then y ic =1, otherwise 0. It is the probability that the model predicts the i-th sample belongs to class c.

[0052] S334: Calculate the model loss based on reconstruction loss, sentiment embedding loss, and sentiment classification loss.

[0053] Specifically, using formula L total =w1*L recon +w2*L emotion +w3*L classification Calculate the model loss L total , where w1, w2, and w3 are the weights of reconstruction loss, sentiment embedding loss, and sentiment classification loss, respectively.

[0054] S34: Update the parameters of the VB-LoRA fine-tuning module by backpropagation based on the model loss.

[0055] During the fine-tuning process, since the parameters of the pre-trained speech synthesis model are frozen, backpropagation only updates the parameters of the VB-LoRA fine-tuning module, i.e., updates the low-rank matrix A and low-rank matrix B, while the parameters W of the pre-trained speech synthesis model remain unchanged. Training stops when the model loss is less than the loss threshold, resulting in a well-trained emotional speech synthesis model.

[0056] The training method for the emotional speech synthesis model in this invention can be applied to intelligent voice dialogue scenarios in finance, insurance, and healthcare. By embedding the emotional speech synthesis model trained using this method into an intelligent voice engine, during dialogue, the server receives the user's voice question-and-answer data through the client. The intelligent voice engine processes the user's voice question-and-answer data to generate corresponding response text. This response text is then input into the emotional speech synthesis model to achieve emotional speech synthesis and output. Finally, the emotional speech is output to the user through the client, improving the quality of intelligent voice dialogue.

[0057] The training method for the emotional speech synthesis model provided in this embodiment of the invention selects at least one layer of the pre-trained speech synthesis model as the target layer, loads the VB-LoRA fine-tuning module onto the target layer, and then fine-tunes it using emotional speech data to obtain a trained emotional speech synthesis model. This enables the pre-trained speech synthesis model, which originally lacked emotional expression capabilities, to achieve emotional speech synthesis and output, promoting the development of adaptive emotional speech synthesis systems and contributing to AI-generated content. The model does not incorporate emotional information during the training phase, but only during the fine-tuning process. This allows for fine-tuning by adding emotional information of different emotional categories, endowing the model with the ability to express different emotional categories, and enhancing the model's scalability and flexibility. Moreover, during fine-tuning of the pre-trained speech synthesis model, only the parameters of the target layer with the VB-LoRA fine-tuning module are adjusted, eliminating the need for full parameter fine-tuning of the entire model, reducing the workload of model fine-tuning and lowering computational costs.

[0058] Figure 5 This is another flowchart illustrating the training method for the emotional speech synthesis model provided in an embodiment of the present invention. For example... Figure 5 As shown, the training method for the emotional speech synthesis model includes steps S01 to S30, wherein steps S10 to S30 are consistent with... Figure 2 Steps S10 to S30 in the training method of the emotional speech synthesis model in the illustrated embodiment are the same and will not be repeated here.

[0059] Figure 5 The training method of the emotional speech synthesis model in the illustrated embodiment and Figure 2 Compared to the training method of the emotional speech synthesis model in the illustrated embodiment, the difference lies in that: steps S01 and S02 are included before step S10.

[0060] S01: Load the speech synthesis model to be trained and obtain the model training data.

[0061] S02: Based on the model training data, train the speech synthesis model to be trained to obtain a pre-trained speech synthesis model.

[0062] Steps S01 and S02 are mainly used for pre-training the speech synthesis model. During pre-training, the speech synthesis model to be pre-trained is first loaded, i.e., the speech synthesis model to be trained is loaded. Simultaneously, the model training data for pre-training is obtained. Then, the loaded speech synthesis model to be trained is pre-trained based on the obtained model training data. The model training data includes pre-training speech and pre-training text, and each pre-training text has a corresponding and unique pre-training speech.

[0063] In practice, the pre-training speech can be obtained by directly recording the speaker or by sampling from an existing corpus. The VCTK corpus contains approximately 44 hours of speech data spoken by 110 native English speakers with different accents, each speaking about 400 sentences. These sentences are derived from newspaper articles, rainbow paragraphs, and evoked paragraphs used for accent profiles. In this embodiment, all speech data in the VCTK corpus is resampled to 16kHz to obtain the pre-training speech. Then, the pre-training speech is content-annotated to obtain pre-training text information corresponding to the pre-training speech.

[0064] In practical applications, when performing speech synthesis output, corresponding text information needs to be input, and then a trained speech synthesis model is used to process the text information to output the corresponding speech information. In this embodiment, the trained speech synthesis model is pre-trained based on the obtained model training data to obtain a pre-trained speech synthesis model. At this time, the pre-trained speech synthesis model itself can be used directly, that is, the pre-trained speech synthesis model can synthesize speech information and is a general speech synthesis model, but it can only perform neutral speech synthesis output and does not have emotional expression capabilities.

[0065] The training method for the emotional speech synthesis model in this invention can be applied to intelligent voice dialogue scenarios in finance, insurance, and healthcare. By embedding the emotional speech synthesis model trained using this method into an intelligent voice engine, during dialogue, the server receives the user's voice question-and-answer data through the client. The intelligent voice engine processes the user's voice question-and-answer data to generate corresponding response text. This response text is then input into the emotional speech synthesis model to achieve emotional speech synthesis and output. Finally, the emotional speech is output to the user through the client, improving the quality of intelligent voice dialogue.

[0066] The training method for the emotional speech synthesis model provided in this embodiment of the invention selects at least one layer of the pre-trained speech synthesis model as the target layer, loads the VB-LoRA fine-tuning module onto the target layer, and then fine-tunes it using emotional speech data to obtain a trained emotional speech synthesis model. This enables the pre-trained speech synthesis model, which originally lacked emotional expression capabilities, to achieve emotional speech synthesis and output, promoting the development of adaptive emotional speech synthesis systems and contributing to AI-generated content. The model does not incorporate emotional information during the training phase, but only during the fine-tuning process. This allows for fine-tuning by adding emotional information of different emotional categories, endowing the model with the ability to express different emotional categories, and enhancing the model's scalability and flexibility. Moreover, during fine-tuning of the pre-trained speech synthesis model, only the parameters of the target layer with the VB-LoRA fine-tuning module are adjusted, eliminating the need for full parameter fine-tuning of the entire model, reducing the workload of model fine-tuning and lowering computational costs.

[0067] In one embodiment, a training device for an emotional speech synthesis model is provided, the training device for the emotional speech synthesis model being connected with... Figure 2 The training methods for the emotional speech synthesis models in the illustrated embodiments correspond one-to-one. For example... Figure 6 As shown, the training device for this emotional speech synthesis model includes an acquisition module 10, a loading module 20, and a fine-tuning module. Detailed descriptions of each functional module are as follows:

[0068] Get module 10, used for execution Figure 2 Step S10 in the training method of the emotional speech synthesis model in the illustrated embodiment is used to obtain a pre-trained speech synthesis model.

[0069] Load module 20, used for execution Figure 2 Step S20 in the training method of the emotional speech synthesis model in the illustrated embodiment is used to select at least one layer of the pre-trained speech synthesis model as the target layer and load the VB-LoRA fine-tuning module for the target layer.

[0070] Fine-tuning module 30 is used for execution Figure 2 Step S30 in the training method of the emotional speech synthesis model in the illustrated embodiment is used to acquire emotional speech data and fine-tune the parameters of the target layer based on the emotional speech data to obtain the trained emotional speech synthesis model. The emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond one-to-one.

[0071] The training device for the emotional speech synthesis model in this embodiment of the invention can be applied to intelligent voice dialogue scenarios in finance, insurance, medical and other businesses. By embedding the emotional speech synthesis model trained using the training device into an intelligent voice engine, during dialogue, the server receives the user's voice question-and-answer data through the client. The intelligent voice engine processes the user's voice question-and-answer data to generate corresponding response text. This response text is then input into the emotional speech synthesis model to achieve emotional speech synthesis and output. Finally, the emotional speech is output to the user through the client, improving the quality of intelligent voice dialogue.

[0072] The training device for the emotional speech synthesis model provided in this embodiment of the invention selects at least one layer of the pre-trained speech synthesis model as the target layer, loads the VB-LoRA fine-tuning module onto the target layer, and then fine-tunes it using emotional speech data to obtain a trained emotional speech synthesis model. This enables the pre-trained speech synthesis model, which originally lacked emotional expression capabilities, to achieve emotional speech synthesis and output, promoting the development of adaptive emotional speech synthesis systems and contributing to AI-generated content. The model does not incorporate emotional information during the training phase, but only during the fine-tuning process. This allows for fine-tuning by adding emotional information of different emotional categories, endowing the model with the ability to express different emotional categories, and enhancing the model's scalability and flexibility. Moreover, during fine-tuning of the pre-trained speech synthesis model, only the parameters of the target layer with the VB-LoRA fine-tuning module are adjusted, eliminating the need for full parameter fine-tuning of the entire model, reducing the workload of model fine-tuning and lowering computational costs.

[0073] Specific limitations regarding the training device for the emotional speech synthesis model can be found in the limitations on the training method for the emotional speech synthesis model above, and will not be repeated here. Each module in the aforementioned training device for the emotional speech synthesis model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0074] In one embodiment, a training device for an emotional speech synthesis model is provided, the training device for the emotional speech synthesis model being connected with... Figure 5 The training methods for the emotional speech synthesis models in the illustrated embodiments correspond one-to-one. For example... Figure 7 As shown, the training device for this emotional speech synthesis model includes a model loading module 01, a pre-training module 02, an acquisition module 10, a loading module 20, and a fine-tuning module 30. Detailed descriptions of each functional module are as follows:

[0075] Model loading module 01 is used for execution Figure 5Step S01 in the training method of the emotional speech synthesis model in the illustrated embodiment is used to load the speech synthesis model to be trained and obtain the model training data.

[0076] Pre-training module 02 is used to perform Figure 5 Step S02 in the training method of the emotional speech synthesis model in the illustrated embodiment is used to train the speech synthesis model to be trained based on the model training data to obtain a pre-trained speech synthesis model.

[0077] Get module 10, used for execution Figure 5 Step S10 in the training method of the emotional speech synthesis model in the illustrated embodiment is used to obtain a pre-trained speech synthesis model.

[0078] Load module 20, used for execution Figure 5 Step S20 in the training method of the emotional speech synthesis model in the illustrated embodiment is used to select at least one layer of the pre-trained speech synthesis model as the target layer and load the VB-LoRA fine-tuning module for the target layer.

[0079] Fine-tuning module 30 is used for execution Figure 5 Step S30 in the training method of the emotional speech synthesis model in the illustrated embodiment is used to acquire emotional speech data and fine-tune the parameters of the target layer based on the emotional speech data to obtain the trained emotional speech synthesis model. The emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond one-to-one.

[0080] The training device for the emotional speech synthesis model in this embodiment of the invention can be applied to intelligent voice dialogue scenarios in finance, insurance, medical and other businesses. By embedding the emotional speech synthesis model trained using the training device into an intelligent voice engine, during dialogue, the server receives the user's voice question-and-answer data through the client. The intelligent voice engine processes the user's voice question-and-answer data to generate corresponding response text. This response text is then input into the emotional speech synthesis model to achieve emotional speech synthesis and output. Finally, the emotional speech is output to the user through the client, improving the quality of intelligent voice dialogue.

[0081] The training device for the emotional speech synthesis model provided in this embodiment of the invention selects at least one layer of the pre-trained speech synthesis model as the target layer, loads the VB-LoRA fine-tuning module onto the target layer, and then fine-tunes it using emotional speech data to obtain a trained emotional speech synthesis model. This enables the pre-trained speech synthesis model, which originally lacked emotional expression capabilities, to achieve emotional speech synthesis and output, promoting the development of adaptive emotional speech synthesis systems and contributing to AI-generated content. The model does not incorporate emotional information during the training phase, but only during the fine-tuning process. This allows for fine-tuning by adding emotional information of different emotional categories, endowing the model with the ability to express different emotional categories, and enhancing the model's scalability and flexibility. Moreover, during fine-tuning of the pre-trained speech synthesis model, only the parameters of the target layer with the VB-LoRA fine-tuning module are adjusted, eliminating the need for full parameter fine-tuning of the entire model, reducing the workload of model fine-tuning and lowering computational costs.

[0082] In one embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 8 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with other external electronic devices via a network connection. When the computer program is executed by the processor, it implements… Figure 2 or Figure 5 The functions or steps of the training method for the emotional speech synthesis model in the illustrated embodiment.

[0083] In one embodiment, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program being implemented when executed by a processor. Figure 2 or Figure 5 The functions or steps of the training method for the emotional speech synthesis model in the illustrated embodiment.

[0084] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0086] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A training method for an emotional speech synthesis model, characterized in that, include: Obtain a pre-trained speech synthesis model; Select at least one layer of the pre-trained speech synthesis model as the target layer, and load the VB-LoRA fine-tuning module into the target layer; Emotional speech data is acquired, and the parameters of the target layer are fine-tuned based on the emotional speech data to obtain a trained emotional speech synthesis model. The emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond one-to-one. The fine-tuning of the parameters of the target layer based on emotional speech data includes: Freeze the parameters of the pre-trained speech synthesis model and initialize the parameters of the VB-LoRA fine-tuning module; Based on the emotional speech data, a forward propagation calculation is performed on the target layer to obtain the output of the target layer; The model loss is calculated based on the output of the target layer. The parameters of the VB-LoRA fine-tuning module are updated by backpropagation based on the model loss; The output of the target layer includes predicted emotional speech, predicted emotional embedding data, and predicted emotional category data. The calculation of model loss based on the output of the target layer includes: The reconstruction loss is calculated based on the predicted emotional speech of the i-th sample in the emotional speech data and the emotional speech in the corresponding sample. The emotional embedding loss is calculated based on the predicted emotional embedding data of the i-th sample in the emotional speech data and the emotional embedding data in the corresponding sample. Based on the predicted emotion category data of the i-th sample in the emotional speech data and the emotion category data in the corresponding sample, calculate the emotion classification loss; The model loss is calculated based on reconstruction loss, sentiment embedding loss, and sentiment classification loss.

2. The training method for the emotional speech synthesis model as described in claim 1, characterized in that, The target layer, after loading the VB-LoRA fine-tuning module, is represented as: h = Wx + BAx, where h is the output of the target layer, x is the input of the target layer, W is the initial weight of the target layer, and B and A are two low-rank matrices obtained by the VB-LoRA fine-tuning module by decomposing the weight changes of the target layer.

3. The training method for the emotional speech synthesis model as described in claim 1, characterized in that, Using formula Calculate reconstruction loss L recon Where N is the number of samples in the emotional speech data, predicted i and target i These are the predicted emotional speech of the i-th sample and the emotional speech in the corresponding sample, respectively. Using formula Calculate the emotional embedding loss L emotion Among them, predicted_emotion o and target_emotion o These are the predicted sentiment embedding data for the i-th sample and the sentiment embedding data for the corresponding sample, respectively. Using formula Calculate the sentiment classification loss L classification Where M is the number of emotion categories, y ic y is an indicator variable; if the true class of the i-th sample is c, then y ic =1, otherwise 0. It is the probability that the model predicts the i-th sample belongs to class c; Using formula L total =w1*L recon +w2*L emotion +w3*L classification Calculate the model loss L total , where w1, w2, and w3 are the weights of reconstruction loss, sentiment embedding loss, and sentiment classification loss, respectively.

4. The training method for the emotional speech synthesis model as described in claim 1, characterized in that, The pre-trained speech synthesis model includes a text encoder, a random duration predictor, a projection layer, a streaming layer, and a decoder. Selecting at least one layer of the pre-trained speech synthesis model as the target layer includes selecting any one or any combination of the text encoder, random duration predictor, projection layer, streaming layer, and decoder as the target layer.

5. The training method for the emotional speech synthesis model as described in claim 1, characterized in that, Before obtaining the pre-trained speech synthesis model, the following steps are included: Load the speech synthesis model to be trained and obtain the model training data; Based on the model training data, the speech synthesis model to be trained is trained to obtain a pre-trained speech synthesis model.

6. A training device for an emotional speech synthesis model, characterized in that, include: The acquisition module is used to acquire the pre-trained speech synthesis model; The loading module is used to select at least one layer of the pre-trained speech synthesis model as the target layer and load the VB-LoRA fine-tuning module for the target layer; The fine-tuning module is used to acquire emotional speech data and perform VB-LoRA fine-tuning on the parameters of the target layer based on the emotional speech data to obtain a trained emotional speech synthesis model. The emotional speech data includes emotional text, emotional speech, emotional embedding data, and emotional category data, and the emotional text, emotional speech, emotional embedding data, and emotional category data correspond one-to-one. The fine-tuning of the parameters of the target layer based on emotional speech data includes: Freeze the parameters of the pre-trained speech synthesis model and initialize the parameters of the VB-LoRA fine-tuning module; Based on the emotional speech data, a forward propagation calculation is performed on the target layer to obtain the output of the target layer; The model loss is calculated based on the output of the target layer. The parameters of the VB-LoRA fine-tuning module are updated by backpropagation based on the model loss; The output of the target layer includes predicted emotional speech, predicted emotional embedding data, and predicted emotional category data. The calculation of model loss based on the output of the target layer includes: The reconstruction loss is calculated based on the predicted emotional speech of the i-th sample in the emotional speech data and the emotional speech in the corresponding sample. The emotional embedding loss is calculated based on the predicted emotional embedding data of the i-th sample in the emotional speech data and the emotional embedding data in the corresponding sample. Based on the predicted emotion category data of the i-th sample in the emotional speech data and the emotion category data in the corresponding sample, calculate the emotion classification loss; The model loss is calculated based on reconstruction loss, sentiment embedding loss, and sentiment classification loss.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the training method for the emotional speech synthesis model as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the training method for the emotional speech synthesis model as described in any one of claims 1 to 5.

Images