Emotional speech synthesis method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a kind of sentiment speech synthesis method and device, electronic equipment and storage medium, belong to image processing technical field, it is applicable to the field of financial technology.The method comprises: the sentiment classification of voice sample data is obtained, and the sentiment classification audio data is obtained;Parameter adjustment is carried out on the target speech synthesis model based on the sentiment classification audio data, and the fine-tuning speech synthesis model is obtained;Parameter weight change is carried out on the general parameter weight in the target speech synthesis model based on the fine-tuning parameter weight in the fine-tuning speech synthesis model, and the sentiment parameter weight is obtained;Parameter adjustment is carried out on the target speech synthesis model based on the sentiment parameter weight, and the sentiment speech synthesis model is obtained;The preset target text data is carried out based on the sentiment speech synthesis model, and the sentiment speech synthesis is carried out.The embodiment of the application can improve the sentiment diversity of speech synthesis.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, is applicable to the field of financial technology, and particularly relates to an emotional speech synthesis method and device, an electronic device, and a storage medium. Background Art

[0002] Emotional speech synthesis refers to the conversion of written text into emotional speech. For example, in the financial field, in the scenario of answering financial questions, intelligent robots can generate emotional speech data by performing emotional speech synthesis on the text of the financial question response, which can improve the user experience when users consult financial issues.

[0003] At present, common emotional speech synthesis usually relies on pre-defined mapping relationships between speech features such as pitch, speed, and pauses and different emotion types. However, this method has difficulty capturing complex emotional changes, and the generated speech often appears unnatural and lacks diversity. Therefore, how to improve the emotional diversity of speech synthesis has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose an emotional speech synthesis method and device, electronic device and storage medium, aiming to improve the emotional diversity of speech synthesis.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application provides an emotional speech synthesis method, the method comprising:

[0006] Acquire text sample data and voice sample data of the text sample data;

[0007] Performing model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model, wherein the target speech synthesis model includes universal parameter weights;

[0008] Performing emotion classification on the speech sample data to obtain emotion classification audio data;

[0009] Based on the emotion classification audio data, adjusting parameters of the target speech synthesis model to obtain a fine-tuned speech synthesis model, wherein the fine-tuned speech synthesis model includes fine-tuning parameter weights;

[0010] Based on the fine-tuning parameter weight, modifying the general parameter weight to obtain the emotional parameter weight;

[0011] Based on the emotion parameter weights, adjusting the parameters of the target speech synthesis model to obtain an emotion speech synthesis model;

[0012] Based on the emotional speech synthesis model, emotional speech synthesis is performed on the preset target text data.

[0013] In some embodiments, performing emotional speech synthesis on preset target text data based on the emotional speech synthesis model includes:

[0014] Based on the emotional speech synthesis model, the target text data is parsed to obtain text emotional features;

[0015] Based on the emotional speech synthesis model, encoding the target text data to obtain a speech encoding vector;

[0016] Based on the emotional features of the text, adjusting the parameters of the emotional speech synthesis model to obtain target parameter weights;

[0017] Based on the target parameter weight, audio generation is performed on the speech coding vector to obtain synthesized emotional speech data.

[0018] In some embodiments, performing emotion classification on the speech sample data to obtain emotion-classified audio data includes:

[0019] Performing emotion recognition on the speech sample data to obtain audio emotion data and emotion confidence, wherein the emotion confidence is used to characterize the accuracy of the audio emotion data;

[0020] Based on the audio emotion data, audio classification is performed on the voice sample data to obtain single emotion audio data, wherein the single emotion audio data has a single audio emotion data;

[0021] Based on the emotion confidence, the single emotion audio data is screened to obtain the emotion classification audio data.

[0022] In some embodiments, performing emotion recognition on the speech sample data to obtain audio emotion data and emotion confidence includes:

[0023] Performing spectrum conversion on the speech sample data to obtain a target Mel spectrum;

[0024] Performing feature extraction on the target Mel spectrum to obtain audio features;

[0025] Performing feature activation on the audio feature to obtain the emotion confidence;

[0026] Based on the emotion confidence, the audio features are subjected to emotion screening to obtain the audio emotion data.

[0027] In some embodiments, the modifying the general parameter weight based on the fine-tuning parameter weight to obtain the emotion parameter weight includes:

[0028] Performing a subtraction operation on the fine-tuning parameter weight and the general parameter weight to obtain an initial emotion parameter weight;

[0029] Performing vector scaling on the initial emotion parameter weight to obtain a scaled emotion parameter weight;

[0030] The scaled emotion parameter weight and the general parameter weight are summed to obtain the emotion parameter weight.

[0031] In some embodiments, adjusting parameters of the target speech synthesis model based on the emotion parameter weight to obtain an emotion speech synthesis model includes:

[0032] Based on the emotion parameter weights, the target speech synthesis model is subjected to parameter replacement to obtain a parameter-replaced emotion speech synthesis model;

[0033] The parameter-changing emotional speech synthesis model is trained to obtain the emotional speech synthesis model.

[0034] In some embodiments, fine-tuning parameters of the target speech synthesis model based on the emotion classification audio data to obtain a fine-tuned speech synthesis model includes:

[0035] Performing emotion identification on the emotion classification audio data to obtain an emotion label;

[0036] Based on the sentiment classification audio data, performing text search on the text sample data to obtain sentiment classification text data;

[0037] Based on the emotion label, the emotion classification audio data and the emotion classification text data, the target speech synthesis model is trained to obtain the fine-tuned speech synthesis model.

[0038] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an emotional speech synthesis device, the device comprising:

[0039] A sample data acquisition module, configured to acquire speech synthesis sample data, wherein the speech synthesis sample data includes text sample data and speech sample data of the text sample data;

[0040] A conventional model training module, configured to perform model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model, wherein the target speech synthesis model includes universal parameter weights;

[0041] A speech data classification module is used to perform emotion classification on the speech sample data to obtain emotion classification audio data;

[0042] A model parameter adjustment module, configured to adjust parameters of the target speech synthesis model based on the emotion classification audio data to obtain a fine-tuned speech synthesis model, wherein the fine-tuned speech synthesis model includes fine-tuning parameter weights;

[0043] An emotional parameter calculation module, configured to modify the general parameter weights based on the fine-tuning parameter weights to obtain emotional parameter weights;

[0044] A target model generation module is used to adjust the parameters of the target speech synthesis model based on the emotion parameter weights to obtain an emotional speech synthesis model;

[0045] Emotional speech synthesis module: used to perform emotional speech synthesis on preset target text data based on the emotional speech synthesis model to obtain synthesized emotional speech data.

[0046] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0047] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0048] The emotional speech synthesis method and device, electronic device and storage medium proposed in the present application obtain text sample data and its corresponding speech sample data, and then train a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model including general parameter weights. Then, the speech sample data is subjected to emotion classification to obtain emotion classification audio data, and the emotion classification audio data is used to adjust the parameters of the target speech synthesis model to obtain a fine-tuning speech synthesis model including fine-tuning parameter weights. Then, the general parameter weights are changed based on the fine-tuning parameter weights to obtain emotion parameter weights, and the parameters of the target speech synthesis model are adjusted accordingly to form an emotional speech synthesis model. Finally, the emotional speech synthesis model is used to perform emotional speech synthesis on the target text data, which can capture more complex emotional changes in the target text data, make the synthesized speech more natural, and improve the emotional diversity of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of the emotional speech synthesis method provided by an embodiment of the present application;

[0050] Figure 2 yes Figure 1 Flowchart of step S103 in FIG.

[0051] Figure 3 yes Figure 2 Flowchart of step S201 in FIG.

[0052] Figure 4 yes Figure 1 Flowchart of step S104 in FIG.

[0053] Figure 5 yes Figure 1 Flowchart of step S105 in FIG.

[0054] Figure 6 yes Figure 1 Flowchart of step S106 in FIG.

[0055] Figure 7 is a flowchart of an emotional speech synthesis method provided by another embodiment of the present application;

[0056] Figure 8 This is a schematic diagram of the structure of the emotional speech synthesis device provided in an embodiment of the present application;

[0057] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0059] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0061] First, the meanings of several terms involved in this application are analyzed:

[0062] Parameter weight: Parameter weight is a key component in machine learning and deep learning models, which represents adjustable numerical values within the model that affect the model's predictions or outputs. During the training process, these weights are optimized based on input data and desired outputs so that the model can accurately learn and model the complex relationships between data. The adjustment of parameter weights directly affects the performance of the model, for example, in a speech synthesis model, different parameter weights can produce different voice characteristics such as pitch, volume, and speed, making the synthesized speech more close to the emotions and intonations of real human speech.

[0063] Confidence: Confidence is an indicator that measures the certainty of a model's prediction, which represents the degree of trust that the model has in its output. In various machine learning tasks, confidence can help us understand the reliability of the model's predictions, for example, in sentiment recognition, high confidence means that the model has a high degree of certainty that the recognized sentiment label matches the actual emotional state in the audio, while in the case of low confidence, it indicates that the model is not very sure about its prediction, and more data or more complex models may be needed to improve the accuracy of the prediction.

[0064] Emotional speech synthesis refers to the conversion of written text into speech with emotions, for example, in the financial problem answering scene in the financial field, an intelligent robot generates emotional speech data by performing emotional speech synthesis on financial problem answering text, which can improve the user experience when users consult financial problems.

[0065] Currently, common emotional speech synthesis usually relies on the mapping relationship between pre-defined voice characteristics such as pitch, speed, and pause and different emotion types, but this method is difficult to capture complex emotional changes, and the generated speech often appears unnatural and lacks diversity, therefore, how to improve the emotional diversity of speech synthesis has become a technical problem to be solved.

[0066] Based on this, the embodiments of the present application provide an emotional speech synthesis method and device, electronic equipment and storage medium, aiming to improve the emotional diversity of speech synthesis.

[0067] The emotional speech synthesis method and device, electronic equipment and storage medium provided by the embodiments of the present application are specifically explained through the following embodiments, first, the emotional speech synthesis method in the embodiments of the present application is described.

[0068] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0069] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0070] The emotional speech synthesis method provided in the embodiment of the present application relates to the field of speech synthesis technology. The emotional speech synthesis method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the emotional speech synthesis method, etc., but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0072] Figure 1 This is an optional flowchart of the emotional speech synthesis method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0073] Step S101, obtaining text sample data and voice sample data of the text sample data;

[0074] Step S102: performing model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model, wherein the target speech synthesis model includes universal parameter weights;

[0075] Step S103, performing emotion classification on the speech sample data to obtain emotion classification audio data;

[0076] Step S104: adjusting parameters of the target speech synthesis model based on the emotion classification audio data to obtain a fine-tuned speech synthesis model, wherein the fine-tuned speech synthesis model includes fine-tuning parameter weights;

[0077] Step S105, based on the fine-tuning parameter weight, modify the general parameter weight to obtain the emotional parameter weight;

[0078] Step S106: adjusting parameters of the target speech synthesis model based on the emotion parameter weights to obtain an emotion speech synthesis model;

[0079] Step S107: performing emotional speech synthesis on the preset target text data based on the emotional speech synthesis model.

[0080] In steps S101 to S107 shown in the embodiment of the present application, text sample data and speech sample data of the text sample data are obtained, and then a preset initial speech synthesis model is trained according to the text sample data and the number of speech samples to obtain a target speech synthesis model, wherein the target speech synthesis model includes general parameter weights, and further, emotion classification is performed on the speech sample data to obtain emotion classification audio data, and then the target speech synthesis model is parameter-adjusted according to the emotion classification audio data to obtain a fine-tuning speech synthesis model, wherein the fine-tuning speech synthesis model includes fine-tuning parameter weights, and then, according to the fine-tuning parameter weights, the general parameter weights are parameter-changed to obtain emotion parameter weights, and then according to the emotion parameter weights, the target speech synthesis model is parameter-adjusted to obtain an emotion speech synthesis model, and finally, according to the emotion speech synthesis model, emotion speech synthesis is performed on the preset target text data. Therefore, the present application obtains text sample data and its corresponding speech sample data, and then trains a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model including general parameter weights. Next, emotion classification is performed on the speech sample data to obtain emotion classification audio data, and then the emotion classification audio data is used to adjust the parameters of the target speech synthesis model to obtain a fine-tuning speech synthesis model including fine-tuning parameter weights. Then, the general parameter weights are changed based on the fine-tuning parameter weights to obtain emotion parameter weights, and the parameters of the target speech synthesis model are adjusted accordingly to form an emotion speech synthesis model. Finally, the emotion speech synthesis model is used to perform emotion speech synthesis on the target text data, which can capture more complex emotion changes in the target text data, making the synthesized speech more natural and improving the emotional diversity of speech synthesis.

[0081] In step S101 of some embodiments, text sample data refers to written text data used to train the model, which includes text information to be converted into speech. Speech sample data refers to actual speech recordings corresponding to the text sample data, used to train the model to imitate human speech.

[0082] This application can collect text and voice data from public datasets to form text sample data and voice sample data, or collect text and voice data from user uploads or professional recordings to form text sample data and voice sample data, thereby ensuring the diversity of text sample data and voice sample data. It should be noted that the text sample data and voice sample data cover different emotions, intonations, speaking speeds and other characteristics.

[0083] In step S102 of some embodiments, the initial speech synthesis model refers to a preset model used for speech synthesis before any training. It should be noted that the initial speech synthesis model usually includes some basic speech synthesis capabilities, but the accuracy of speech synthesis is not high.

[0084] This application can use deep learning technology to train the initial speech synthesis model to learn the mapping relationship between text sample data and speech sample data, and adjust the parameters of the initial speech synthesis model during the learning process until the initial speech synthesis model can generate accurate speech output, thereby obtaining a target speech synthesis model containing universal parameter weights.

[0085] In step S103 of some embodiments, the speech sample data is analyzed by emotion recognition technology to determine its emotion attributes and corresponding confidence levels. The confidence levels are used to measure the reliability of the recognition results. Then, based on the identified emotion attributes, the speech sample data are classified into single emotion audio data with a single emotion. Finally, these single emotion audio data are screened according to the emotion confidence levels to obtain emotion classification audio data.

[0086] For details, see Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S203:

[0087] Step S201: performing emotion recognition on the speech sample data to obtain audio emotion data and emotion confidence, wherein the emotion confidence is used to characterize the accuracy of the audio emotion data;

[0088] Step S202: Based on the audio emotion data, audio classification is performed on the speech sample data to obtain single emotion audio data, wherein the single emotion audio data has a single audio emotion data;

[0089] Step S203: Based on the emotion confidence, the single emotion audio data is screened to obtain emotion classification audio data.

[0090] In step S201 of some embodiments, emotion recognition refers to parsing speech sample data using an emotion recognition algorithm to determine the type of emotion contained in the speech sample data.

[0091] This application first performs a spectrum conversion operation on the collected speech sample data to convert the original audio signal into a target Mel spectrum. Next, key audio features are extracted from the target Mel spectrum. Then, the extracted audio features are analyzed through a feature activation process to obtain emotion confidence. Finally, the audio features are emotion-screened based on the emotion confidence to ensure that only those audio features with sufficiently high confidence are identified as valid audio emotion data, thereby obtaining accurate emotion classification results.

[0092] For details, see Figure 3 In some embodiments, step S201 may include but is not limited to steps S301 to S304:

[0093] Step S301, performing spectrum conversion on the speech sample data to obtain a target Mel spectrum;

[0094] Step S302, extracting features from the target Mel spectrum to obtain audio features;

[0095] Step S303: Activate the audio features to obtain the emotion confidence.

[0096] Step S304: Based on the emotion confidence, the audio features are subjected to emotion screening to obtain audio emotion data.

[0097] In step S301 of some embodiments, spectrum conversion refers to converting the speech sample data from the time domain to the frequency domain so as to analyze the frequency components of the speech.

[0098] This application obtains standard speech data by preprocessing speech sample data, where the preprocessing includes denoising and normalization. Then, the standard speech data is converted from a time series to a frequency series using Fourier transform. Then, a Mel filter bank is applied to convert the frequency series into a Mel frequency scale to obtain a target Mel spectrum.

[0099] In step S302 of some embodiments, feature extraction refers to extracting parameters that can represent speech features from the Mel spectrum, and these parameters can capture emotional information in the speech.

[0100] This application can obtain audio features by extracting features such as Mel-frequency cepstral coefficients, pitch, intensity, and duration from the target Mel spectrum.

[0101] In step S303 of some embodiments, feature activation refers to processing audio features through an activation function to obtain emotion confidence.

[0102] This application inputs the extracted audio features into a classifier, such as a support vector machine or a decision tree. The classifier learns the mapping relationship between the audio features and emotions based on the training data, and assigns an emotion label and confidence level, i.e., emotion confidence level, to the audio features.

[0103] In step S304 of some embodiments, emotion tags with emotion confidence greater than a preset emotion confidence threshold are retained and used as emotion data of the speech sample data to obtain audio emotion data.

[0104] In steps S301 to S304 shown in this embodiment, the original audio signal is converted into a target Mel spectrum by performing a spectrum conversion operation on the collected speech sample data. This process involves converting the time domain signal of the sound into a frequency domain signal in order to better capture the characteristics of the sound. Next, key audio features are extracted from the target Mel spectrum. These features can represent the emotional information in the speech signal. Then, through the feature activation process, the extracted audio features are analyzed to obtain the emotional confidence. This confidence reflects the strength of the association between the audio features and the specific emotion. Finally, the audio features are emotionally screened based on the emotional confidence to ensure that only those audio features with sufficiently high confidence are identified as valid audio emotion data, thereby obtaining accurate emotion classification results. This process ensures the accuracy and reliability of emotion recognition and provides a high-quality data foundation for subsequent emotional speech synthesis.

[0105] In step S202 of some embodiments, audio classification refers to classifying the speech sample data into corresponding emotion categories based on the recognized audio emotion data.

[0106] This application obtains single-emotion audio data representing a single audio emotion data by using the identified audio emotion data as an emotion label and then classifying the corresponding voice sample data into a preset emotion category according to the emotion label. For example, the voice sample data with the audio emotion data of "happy" is classified into the single-emotion audio data "happy".

[0107] It is important to note that after classification, the speech sample data in each category is checked to ensure that each speech sample data contains only one emotion. If the speech sample data is found to contain multiple emotions, it is removed from the single emotion audio data.

[0108] In step S203 of some embodiments, data screening refers to selecting audio data with a confidence level higher than a specific threshold based on the emotion confidence level.

[0109] This application sets a threshold for emotion confidence, and then filters out speech sample data with emotion confidence higher than the threshold from the single emotion audio data for storage, thereby obtaining emotion classification audio data.

[0110] In steps S201 to S203 shown in this embodiment, the speech sample data is analyzed by emotion recognition technology to determine its emotion attributes and corresponding confidence levels. The confidence levels are used to measure the reliability of the recognition results. Then, based on the identified emotion attributes, the speech sample data are classified into single emotion audio data with a single emotion. Finally, these single emotion audio data are screened according to the emotion confidence levels to ensure that the screened emotion classification audio data are both accurate and reliable. This screening process helps to improve the training effect of the emotion speech synthesis model, enabling it to more realistically simulate human emotions and enhance the naturalness and diversity of speech synthesis.

[0111] In step S104 of some embodiments, emotion labels are obtained by emotion identification of emotion classification audio data. Subsequently, text search operations are performed on text sample data using these emotion classification audio data to obtain emotion classification text data that matches the audio data. Finally, the target speech synthesis model is carefully trained by combining the emotion labels, emotion classification audio data, and emotion classification text data. Through this training process, the parameters of the model are optimized, thereby generating a fine-tuned speech synthesis model that can more accurately simulate emotional information.

[0112] For details, see Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S403:

[0113] Step S401, emotion identification is performed on the emotion classification audio data to obtain an emotion label;

[0114] Step S402: performing text search on the text sample data based on the sentiment classification audio data to obtain sentiment classification text data;

[0115] Step S403: Based on the emotion label, the emotion classification audio data, and the emotion classification text data, the target speech synthesis model is trained to obtain a fine-tuned speech synthesis model.

[0116] In step S401 of some embodiments, emotion identification refers to assigning a label representing the emotion category of the audio data.

[0117] This application can predefine a set of emotion categories, and then analyze audio features (such as pitch, rhythm, intensity) to determine the emotion category of the audio data and assign it a corresponding emotion label.

[0118] In step S402 of some embodiments, text search refers to finding text that matches the sentiment classification audio data in the text sample data.

[0119] Before performing text search, this application needs to create an index for each audio sample data and record its corresponding text sample data. Then, based on the index, text sample data that matches the emotion classification audio data, i.e., emotion classification text data, can be searched.

[0120] In step S403 of some embodiments, model training refers to using emotion tags, emotion classification audio data, and emotion classification text data to adjust and optimize the speech synthesis model so that it can generate speech with specific emotion characteristics.

[0121] This application inputs emotion classification audio data and corresponding emotion classification text data as training samples into the target speech synthesis model. Then, by adjusting the model parameters, the target speech synthesis model can learn how to generate emotion classification audio data with corresponding emotion characteristics based on the emotion classification text data, thereby obtaining a fine-tuned speech synthesis model.

[0122] In steps S401 to S403 shown in this embodiment, emotion labels are obtained by emotion identification of emotion classification audio data. This process involves identifying emotion features in the audio and matching them with specific emotion categories. Subsequently, these emotion classification audio data are used to perform text search operations on the text sample data to obtain emotion classification text data that matches the audio data. Finally, the target speech synthesis model is carefully trained by combining the emotion labels, emotion classification audio data and emotion classification text data. Through this training process, the parameters of the model are optimized, thereby generating a fine-tuned speech synthesis model that can more accurately simulate human emotions.

[0123] In step S105 of some embodiments, the fine-tuning parameter weight is subtracted from the general parameter weight to obtain an initial emotion parameter weight. Subsequently, a vector scaling operation is performed on the obtained initial emotion parameter weight to adjust its numerical value, thereby obtaining a scaled emotion parameter weight. Finally, the scaled emotion parameter weight is summed with the general parameter weight to combine their effects to generate a final emotion parameter weight.

[0124] For details, see Figure 5 In some embodiments, step S105 may include but is not limited to steps S501 to S503:

[0125] Step S501, performing a subtraction operation on the fine-tuning parameter weight and the general parameter weight to obtain the initial emotion parameter weight;

[0126] Step S502, performing vector scaling on the initial emotion parameter weight to obtain a scaled emotion parameter weight;

[0127] Step S503 , performing parameter weight summation on the scaled emotion parameter weight and the general parameter weight to obtain the emotion parameter weight.

[0128] In step S501 of some embodiments, the initial emotion parameter weights are obtained by subtracting the corresponding general parameter weight from each value of the fine-tuning parameter weight. For example, assuming the fine-tuning parameter weights are [0.5, 0.3, 0.2] and the general parameter weights are [0.4, 0.2, 0.4], after the subtraction operation, the initial emotion parameter weights obtained are [0.1, 0.1, -0.2].

[0129] In step S502 of some embodiments, vector scaling refers to adjusting the numerical value of the initial emotion parameter weight.

[0130] This application can achieve vector scaling of the initial emotional parameter weights by multiplying each value of the initial emotional parameter weights by a scaling factor to obtain scaled emotional parameter weights. For example, if the initial emotional parameter weights are [0.1, 0.1, -0.2], a scaling factor of 2 is selected, then the scaled emotional parameter weights become [0.2, 0.2, -0.4].

[0131] In step S503 of some embodiments, summing the parameter weights refers to adding the scaled emotion parameter weights to the general parameter weights to integrate the effects of the two.

[0132] This application obtains the emotional parameter weight by adding each value of the scaled emotional parameter weight to the corresponding value of the universal parameter weight. For example, assuming the scaled emotional parameter weights are [0.2, 0.2, -0.4] and the universal parameter weights are [0.4, 0.2, 0.4], after the summation operation, the obtained emotional parameter weights are [0.6, 0.4, 0.0].

[0133] In steps S501 to S503 shown in this embodiment, the fine-tuning parameter weights are subtracted from the general parameter weights by performing a subtraction operation to obtain the initial emotional parameter weights. This step reveals the difference between the two weight sets. Subsequently, a vector scaling operation is performed on the initial emotional parameter weights to adjust their values ​​to obtain scaled emotional parameter weights. This process is intended to amplify or reduce the differences as necessary. Finally, the scaled emotional parameter weights are added to the general parameter weights, and the effects of the two are combined through a parameter weight summation operation to generate the final emotional parameter weights, so that the speech synthesis model can more accurately capture and express the subtle changes in human emotions, thereby improving the naturalness and emotional expression ability of the synthesized speech.

[0134] In step S106 of some embodiments, the parameters in the target speech synthesis model are replaced according to the emotional parameter weights to generate a parameter-changed emotional speech synthesis model. Subsequently, the parameter-changed emotional speech synthesis model is further optimized through model training so that it can simulate and express emotions more accurately, and finally form an emotional speech synthesis model that can output speech with rich emotional colors.

[0135] For details, see Figure 6 In some embodiments, step S106 may include but is not limited to steps S601 to S602:

[0136] Step S601: Based on the emotional parameter weights, the target speech synthesis model is subjected to parameter replacement to obtain a parameter-replaced emotional speech synthesis model.

[0137] Step S602: Perform model training on the parameter-swapped emotional speech synthesis model to obtain an emotional speech synthesis model.

[0138] In step S601 of some embodiments, parameter replacement refers to replacing the original parameter weights in the target speech synthesis model with emotion parameter weights so that the model can express specific emotions.

[0139] This application first determines the parameters that need to be adjusted in the target speech synthesis model, and then replaces the values ​​of these parameters with the values ​​of the emotion parameter weights to obtain a parameter-changed emotion speech synthesis model.

[0140] In step S602 of some embodiments, model training refers to optimizing the parameters of the parameter-changing emotional speech synthesis model through a large amount of data and an iterative process, so that it can more accurately generate speech with specific emotional characteristics.

[0141] This application inputs audio data with emotion labels and corresponding text data as training sets into the parameter-changing emotional speech synthesis model. The parameter-changing emotional speech synthesis model learns these data and adjusts internal parameters to minimize the difference between the output speech and the target audio, thereby obtaining an emotional speech synthesis model.

[0142] In steps S601 to S602 shown in this embodiment, the parameters in the target speech synthesis model are replaced according to the emotional parameter weights, thereby generating a parameter-changed emotional speech synthesis model. This process involves updating the original parameter weights to new weights that can reflect specific emotional characteristics. Subsequently, the parameter-changed emotional speech synthesis model is further optimized through model training so that it can simulate and express emotions more accurately, and finally form an emotional speech synthesis model that can output speech with rich emotional colors, so that the emotional speech synthesis model can more accurately simulate and reproduce the subtle differences in human emotions, thereby generating more natural and expressive speech output, and improving the user experience and interaction quality of speech synthesis technology in various application scenarios.

[0143] In step S107 of some embodiments, the emotional speech synthesis model can be used to parse the preset target text data to identify and extract the text emotional features in the text. Subsequently, the emotional speech synthesis model performs encoding processing on the target text data and converts it into a speech coding vector. Then, based on the extracted text emotional features, the emotional speech synthesis model adjusts the parameters to obtain target parameter weights that adapt to the emotions of the target text. Finally, using the target parameter weights, the emotional speech synthesis model generates audio for the speech coding vector, thereby generating synthetic emotional speech data that matches the emotion of the target text data.

[0144] For details, see Figure 7 In some embodiments, step S107 may include but is not limited to steps S701 to S704:

[0145] Step S701: Based on the emotional speech synthesis model, the target text data is parsed to obtain the text emotional features;

[0146] Step S702: encoding the target text data based on the emotional speech synthesis model to obtain a speech coding vector;

[0147] Step S703: adjusting parameters of the emotional speech synthesis model based on the text emotional features to obtain target parameter weights;

[0148] Step S704: Based on the target parameter weights, audio is generated for the speech coding vector to obtain synthesized emotional speech data.

[0149] In step S701 of some embodiments, text parsing refers to analyzing text data to identify and extract emotional features therein.

[0150] This application can use natural language processing technology, such as sentiment analysis, to identify the emotional tendency in the target text data. It should be noted that natural language processing technology includes steps such as identifying keywords, phrases or sentence structures.

[0151] In step S702 of some embodiments, encoding processing refers to converting text data into a numerical form that can be understood by the model, that is, a speech encoding vector.

[0152] This application first converts the target text data into a series of phonemes or phonemes, and then further converts the phonemes or phonemes into coding vectors, thereby realizing the encoding processing of the target text data using the emotional speech synthesis model to obtain the speech coding vector.

[0153] In step S703 of some embodiments, parameter adjustment refers to adjusting parameters of the speech synthesis model according to the emotional characteristics of the text, so as to generate speech with corresponding emotional coloring.

[0154] This application determines the text emotion of the target text data by performing feature analysis on the text emotion features, thereby determining the emotion type required for the generated synthetic speech, and then adjusts the parameters in the emotional speech synthesis model accordingly based on the emotion type, thereby obtaining the target parameter weights that are suitable for the text emotion of the target text data.

[0155] In step S704 of some embodiments, audio generation refers to converting the speech coding vector into an actual audio signal using the adjusted parameter weights.

[0156] This application can use the emotional speech synthesis model to convert the speech coding vector into waveform data, and then use waveform synthesis technology to convert the waveform into an audio signal, thereby generating synthesized emotional speech data.

[0157] In steps S701 to S704 shown in this embodiment, the target text data is parsed using the emotional speech synthesis model to identify and extract emotional features in the text. Next, the target text is encoded using the emotional speech synthesis model to generate a corresponding speech encoding vector. Then, based on the extracted text emotional features, the emotional speech synthesis model is used to adjust its parameters to obtain target parameter weights suitable for generating specific emotional speech. Finally, based on the target parameter weights, the emotional speech synthesis model generates audio for the speech encoding vector, thereby producing synthesized emotional speech data that matches the text content. This process enables speech synthesis to not only convey information but also express corresponding emotions, thereby improving the application scope and user experience of speech synthesis.

[0158] This application obtains text sample data and its corresponding voice sample data, and then trains a preset initial speech synthesis model based on the text sample data and the voice sample data to obtain a target speech synthesis model containing general parameter weights. Next, emotion classification is performed on the speech sample data to obtain emotion classification audio data. The emotion classification audio data is then used to adjust the parameters of the target speech synthesis model to obtain a fine-tuning speech synthesis model containing fine-tuning parameter weights. Then, the general parameter weights are changed based on the fine-tuning parameter weights to obtain emotion parameter weights, and the parameters of the target speech synthesis model are adjusted accordingly to form an emotion speech synthesis model. Finally, the emotion speech synthesis model is used to perform emotion speech synthesis on the target text data, which can capture more complex emotion changes in the target text data, make the synthesized speech more natural, and improve the emotional diversity of speech synthesis.

[0159] See also Figure 8 The embodiment of the present application further provides an emotional speech synthesis device, which can implement the above-mentioned emotional speech synthesis method, and the device includes:

[0160] The sample data acquisition module 801 is used to acquire speech synthesis sample data, wherein the speech synthesis sample data includes text sample data and speech sample data of the text sample data;

[0161] A conventional model training module 802 is configured to perform model training on a preset initial speech synthesis model based on text sample data and speech sample data to obtain a target speech synthesis model, wherein the target speech synthesis model includes general parameter weights;

[0162] The speech data classification module 803 is used to perform emotion classification on the speech sample data to obtain emotion classification audio data;

[0163] A model parameter adjustment module 804 is configured to adjust parameters of a target speech synthesis model based on the emotion classification audio data to obtain a fine-tuned speech synthesis model, wherein the fine-tuned speech synthesis model includes fine-tuning parameter weights;

[0164] The emotional parameter calculation module 805 is used to modify the general parameter weights based on the fine-tuning parameter weights to obtain the emotional parameter weights;

[0165] A target model generation module 806 is configured to adjust parameters of a target speech synthesis model based on the emotion parameter weights to obtain an emotion speech synthesis model;

[0166] The emotional speech synthesis module 807 is used to perform emotional speech synthesis on preset target text data based on the emotional speech synthesis model to obtain synthesized emotional speech data.

[0167] The specific implementation of the emotional speech synthesis device is basically the same as the specific embodiment of the above-mentioned emotional speech synthesis method, and will not be repeated here.

[0168] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned emotional speech synthesis method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0169] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0170] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0171] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 902 and are called by the processor 901 to execute the emotional speech synthesis method of the embodiments of this application.

[0172] Input / output interface 903, used to implement information input and output;

[0173] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0174] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0175] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0176] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the emotion speech synthesis method.

[0177] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0178] The embodiment of the present application provides an emotion speech synthesis method, an emotion speech synthesis device, an electronic device and a storage medium. The emotion speech synthesis method comprises the following steps: obtaining text sample data and speech sample data of the text sample data; performing model training on a preset initial speech synthesis model according to the text sample data and the speech sample data, to obtain a target speech synthesis model, wherein the target speech synthesis model comprises a general parameter weight; further performing emotion classification on the speech sample data to obtain emotion classification audio data; performing parameter adjustment on the target speech synthesis model according to the emotion classification audio data, to obtain a fine-tuning speech synthesis model, wherein the fine-tuning speech synthesis model comprises a fine-tuning parameter weight; then performing parameter weight change on the general parameter weight according to the fine-tuning parameter weight, to obtain an emotion parameter weight; performing parameter adjustment on the target speech synthesis model according to the emotion parameter weight, to obtain an emotion speech synthesis model; and finally performing emotion speech synthesis on preset target text data according to the emotion speech synthesis model.

[0179] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0180] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0181] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0182] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0183] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0184] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0186] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0188] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0189] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. An emotional speech synthesis method, characterized in that: The method comprises: Acquire text sample data and voice sample data of the text sample data; Performing model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model, wherein the target speech synthesis model includes universal parameter weights; Performing emotion classification on the speech sample data to obtain emotion classification audio data; Based on the emotion classification audio data, adjusting parameters of the target speech synthesis model to obtain a fine-tuned speech synthesis model, wherein the fine-tuned speech synthesis model includes fine-tuning parameter weights; Performing a subtraction operation on the fine-tuning parameter weight and the general parameter weight to obtain an initial emotion parameter weight; Performing vector scaling on the initial emotion parameter weight to obtain a scaled emotion parameter weight; Performing parameter weight summation on the scaled emotion parameter weight and the general parameter weight to obtain an emotion parameter weight; Based on the emotion parameter weights, adjusting the parameters of the target speech synthesis model to obtain an emotion speech synthesis model; Based on the emotional speech synthesis model, emotional speech synthesis is performed on the preset target text data.

2. The method according to claim 1, characterized in that The emotional speech synthesis is performed on the preset target text data based on the emotional speech synthesis model, including: Based on the emotional speech synthesis model, the target text data is parsed to obtain text emotional features; Based on the emotional speech synthesis model, encoding the target text data to obtain a speech encoding vector; Based on the emotional features of the text, adjusting parameters of the emotional speech synthesis model to obtain target parameter weights; Based on the target parameter weight, audio generation is performed on the speech coding vector to obtain synthesized emotional speech data.

3. The method according to claim 1, characterized in that The performing emotion classification on the speech sample data to obtain emotion classification audio data includes: Performing emotion recognition on the speech sample data to obtain audio emotion data and emotion confidence, wherein the emotion confidence is used to characterize the accuracy of the audio emotion data; Based on the audio emotion data, audio classification is performed on the voice sample data to obtain single emotion audio data, wherein the single emotion audio data has a single audio emotion data; Based on the emotion confidence, the single emotion audio data is screened to obtain the emotion classification audio data.

4. The method according to claim 3, characterized in that The performing emotion recognition on the speech sample data to obtain audio emotion data and emotion confidence includes: Performing spectrum conversion on the speech sample data to obtain a target Mel spectrum; Performing feature extraction on the target Mel spectrum to obtain audio features; Performing feature activation on the audio feature to obtain the emotion confidence; Based on the emotion confidence, the audio features are subjected to emotion screening to obtain the audio emotion data.

5. The method according to any one of claims 1 to 4, characterized in that The step of adjusting parameters of the target speech synthesis model based on the emotion parameter weight to obtain an emotion speech synthesis model includes: Based on the emotion parameter weights, the target speech synthesis model is subjected to parameter replacement to obtain a parameter-replaced emotion speech synthesis model; The parameter-changing emotional speech synthesis model is trained to obtain the emotional speech synthesis model.

6. The method according to any one of claims 1 to 4, characterized in that Fine-tuning parameters of the target speech synthesis model based on the emotion classification audio data to obtain a fine-tuned speech synthesis model includes: Performing emotion identification on the emotion classification audio data to obtain an emotion label; Based on the sentiment classification audio data, performing text search on the text sample data to obtain sentiment classification text data; Based on the emotion label, the emotion classification audio data and the emotion classification text data, the target speech synthesis model is trained to obtain the fine-tuned speech synthesis model.

7. An emotional speech synthesis device, characterized in that: The device comprises: A sample data acquisition module, configured to acquire speech synthesis sample data, wherein the speech synthesis sample data includes text sample data and speech sample data of the text sample data; A conventional model training module, configured to perform model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model, wherein the target speech synthesis model includes universal parameter weights; A speech data classification module is used to perform emotion classification on the speech sample data to obtain emotion classification audio data; A model parameter adjustment module, configured to adjust parameters of the target speech synthesis model based on the emotion classification audio data to obtain a fine-tuned speech synthesis model, wherein the fine-tuned speech synthesis model includes fine-tuning parameter weights; An emotional parameter calculation module is used to perform a subtraction operation on the fine-tuning parameter weight and the general parameter weight to obtain an initial emotional parameter weight, perform vector scaling on the initial emotional parameter weight to obtain a scaled emotional parameter weight, and perform parameter weight summation on the scaled emotional parameter weight and the general parameter weight to obtain an emotional parameter weight; A target model generation module is used to adjust the parameters of the target speech synthesis model based on the emotion parameter weights to obtain an emotional speech synthesis model; Emotional speech synthesis module: used to perform emotional speech synthesis on preset target text data based on the emotional speech synthesis model to obtain synthesized emotional speech data.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the emotional speech synthesis method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the emotional speech synthesis method according to any one of claims 1 to 6 is implemented.

Images