Emotional speech synthesis method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an emotional speech synthesis method and device, electronic equipment and a storage medium, belongs to the technical field of image processing, and is suitable for the field of financial science and technology. The method comprises the following steps: performing sentiment classification on voice sample data to obtain sentiment classification audio data; based on the emotion classification audio data, performing parameter adjustment on the target speech synthesis model to obtain a fine adjustment speech synthesis model; on the basis of a fine tuning parameter weight in the fine tuning speech synthesis model, performing parameter weight change on a general parameter weight in the target speech synthesis model to obtain an emotion parameter weight; based on the emotional parameter weight, performing parameter adjustment on the target speech synthesis model to obtain an emotional speech synthesis model; and based on the emotional speech synthesis model, performing emotional speech synthesis on preset target text data. According to the embodiment of the invention, the emotion diversity of speech synthesis can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, applicable to the field of financial technology, and in particular to an emotional speech synthesis method and device, electronic device and storage medium. Background Art

[0002] Emotional speech synthesis refers to the conversion of written text into speech with emotion. For example, in the scenario of answering financial questions in the financial field, intelligent robots can generate emotional speech data by performing emotional speech synthesis on the text of the answers to financial questions, which can improve the user experience when users consult financial issues.

[0003] At present, common emotional speech synthesis usually relies on pre-defined mapping relationships between speech features such as pitch, speed, pauses and different emotion types. However, this method is difficult to capture complex emotional changes, and the generated speech often appears unnatural and lacks diversity. Therefore, how to improve the emotional diversity of speech synthesis has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an emotional speech synthesis method and device, an electronic device and a storage medium, aiming to improve the emotional diversity of speech synthesis.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes an emotional speech synthesis method, the method comprising:

[0006] Acquire text sample data and voice sample data of the text sample data;

[0007] Performing model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model, wherein the target speech synthesis model includes universal parameter weights;

[0008] Performing emotion classification on the speech sample data to obtain emotion classification audio data;

[0009] Based on the emotion classification audio data, adjusting parameters of the target speech synthesis model to obtain a fine-tuning speech synthesis model, wherein the fine-tuning speech synthesis model includes fine-tuning parameter weights;

[0010] Based on the fine-tuning parameter weight, modify the general parameter weight to obtain the emotional parameter weight;

[0011] Based on the emotion parameter weights, adjusting the parameters of the target speech synthesis model to obtain an emotion speech synthesis model;

[0012] Based on the emotional speech synthesis model, emotional speech synthesis is performed on preset target text data.

[0013] In some embodiments, performing emotional speech synthesis on preset target text data based on the emotional speech synthesis model includes:

[0014] Based on the emotional speech synthesis model, the target text data is parsed to obtain text emotional features;

[0015] Based on the emotional speech synthesis model, encoding the target text data to obtain a speech encoding vector;

[0016] Based on the text emotional features, adjusting the parameters of the emotional speech synthesis model to obtain target parameter weights;

[0017] Based on the target parameter weight, audio generation is performed on the speech coding vector to obtain synthetic emotional speech data.

[0018] In some embodiments, the performing emotion classification on the speech sample data to obtain emotion classification audio data includes:

[0019] Performing emotion recognition on the speech sample data to obtain audio emotion data and emotion confidence, wherein the emotion confidence is used to characterize the accuracy of the audio emotion data;

[0020] Based on the audio emotion data, audio classification is performed on the speech sample data to obtain single emotion audio data, wherein the single emotion audio data has a single audio emotion data;

[0021] Based on the emotion confidence, the single emotion audio data is screened to obtain the emotion classification audio data.

[0022] In some embodiments, performing emotion recognition on the speech sample data to obtain audio emotion data and emotion confidence includes:

[0023] Performing spectrum conversion on the speech sample data to obtain a target Mel spectrum;

[0024] Extracting features from the target Mel spectrum to obtain audio features;

[0025] Performing feature activation on the audio feature to obtain the emotion confidence;

[0026] Based on the emotion confidence, the audio features are subjected to emotion screening to obtain the audio emotion data.

[0027] In some embodiments, the step of modifying the general parameter weight based on the fine-tuning parameter weight to obtain the emotion parameter weight includes:

[0028] Subtracting the general parameter weight from the fine-tuning parameter weight to obtain an initial emotion parameter weight;

[0029] Performing vector scaling on the initial emotion parameter weight to obtain a scaled emotion parameter weight;

[0030] The scaled emotion parameter weight and the general parameter weight are summed to obtain the emotion parameter weight.

[0031] In some embodiments, adjusting the parameters of the target speech synthesis model based on the emotion parameter weight to obtain the emotion speech synthesis model includes:

[0032] Based on the emotion parameter weights, the target speech synthesis model is subjected to parameter replacement to obtain a parameter-changed emotion speech synthesis model;

[0033] Model training is performed on the parameter-changing emotional speech synthesis model to obtain the emotional speech synthesis model.

[0034] In some embodiments, fine-tuning parameters of the target speech synthesis model based on the emotion classification audio data to obtain the fine-tuned speech synthesis model includes:

[0035] Performing emotion identification on the emotion classification audio data to obtain an emotion label;

[0036] Based on the sentiment classification audio data, performing text search on the text sample data to obtain sentiment classification text data;

[0037] Based on the emotion label, the emotion classification audio data and the emotion classification text data, the target speech synthesis model is trained to obtain the fine-tuning speech synthesis model.

[0038] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an emotional speech synthesis device, the device comprising:

[0039] A sample data acquisition module, used to acquire speech synthesis sample data, wherein the speech synthesis sample data includes text sample data and speech sample data of the text sample data;

[0040] A conventional model training module, used for performing model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model, wherein the target speech synthesis model includes universal parameter weights;

[0041] A speech data classification module is used to perform emotion classification on the speech sample data to obtain emotion classification audio data;

[0042] A model parameter adjustment module, used to adjust the parameters of the target speech synthesis model based on the emotion classification audio data to obtain a fine-tuning speech synthesis model, wherein the fine-tuning speech synthesis model includes a fine-tuning parameter weight;

[0043] An emotional parameter calculation module, used for modifying the general parameter weights based on the fine-tuning parameter weights to obtain emotional parameter weights;

[0044] A target model generation module, used to adjust the parameters of the target speech synthesis model based on the emotion parameter weights to obtain an emotional speech synthesis model;

[0045] Emotional speech synthesis module: used to perform emotional speech synthesis on preset target text data based on the emotional speech synthesis model to obtain synthesized emotional speech data.

[0046] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0047] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0048] The emotional speech synthesis method and device, electronic device and storage medium proposed in the present application obtain text sample data and its corresponding speech sample data, and then perform model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model including general parameter weights. Next, emotion classification is performed on the speech sample data to obtain emotion classification audio data, and then the emotion classification audio data is used to adjust the parameters of the target speech synthesis model to obtain a fine-tuning speech synthesis model including fine-tuning parameter weights. Then, the general parameter weights are changed based on the fine-tuning parameter weights to obtain the emotion parameter weights, and the parameters of the target speech synthesis model are adjusted accordingly to form an emotional speech synthesis model. Finally, the emotional speech synthesis model is used to perform emotional speech synthesis on the target text data, which can capture more complex emotional changes in the target text data, make the synthesized speech more natural, and improve the emotional diversity of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the emotional speech synthesis method provided by an embodiment of the present application;

[0050] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0051] Figure 3 yes Figure 2 Flow chart of step S201 in FIG.

[0052] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0053] Figure 5 yes Figure 1 Flow chart of step S105 in FIG.

[0054] Figure 6 yes Figure 1 Flow chart of step S106 in FIG.

[0055] Figure 7 is a flow chart of an emotional speech synthesis method provided by another embodiment of the present application;

[0056] Figure 8 is a schematic diagram of the structure of the emotional speech synthesis device provided in an embodiment of the present application;

[0057] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] First, some nouns involved in this application are analyzed:

[0062] Parameter weights: Parameter weights are a key component in machine learning and deep learning models. They represent adjustable values ​​within the model that affect the model's predictions or outputs. During training, these weights are optimized based on the input data and the expected output so that the model can accurately learn and simulate the complex relationships between the data. Adjustment of parameter weights directly affects the performance of the model. For example, in a speech synthesis model, different parameter weights can produce different sound characteristics, such as pitch, volume, and speaking speed, making the synthesized speech closer to the emotion and intonation of real human speech.

[0063] Confidence: Confidence is a measure of the model's certainty about its predictions, which indicates how much the model trusts its output. In various machine learning tasks, confidence can help us understand the reliability of model predictions. For example, in emotion recognition, high confidence means that the model is highly confident that the identified emotion label matches the emotional state in the actual audio, while low confidence means that the model is not sure about its predictions and may need more data or a more complex model to improve the accuracy of the prediction.

[0064] Emotional speech synthesis refers to the conversion of written text into speech with emotion. For example, in the scenario of answering financial questions in the financial field, intelligent robots can generate emotional speech data by performing emotional speech synthesis on the text of the answers to financial questions, which can improve the user experience when users consult financial issues.

[0065] At present, common emotional speech synthesis usually relies on pre-defined mapping relationships between speech features such as pitch, speed, pauses and different emotion types. However, this method is difficult to capture complex emotional changes, and the generated speech often appears unnatural and lacks diversity. Therefore, how to improve the emotional diversity of speech synthesis has become a technical problem that needs to be solved urgently.

[0066] Based on this, the embodiments of the present application provide an emotional speech synthesis method and device, an electronic device and a storage medium, aiming to improve the emotional diversity of speech synthesis.

[0067] The emotional speech synthesis method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the emotional speech synthesis method in the embodiments of the present application is described.

[0068] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0069] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0070] The emotional speech synthesis method provided in the embodiment of the present application relates to the field of speech synthesis technology. The emotional speech synthesis method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the emotional speech synthesis method, etc., but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] Figure 1 is an optional flowchart of the emotional speech synthesis method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0073] Step S101, obtaining text sample data and voice sample data of the text sample data;

[0074] Step S102, performing model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model, wherein the target speech synthesis model includes general parameter weights;

[0075] Step S103, performing emotion classification on the speech sample data to obtain emotion classification audio data;

[0076] Step S104, adjusting parameters of the target speech synthesis model based on the emotion classification audio data to obtain a fine-tuned speech synthesis model, wherein the fine-tuned speech synthesis model includes fine-tuning parameter weights;

[0077] Step S105, based on the fine-tuning parameter weight, modify the general parameter weight to obtain the emotional parameter weight;

[0078] Step S106, adjusting parameters of the target speech synthesis model based on the emotion parameter weights to obtain an emotion speech synthesis model;

[0079] Step S107: performing emotional speech synthesis on the preset target text data based on the emotional speech synthesis model.

[0080] In steps S101 to S107 shown in the embodiment of the present application, text sample data and speech sample data of the text sample data are obtained, and then a preset initial speech synthesis model is trained according to the text sample data and the number of speech samples to obtain a target speech synthesis model, wherein the target speech synthesis model includes general parameter weights, and further, emotion classification is performed on the speech sample data to obtain emotion classification audio data, and then, according to the emotion classification audio data, parameters of the target speech synthesis model are adjusted to obtain a fine-tuning speech synthesis model, wherein the fine-tuning speech synthesis model includes fine-tuning parameter weights, and then, according to the fine-tuning parameter weights, parameter weights of the general parameter weights are changed to obtain emotion parameter weights, and then, according to the emotion parameter weights, parameters of the target speech synthesis model are adjusted to obtain an emotion speech synthesis model, and finally, according to the emotion speech synthesis model, emotion speech synthesis is performed on the preset target text data. Therefore, the present application obtains text sample data and its corresponding speech sample data, and then performs model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model including general parameter weights. Next, emotion classification is performed on the speech sample data to obtain emotion classification audio data. The emotion classification audio data is then used to adjust the parameters of the target speech synthesis model to obtain a fine-tuned speech synthesis model including fine-tuning parameter weights. Then, the general parameter weights are changed based on the fine-tuning parameter weights to obtain the emotion parameter weights, and the parameters of the target speech synthesis model are adjusted accordingly to form an emotion speech synthesis model. Finally, the emotion speech synthesis model is used to perform emotion speech synthesis on the target text data, which can capture more complex emotion changes in the target text data, making the synthesized speech more natural and improving the emotional diversity of speech synthesis.

[0081] In step S101 of some embodiments, text sample data refers to written text data used to train the model, which contains text information to be converted into speech. Speech sample data refers to actual speech recordings corresponding to the text sample data, which are used to train the model to imitate the way humans speak.

[0082] The present application may collect text and voice data from public datasets to form text sample data and voice sample data, or may collect text and voice data from user uploads or professional recordings to form text sample data and voice sample data, thereby ensuring the diversity of the text sample data and voice sample data. It should be noted that the text sample data and voice sample data cover different emotions, intonations, speaking speeds and other features.

[0083] In step S102 of some embodiments, the initial speech synthesis model refers to a preset model used for speech synthesis before any training. It should be noted that the initial speech synthesis model usually includes some basic speech synthesis capabilities, but the accuracy of speech synthesis is not high.

[0084] The present application can utilize deep learning technology to train an initial speech synthesis model to learn the mapping relationship between text sample data and speech sample data, and during the learning process, adjust the parameters of the initial speech synthesis model until the initial speech synthesis model can generate accurate speech output, thereby obtaining a target speech synthesis model containing universal parameter weights.

[0085] In step S103 of some embodiments, the speech sample data is analyzed by emotion recognition technology to determine its emotion attributes and corresponding confidence levels. The confidence levels are used to measure the reliability of the recognition results. Next, based on the identified emotion attributes, the speech sample data are classified into single emotion audio data with a single emotion. Finally, these single emotion audio data are screened according to the emotion confidence levels to obtain emotion classification audio data.

[0086] For details, see Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S203:

[0087] Step S201, performing emotion recognition on the speech sample data to obtain audio emotion data and emotion confidence, wherein the emotion confidence is used to characterize the accuracy of the audio emotion data;

[0088] Step S202, based on the audio emotion data, performing audio classification on the speech sample data to obtain single emotion audio data, wherein the single emotion audio data has a single audio emotion data;

[0089] Step S203: based on the emotion confidence, the single emotion audio data is screened to obtain emotion classification audio data.

[0090] In step S201 of some embodiments, emotion recognition refers to parsing speech sample data using an emotion recognition algorithm to determine the type of emotion contained in the speech sample data.

[0091] This application first performs a spectrum conversion operation on the collected speech sample data to convert the original audio signal into a target Mel spectrum, then extracts key audio features from the target Mel spectrum, and then analyzes the extracted audio features through a feature activation process to obtain emotion confidence. Finally, the audio features are emotion-screened based on the emotion confidence to ensure that only those audio features with sufficiently high confidence are identified as valid audio emotion data, thereby obtaining accurate emotion classification results.

[0092] For details, see Figure 3 In some embodiments, step S201 may include but is not limited to steps S301 to S304:

[0093] Step S301, performing spectrum conversion on the speech sample data to obtain a target Mel spectrum;

[0094] Step S302, extracting features from the target Mel spectrum to obtain audio features;

[0095] Step S303, performing feature activation on the audio features to obtain emotion confidence;

[0096] Step S304: Based on the emotion confidence, the audio features are subjected to emotion screening to obtain audio emotion data.

[0097] In step S301 of some embodiments, spectrum conversion refers to converting the speech sample data from the time domain to the frequency domain so as to analyze the frequency components of the speech.

[0098] The present application obtains standard speech data by preprocessing speech sample data, wherein the preprocessing includes denoising and normalization, and then converts the standard speech data from a time series into a frequency series using Fourier transform, and then converts the frequency series into a Mel frequency scale using a Mel filter group, thereby obtaining a target Mel spectrum.

[0099] In step S302 of some embodiments, feature extraction refers to extracting parameters that can represent speech features from the Mel spectrum, and these parameters can capture emotional information in the speech.

[0100] The present application can obtain audio features by extracting features such as Mel-frequency cepstral coefficients, pitch, intensity, and duration from the target Mel spectrum.

[0101] In step S303 of some embodiments, feature activation refers to processing audio features through an activation function to obtain emotion confidence.

[0102] This application inputs the extracted audio features into a classifier, such as a support vector machine or a decision tree. The classifier learns the mapping relationship between the audio features and emotions based on the training data, and assigns an emotion label and confidence, i.e., emotion confidence, to the audio features.

[0103] In step S304 of some embodiments, the emotion tags whose emotion confidence is greater than a preset emotion confidence threshold are retained and used as the emotion data of the speech sample data, thereby obtaining the audio emotion data.

[0104] In steps S301 to S304 shown in this embodiment, the original audio signal is converted into a target Mel spectrum by performing a spectrum conversion operation on the collected speech sample data. This process involves converting the time domain signal of the sound into a frequency domain signal in order to better capture the characteristics of the sound. Next, key audio features are extracted from the target Mel spectrum. These features can represent the emotional information in the speech signal. Then, through the feature activation process, the extracted audio features are analyzed to obtain the emotional confidence. This confidence reflects the strength of the association between the audio features and the specific emotions. Finally, the audio features are emotionally screened based on the emotional confidence to ensure that only those audio features with sufficiently high confidence are identified as valid audio emotional data, thereby obtaining accurate emotion classification results. This process ensures the accuracy and reliability of emotion recognition and provides a high-quality data foundation for subsequent emotional speech synthesis.

[0105] In step S202 of some embodiments, audio classification refers to classifying the speech sample data into corresponding emotion categories according to the recognized audio emotion data.

[0106] The present application obtains single-emotion audio data representing single audio emotion data by using the identified audio emotion data as an emotion label and then classifying the corresponding speech sample data into a preset emotion category according to the emotion label. For example, speech sample data with the audio emotion data of "happy" can be classified into the single-emotion audio data of "happy".

[0107] It is important to know that after the classification is completed, the speech sample data in each category is checked to ensure that each speech sample data contains only one emotion. If the speech sample data is found to contain multiple emotions, it is removed from the single emotion audio data.

[0108] In step S203 of some embodiments, data screening refers to selecting audio data with confidence levels higher than a specific threshold based on emotion confidence levels.

[0109] This application sets a threshold for emotion confidence, and then selects speech sample data with emotion confidence higher than the threshold from the single emotion audio data for storage, so as to obtain emotion classification audio data.

[0110] In steps S201 to S203 shown in this embodiment, the speech sample data is analyzed by emotion recognition technology to determine its emotion attributes and corresponding confidence levels. The confidence levels are used to measure the reliability of the recognition results. Next, based on the identified emotion attributes, the speech sample data are classified into single emotion audio data with a single emotion. Finally, these single emotion audio data are screened based on the emotion confidence levels to ensure that the screened emotion classification audio data are both accurate and reliable. This screening process helps to improve the training effect of the emotion speech synthesis model, enabling it to more realistically simulate human emotions and enhance the naturalness and diversity of speech synthesis.

[0111] In step S104 of some embodiments, emotion labels are obtained by emotion identification of emotion classification audio data. Subsequently, text search operations are performed on text sample data using the emotion classification audio data to obtain emotion classification text data that matches the audio data. Finally, the target speech synthesis model is carefully trained in combination with the emotion labels, emotion classification audio data, and emotion classification text data. Through this training process, the parameters of the model are optimized, thereby generating a fine-tuned speech synthesis model that can more accurately simulate emotional information.

[0112] For details, see Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S403:

[0113] Step S401, emotion identification is performed on the emotion classification audio data to obtain an emotion label;

[0114] Step S402, based on the sentiment classification audio data, performing text search on the text sample data to obtain sentiment classification text data;

[0115] Step S403, based on the emotion label, the emotion classification audio data and the emotion classification text data, the target speech synthesis model is trained to obtain a fine-tuned speech synthesis model.

[0116] In step S401 of some embodiments, emotion identification refers to assigning a label representing the emotion category of the audio data.

[0117] The present application can predefine a set of emotion categories, and then analyze audio features (such as pitch, rhythm, intensity) to determine the emotion category of the audio data and assign corresponding emotion tags to it.

[0118] In step S402 of some embodiments, text search refers to finding text matching the sentiment classification audio data in the text sample data.

[0119] Before performing text search, the present application needs to create an index for each audio sample data and record its corresponding text sample data. Then, based on the index, text sample data matching the sentiment classification audio data, i.e. sentiment classification text data, can be searched.

[0120] In step S403 of some embodiments, model training refers to using emotion tags, emotion classification audio data, and emotion classification text data to adjust and optimize the speech synthesis model so that it can generate speech with specific emotion characteristics.

[0121] This application inputs sentiment classification audio data and corresponding sentiment classification text data as training samples into a target speech synthesis model, and then, by adjusting the model parameters, enables the target speech synthesis model to learn how to generate sentiment classification audio data with corresponding sentiment features based on the sentiment classification text data, thereby obtaining a fine-tuned speech synthesis model.

[0122] In steps S401 to S403 shown in the present embodiment, emotion labels are obtained by emotion identification of emotion classification audio data. This process involves identifying emotion features in the audio and matching them with specific emotion categories. Subsequently, these emotion classification audio data are used to perform text search operations on the text sample data to obtain emotion classification text data that matches the audio data. Finally, the target speech synthesis model is carefully trained in combination with the emotion labels, emotion classification audio data, and emotion classification text data. Through this training process, the parameters of the model are optimized, thereby generating a fine-tuned speech synthesis model that can more accurately simulate human emotions.

[0123] In step S105 of some embodiments, the fine-tuning parameter weight is subtracted from the general parameter weight to obtain the initial emotional parameter weight, and then a vector scaling operation is performed on the obtained initial emotional parameter weight to adjust its numerical value, thereby obtaining the scaled emotional parameter weight. Finally, the scaled emotional parameter weight is summed with the general parameter weight to combine the effects of the two to generate the final emotional parameter weight.

[0124] For details, see Figure 5 In some embodiments, step S105 may include but is not limited to steps S501 to S503:

[0125] Step S501, performing a subtraction operation on the fine-tuning parameter weight and the general parameter weight to obtain an initial emotion parameter weight;

[0126] Step S502, performing vector scaling on the initial emotion parameter weights to obtain scaled emotion parameter weights;

[0127] Step S503, summing the scaled emotion parameter weight and the general parameter weight to obtain the emotion parameter weight.

[0128] In step S501 of some embodiments, the initial emotional parameter weights can be obtained by subtracting the corresponding general parameter weights from each value of the fine-tuning parameter weights. For example, assuming that the fine-tuning parameter weights are [0.5, 0.3, 0.2] and the general parameter weights are [0.4, 0.2, 0.4]. After the subtraction operation, the initial emotional parameter weights obtained are [0.1, 0.1, -0.2].

[0129] In step S502 of some embodiments, vector scaling refers to adjusting the numerical value of the initial emotion parameter weight.

[0130] The present application can achieve vector scaling of the initial emotional parameter weights by multiplying each value of the initial emotional parameter weights by a scaling factor to obtain scaled emotional parameter weights. For example, if the initial emotional parameter weights are [0.1, 0.1, -0.2], and a scaling factor of 2 is selected, the scaled emotional parameter weights become [0.2, 0.2, -0.4].

[0131] In step S503 of some embodiments, summing the parameter weights refers to adding the scaled emotion parameter weights to the general parameter weights to integrate the effects of the two.

[0132] This application obtains the emotional parameter weight by adding each value of the scaled emotional parameter weight to the value of the corresponding general parameter weight. For example, assuming that the scaled emotional parameter weight is [0.2, 0.2, -0.4] and the general parameter weight is [0.4, 0.2, 0.4]. After the summation operation, the obtained emotional parameter weight is [0.6, 0.4, 0.0].

[0133] In steps S501 to S503 shown in this embodiment, the fine-tuning parameter weights are subtracted from the general parameter weights by performing a subtraction operation to obtain the initial emotion parameter weights. This step reveals the difference between the two weight sets. Subsequently, a vector scaling operation is performed on the initial emotion parameter weights to adjust their values ​​to obtain the scaled emotion parameter weights. This process is intended to amplify or reduce the differences as necessary. Finally, the scaled emotion parameter weights are added to the general parameter weights, and the effects of the two are combined through the parameter weight summation operation to generate the final emotion parameter weights, so that the speech synthesis model can more accurately capture and express the subtle changes in human emotions, thereby improving the naturalness and emotional expression ability of the synthesized speech.

[0134] In step S106 of some embodiments, the parameters in the target speech synthesis model are replaced according to the emotion parameter weights to generate a parameter-changed emotion speech synthesis model. Subsequently, the parameter-changed emotion speech synthesis model is further optimized through model training so that it can simulate and express emotions more accurately, and finally form an emotion speech synthesis model that can output speech with rich emotional colors.

[0135] For details, see Figure 6 In some embodiments, step S106 may include but is not limited to steps S601 to S602:

[0136] Step S601, based on the emotion parameter weights, replace the parameters of the target speech synthesis model to obtain a parameter-replaced emotion speech synthesis model;

[0137] Step S602, performing model training on the parameter-changing emotional speech synthesis model to obtain an emotional speech synthesis model.

[0138] In step S601 of some embodiments, parameter replacement refers to replacing the original parameter weights in the target speech synthesis model with emotion parameter weights so that the model can express specific emotions.

[0139] This application first determines the parameters that need to be adjusted in the target speech synthesis model, and then replaces the values ​​of these parameters with the values ​​of the emotion parameter weights to obtain a parameter-changed emotion speech synthesis model.

[0140] In step S602 of some embodiments, model training refers to optimizing the parameters of the parameter-changing emotional speech synthesis model through a large amount of data and an iterative process so that it can more accurately generate speech with specific emotional characteristics.

[0141] This application inputs audio data with emotion labels and corresponding text data as training sets into the parameter-changing emotional speech synthesis model. The parameter-changing emotional speech synthesis model learns these data and adjusts internal parameters to minimize the difference between the output speech and the target audio, thereby obtaining an emotional speech synthesis model.

[0142] In steps S601 to S602 shown in the present embodiment, the parameters in the target speech synthesis model are replaced according to the emotion parameter weights, thereby generating a parameter-changed emotion speech synthesis model. This process involves updating the original parameter weights to new weights that can reflect specific emotion characteristics. Subsequently, the parameter-changed emotion speech synthesis model is further optimized through model training so that it can simulate and express emotions more accurately, and finally form an emotion speech synthesis model that can output speech with rich emotion colors, so that the emotion speech synthesis model can more accurately simulate and reproduce the subtle differences in human emotions, thereby generating more natural and expressive speech output, and improving the user experience and interaction quality of speech synthesis technology in various application scenarios.

[0143] In step S107 of some embodiments, the emotional speech synthesis model can be used to parse the preset target text data to identify and extract text emotional features in the text. Subsequently, the emotional speech synthesis model performs encoding processing on the target text data to convert it into a speech coding vector. Next, the emotional speech synthesis model adjusts parameters based on the extracted text emotional features to obtain target parameter weights that are adapted to the emotion of the target text. Finally, the emotional speech synthesis model generates audio for the speech coding vector using the target parameter weights, thereby producing synthetic emotional speech data that matches the emotion of the target text data.

[0144] For details, see Figure 7 In some embodiments, step S107 may include but is not limited to steps S701 to S704:

[0145] Step S701, based on the emotional speech synthesis model, performing text analysis on the target text data to obtain text emotional features;

[0146] Step S702, encoding the target text data based on the emotional speech synthesis model to obtain a speech coding vector;

[0147] Step S703, adjusting parameters of the emotional speech synthesis model based on the text emotional features to obtain target parameter weights;

[0148] Step S704: Based on the target parameter weights, audio is generated for the speech coding vector to obtain synthesized emotional speech data.

[0149] In step S701 of some embodiments, text parsing refers to analyzing text data to identify and extract sentiment features therein.

[0150] The present application can identify the sentiment tendency in the target text data through natural language processing technology, such as sentiment analysis. It should be noted that natural language processing technology includes steps such as identifying keywords, phrases or sentence structures.

[0151] In step S702 of some embodiments, the encoding process refers to converting the text data into a numerical form that can be understood by the model, that is, a speech encoding vector.

[0152] The present application first converts the target text data into a series of phonemes or phonemes, and then further converts the phonemes or phonemes into coding vectors, thereby realizing the encoding processing of the target text data using the emotional speech synthesis model to obtain the speech coding vector.

[0153] In step S703 of some embodiments, parameter adjustment refers to adjusting the parameters of the speech synthesis model according to the emotional characteristics of the text so as to generate speech with corresponding emotional coloring.

[0154] The present application determines the text emotion of the target text data by performing feature analysis on the text emotion features, thereby determining the emotion type required for the generated synthetic speech, and then according to the emotion type, makes corresponding adjustments to the parameters in the emotional speech synthesis model, thereby obtaining the target parameter weights that are suitable for the text emotion of the target text data.

[0155] In step S704 of some embodiments, audio generation refers to converting the speech coding vector into an actual audio signal using the adjusted parameter weights.

[0156] The present application can utilize an emotional speech synthesis model to convert a speech coding vector into waveform data, and then utilize waveform synthesis technology to convert the waveform into an audio signal, thereby generating synthesized emotional speech data.

[0157] In steps S701 to S704 shown in this embodiment, the target text data is parsed using the emotional speech synthesis model to identify and extract the emotional features in the text, then the target text is encoded using the emotional speech synthesis model to generate a corresponding speech encoding vector, then, according to the extracted text emotional features, the emotional speech synthesis model is used to adjust its parameters to obtain a target parameter weight suitable for generating a specific emotional speech, and finally, according to the target parameter weight, the emotional speech synthesis model generates audio for the speech encoding vector, thereby generating synthetic emotional speech data that matches the text content. This process enables speech synthesis to not only convey information, but also express corresponding emotions, thereby improving the application scope and user experience of speech synthesis.

[0158] The present application obtains text sample data and its corresponding speech sample data, and then performs model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model including general parameter weights. Next, emotion classification is performed on the speech sample data to obtain emotion classification audio data. The emotion classification audio data is then used to adjust the parameters of the target speech synthesis model to obtain a fine-tuned speech synthesis model including fine-tuning parameter weights. Then, the general parameter weights are changed based on the fine-tuning parameter weights to obtain the emotion parameter weights, and the parameters of the target speech synthesis model are adjusted accordingly to form an emotion speech synthesis model. Finally, the emotion speech synthesis model is used to perform emotion speech synthesis on the target text data, which can capture more complex emotion changes in the target text data, make the synthesized speech more natural, and improve the emotion diversity of speech synthesis.

[0159] See also Figure 8 The embodiment of the present application also provides an emotional speech synthesis device, which can implement the above-mentioned emotional speech synthesis method, and the device includes:

[0160] The sample data acquisition module 801 is used to acquire speech synthesis sample data, wherein the speech synthesis sample data includes text sample data and speech sample data of the text sample data;

[0161] A conventional model training module 802 is used to perform model training on a preset initial speech synthesis model based on text sample data and speech sample data to obtain a target speech synthesis model, wherein the target speech synthesis model includes general parameter weights;

[0162] The speech data classification module 803 is used to perform emotion classification on the speech sample data to obtain emotion classification audio data;

[0163] A model parameter adjustment module 804 is used to adjust the parameters of the target speech synthesis model based on the emotion classification audio data to obtain a fine-tuned speech synthesis model, wherein the fine-tuned speech synthesis model includes fine-tuning parameter weights;

[0164] The emotional parameter calculation module 805 is used to modify the general parameter weights based on the fine-tuning parameter weights to obtain the emotional parameter weights;

[0165] A target model generation module 806 is used to adjust the parameters of the target speech synthesis model based on the emotion parameter weights to obtain an emotional speech synthesis model;

[0166] The emotional speech synthesis module 807 is used to perform emotional speech synthesis on preset target text data based on the emotional speech synthesis model to obtain synthesized emotional speech data.

[0167] The specific implementation of the emotional speech synthesis device is basically the same as the specific implementation of the above-mentioned emotional speech synthesis method, and will not be repeated here.

[0168] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned emotional speech synthesis method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0169] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0170] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0171] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the emotional speech synthesis method of the embodiment of this application;

[0172] Input / output interface 903, used to implement information input and output;

[0173] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0174] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0175] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0176] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned emotional speech synthesis method is implemented.

[0177] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0178] The emotional speech synthesis method, emotional speech synthesis device, electronic device and storage medium provided in the embodiments of the present application obtain text sample data and speech sample data of the text sample data, and then perform model training on a preset initial speech synthesis model according to the text sample data and the number of speech samples to obtain a target speech synthesis model, wherein the target speech synthesis model includes general parameter weights, and further, emotion classification is performed on the speech sample data to obtain emotion classification audio data, and then parameters of the target speech synthesis model are adjusted according to the emotion classification audio data to obtain a fine-tuning speech synthesis model, wherein the fine-tuning speech synthesis model includes fine-tuning parameter weights, and then, according to the fine-tuning parameter weights, parameter weights of the general parameter weights are changed to obtain emotion parameter weights, and then, according to the emotion parameter weights, parameters of the target speech synthesis model are adjusted to obtain an emotional speech synthesis model, and finally, according to the emotional speech synthesis model, emotional speech synthesis is performed on the preset target text data.

[0179] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0180] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0181] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0183] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0184] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0185] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0186] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0188] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0189] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An emotional speech synthesis method, characterized in that: The method comprises: Acquire text sample data and voice sample data of the text sample data; Performing model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model, wherein the target speech synthesis model includes universal parameter weights; Performing emotion classification on the speech sample data to obtain emotion classification audio data; Based on the emotion classification audio data, adjusting parameters of the target speech synthesis model to obtain a fine-tuning speech synthesis model, wherein the fine-tuning speech synthesis model includes fine-tuning parameter weights; Based on the fine-tuning parameter weight, modify the general parameter weight to obtain the emotional parameter weight; Based on the emotion parameter weights, adjusting the parameters of the target speech synthesis model to obtain an emotion speech synthesis model; Based on the emotional speech synthesis model, emotional speech synthesis is performed on preset target text data.

2. The method according to claim 1, characterized in that The step of performing emotional speech synthesis on preset target text data based on the emotional speech synthesis model includes: Based on the emotional speech synthesis model, the target text data is parsed to obtain text emotional features; Based on the emotional speech synthesis model, encoding the target text data to obtain a speech encoding vector; Based on the text emotional features, adjusting the parameters of the emotional speech synthesis model to obtain target parameter weights; Based on the target parameter weight, audio generation is performed on the speech coding vector to obtain synthetic emotional speech data.

3. The method according to claim 1, characterized in that The performing emotion classification on the speech sample data to obtain emotion classification audio data includes: Performing emotion recognition on the speech sample data to obtain audio emotion data and emotion confidence, wherein the emotion confidence is used to characterize the accuracy of the audio emotion data; Based on the audio emotion data, audio classification is performed on the speech sample data to obtain single emotion audio data, wherein the single emotion audio data has a single audio emotion data; Based on the emotion confidence, the single emotion audio data is screened to obtain the emotion classification audio data.

4. The method according to claim 3, characterized in that The performing emotion recognition on the speech sample data to obtain audio emotion data and emotion confidence includes: Performing spectrum conversion on the speech sample data to obtain a target Mel spectrum; Extracting features from the target Mel spectrum to obtain audio features; Performing feature activation on the audio feature to obtain the emotion confidence; Based on the emotion confidence, the audio features are subjected to emotion screening to obtain the audio emotion data.

5. The method according to claim 1, characterized in that The step of modifying the general parameter weight based on the fine-tuning parameter weight to obtain the emotion parameter weight includes: Subtracting the general parameter weight from the fine-tuning parameter weight to obtain an initial emotion parameter weight; Performing vector scaling on the initial emotion parameter weight to obtain a scaled emotion parameter weight; The scaled emotion parameter weight and the general parameter weight are summed to obtain the emotion parameter weight.

6. The method according to any one of claims 1 to 5, characterized in that: The step of adjusting the parameters of the target speech synthesis model based on the emotion parameter weight to obtain the emotion speech synthesis model includes: Based on the emotion parameter weights, the target speech synthesis model is subjected to parameter replacement to obtain a parameter-changed emotion speech synthesis model; Model training is performed on the parameter-changing emotional speech synthesis model to obtain the emotional speech synthesis model.

7. The method according to any one of claims 1 to 5, characterized in that: The step of fine-tuning parameters of the target speech synthesis model based on the emotion classification audio data to obtain a fine-tuned speech synthesis model includes: Performing emotion identification on the emotion classification audio data to obtain an emotion label; Based on the sentiment classification audio data, performing text search on the text sample data to obtain sentiment classification text data; Based on the emotion label, the emotion classification audio data and the emotion classification text data, the target speech synthesis model is trained to obtain the fine-tuning speech synthesis model.

8. An emotional speech synthesis device, characterized in that: The device comprises: A sample data acquisition module, used to acquire speech synthesis sample data, wherein the speech synthesis sample data includes text sample data and speech sample data of the text sample data; A conventional model training module, used for performing model training on a preset initial speech synthesis model based on the text sample data and the speech sample data to obtain a target speech synthesis model, wherein the target speech synthesis model includes universal parameter weights; A speech data classification module is used to perform emotion classification on the speech sample data to obtain emotion classification audio data; A model parameter adjustment module, used to adjust the parameters of the target speech synthesis model based on the emotion classification audio data to obtain a fine-tuning speech synthesis model, wherein the fine-tuning speech synthesis model includes a fine-tuning parameter weight; An emotional parameter calculation module, used for modifying the general parameter weights based on the fine-tuning parameter weights to obtain emotional parameter weights; A target model generation module, used to adjust the parameters of the target speech synthesis model based on the emotion parameter weights to obtain an emotional speech synthesis model; Emotional speech synthesis module: used to perform emotional speech synthesis on preset target text data based on the emotional speech synthesis model to obtain synthesized emotional speech data.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the emotional speech synthesis method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the emotional speech synthesis method according to any one of claims 1 to 7 is implemented.

Images