Speech synthesis method and device, computer device, and storage medium

Abstract

Embodiments of the present application provide a speech synthesis method and device, computer equipment and a storage medium, belonging to the technical field of financial technology. The method comprises: obtaining audio training data, target text and audio reference data; inputting the audio training data into a preset original emotion intensity prediction model to perform emotion intensity prediction to obtain a preliminary emotion intensity; adjusting parameters of the original emotion intensity prediction model according to the preliminary emotion intensity to obtain a target emotion intensity prediction model; inputting the audio reference data into a preset emotion encoder to perform feature coding to obtain an emotion feature vector; inputting the audio reference data into the target emotion intensity prediction model to perform emotion intensity prediction to obtain a target emotion intensity; and performing speech synthesis on the emotion feature vector, the target emotion intensity and the target text to obtain target speech data. The embodiments of the present application can generate speech data with emotion intensity, and the emotion of the generated speech data is more abundant and the human cost of manual identification is saved.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a speech synthesis method and apparatus, computer equipment and storage medium. Background Technology

[0002] With the development of computers, an increasing number of financial institutions are providing intelligent customer service to answer customers' financial inquiries. Intelligent customer service utilizes speech synthesis technology to achieve fluent speech, a technique that converts text into speech. In related technologies, to improve the quality of financial services and enhance the customer experience when seeking financial advice, speech synthesis methods generate speech with specific emotions. However, the generated speech often lacks emotional depth. To enrich the emotional content of the speech, the emotional intensity of each training data point can be pre-labeled manually, and speech synthesis can then achieve different emotional intensities based on the training data. However, this requires significant manpower. Therefore, how to enrich the emotional content of speech without increasing labor costs has become a pressing technical problem to be solved. Summary of the Invention

[0003] The main objective of this application is to provide a speech synthesis method and apparatus, computer equipment and storage medium, which aims to enrich the emotion of speech and reduce labor costs.

[0004] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, the method comprising:

[0005] Acquire speech synthesis reference data; wherein, the speech synthesis reference data includes: audio training data, target text, and audio reference data;

[0006] The audio training data is input into a preset original emotion intensity prediction model to predict the emotion intensity and obtain a preliminary emotion intensity.

[0007] Based on the preliminary emotional intensity, the parameters of the original emotional intensity prediction model are adjusted to obtain the target emotional intensity prediction model;

[0008] The audio reference data is input into a preset emotion encoder for feature encoding to obtain an emotion feature vector;

[0009] The audio reference data is input into the target emotion intensity prediction model to predict the emotion intensity, thereby obtaining the target emotion intensity.

[0010] The emotional feature vector, the target emotional intensity, and the target text are combined to synthesize speech data to obtain target speech data.

[0011] In some embodiments, adjusting the parameters of the original emotion intensity prediction model based on the preliminary emotion intensity to obtain the target emotion intensity prediction model includes:

[0012] The initial emotion category is obtained by acquiring the emotion category of the initial emotion intensity; wherein, the initial emotion category includes: a neutral emotion category without emotion and a positive and negative emotion category with emotion;

[0013] Obtain the preliminary emotional intensity of the neutral emotional category to obtain the reference emotional intensity; obtain the preliminary emotional intensity of the positive and negative emotional categories to obtain the current emotional intensity;

[0014] If the current emotional intensity is greater than or equal to the reference emotional intensity, then the current emotional intensity is used as the candidate emotional intensity.

[0015] Based on the candidate emotion intensities, emotion intensity loss is calculated to obtain the emotion intensity loss dataset;

[0016] The original emotional intensity prediction model is adjusted based on the emotional intensity loss dataset to obtain the target emotional intensity prediction model.

[0017] In some embodiments, the emotion encoder includes a feature extraction layer and an encoding layer; the step of inputting the audio reference data into a preset emotion encoder for feature encoding to obtain an emotion feature vector includes:

[0018] The audio reference data is input into the feature extraction layer to extract emotional features, thereby obtaining audio emotional features;

[0019] The audio emotional features are input into the encoding layer for feature encoding processing to obtain the emotional feature vector.

[0020] In some embodiments, the step of synthesizing the emotion feature vector, the target emotion intensity, and the target text to obtain target speech data includes:

[0021] The emotional feature vector, the target emotional intensity, and the target text are input into a preset target speech synthesis model; wherein, the target speech synthesis model includes: a phoneme coding network, a speech parameter predictor, a decoder, and a vocoder;

[0022] The target text is phonetically encoded using the phoneme encoding network to obtain a phoneme vector.

[0023] The speech parameter predictor performs speech parameter prediction on the phoneme vector, the emotion feature vector and the target emotion intensity to obtain the target speech parameters.

[0024] The target speech parameters are decoded by the decoder to obtain speech spectrum data;

[0025] The vocoder performs audio conversion on the speech spectrum data to obtain the target speech data.

[0026] In some embodiments, the phoneme coding network includes: a phoneme conversion layer and an encoder; the step of performing phoneme coding processing on the target text through the phoneme coding network to obtain a phoneme vector includes:

[0027] The target text is converted into phonemes through the phoneme conversion layer to obtain text phoneme information;

[0028] The encoder encodes the text phoneme information to obtain the phoneme vector.

[0029] In some embodiments, before inputting the emotion feature vector, the target emotion intensity, and the target text into a preset target speech synthesis model, the method further includes:

[0030] Constructing the target speech synthesis model specifically includes:

[0031] Acquire speech synthesis training data; wherein, the speech synthesis training data includes: training feature vectors, training emotion intensity, and training text;

[0032] The training feature vector, the training emotion intensity, and the training text are input into a preset original speech synthesis model to perform speech synthesis and obtain speech prediction data.

[0033] The parameters of the original speech synthesis model are adjusted based on the speech prediction data and the preset speech reference data to obtain the target speech synthesis model.

[0034] In some embodiments, adjusting the parameters of the original speech synthesis model based on the speech prediction data and preset speech reference data to obtain the target speech synthesis model includes:

[0035] The speech prediction data and the speech reference data are subjected to loss calculation to obtain synthetic loss data;

[0036] The parameters of the original speech synthesis model are adjusted based on the synthesis loss data to obtain the target speech synthesis model.

[0037] To achieve the above objectives, a second aspect of this application provides a speech synthesis apparatus, the apparatus comprising:

[0038] The data acquisition module is used to acquire speech synthesis reference data; wherein, the speech synthesis reference data includes: audio training data, target text, and audio reference data;

[0039] The preliminary emotion intensity prediction module is used to input the audio training data into a preset original emotion intensity prediction model to predict the emotion intensity and obtain the preliminary emotion intensity.

[0040] The parameter adjustment module is used to adjust the parameters of the original emotion intensity prediction model based on the preliminary emotion intensity to obtain the target emotion intensity prediction model.

[0041] The feature encoding module is used to input the audio reference data into a preset emotion encoder for feature encoding to obtain an emotion feature vector;

[0042] The target emotion intensity prediction module is used to input the audio reference data into the target emotion intensity prediction model to predict the emotion intensity and obtain the target emotion intensity.

[0043] The speech synthesis module is used to synthesize the emotion feature vector, the target emotion intensity, and the target text to obtain target speech data.

[0044] To achieve the above objectives, a third aspect of the present application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0045] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0046] The speech synthesis method, apparatus, computer device, and storage medium proposed in this application predict the target emotional intensity of audio reference data using a target emotional intensity prediction model, thereby achieving automatic emotional intensity labeling. An emotional encoder encodes the audio reference data to obtain an emotional feature vector, and then speech synthesis is performed based on the emotional feature vector, the target emotional intensity, and the target text to synthesize target speech data with specific emotional intensities. Therefore, constructing a target emotional intensity prediction model capable of predicting emotional intensity, and then generating target speech data with emotional intensity from the automatically predicted emotional intensity, can accurately generate speech data with emotional intensity, enriching the emotional content of the speech data, while saving the labor cost of manually labeling emotional intensity. Furthermore, in the financial services field, generating speech data with different emotional intensities to answer customers' financial business questions can improve the customer experience. Attached Figure Description

[0047] Figure 1 This is a flowchart of the speech synthesis method provided in the embodiments of this application;

[0048] Figure 2 yes Figure 1 The flowchart of step S103 in the process;

[0049] Figure 3 yes Figure 1 The flowchart of step S104 in the process;

[0050] Figure 4 yes Figure 1 The flowchart of step S106 in the process;

[0051] Figure 5 yes Figure 4 The flowchart of step S402 in the document;

[0052] Figure 6 This is a flowchart of a speech synthesis method provided in another embodiment of this application;

[0053] Figure 7 yes Figure 6 The flowchart of step S603 in the process;

[0054] Figure 8 This is a schematic diagram of the speech synthesis device provided in the embodiments of this application;

[0055] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] First, let's analyze some of the terms used in this application:

[0060] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0061] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. NLP involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0062] Speech synthesis, also known as text-to-speech (TTS), can convert any text information into standard, fluent speech in real time. It is widely used in human-computer voice interaction technology. Speech synthesis involves multiple disciplines such as acoustics, linguistics, digital signal processing, and computer science, and its main challenge is converting text information into sound information. In short, speech synthesis is a technology that converts text into speech data.

[0063] A vocoder is a speech analysis and synthesis system for speech signals. During transmission, a vocoder only uses model parameters; during encoding and decoding, it uses model parameter estimation and speech synthesis techniques to generate the speech signal. A vocoder is an encoder and decoder that analyzes and synthesizes speech; it is also called a speech analysis and synthesis system or a speech band compression system.

[0064] With the development of computers, more and more financial institutions are offering intelligent customer service to answer customers' financial inquiries. These intelligent customer service systems utilize speech synthesis technology to enable fluent speech.

[0065] In related technologies, speech synthesis can generate speech data with specific emotions, but it lacks intensity control for each emotion. To achieve emotion intensity control, training data is typically pre-labeled with emotion intensity by humans, and then the speech synthesis model is trained using this data. The resulting model generates speech data with emotional intensity, enabling intelligent customer service to output emotionally charged speech to answer different types of financial inquiries from customers. However, manually labeling the training data with emotion intensity in advance is not only labor-intensive but also reduces the efficiency of speech synthesis.

[0066] Based on this, embodiments of this application provide a speech synthesis method and apparatus, a computer device, and a storage medium, aiming to automatically train a target emotion intensity prediction model capable of predicting emotion intensity, and to obtain the target emotion intensity by predicting the emotion intensity of audio reference data through the target emotion intensity prediction model, thereby achieving automatic identification of emotion intensity. An emotion encoder encodes the audio reference data to obtain an emotion feature vector, and then speech synthesis is performed based on the emotion feature vector, the target emotion intensity, and the target text to synthesize target speech data with a specific emotion intensity. Therefore, generating target speech data with a specific emotion intensity not only saves the labor cost of manually identifying emotion intensity but also enables fine-grained control of emotion intensity. Simultaneously, it enriches the output of speech data with different emotion intensities for intelligent customer service in the financial industry, allowing for the output of speech data matching the emotion intensity according to different types of financial business questions, while endowing intelligent customers with the ability to speak naturally and emotionally like a human, thus improving the customer experience when consulting about financial business.

[0067] The speech synthesis method, apparatus, computer equipment, and storage medium provided in this application are specifically described through the following embodiments. First, the speech synthesis method in this application is described.

[0068] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0069] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0070] The speech synthesis method provided in this application relates to the fields of artificial intelligence and fintech. The speech synthesis method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech synthesis method, but is not limited to the above forms.

[0071] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0072] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user voice data, user behavior data, user historical data, and user location information, user permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user will be obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent will the necessary user-related data for the normal operation of the embodiments of this application be obtained.

[0073] Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0074] Step S101: Obtain speech synthesis reference data; wherein, the speech synthesis reference data includes: audio training data, target text, and audio reference data;

[0075] Step S102: Input the audio training data into the preset original emotion intensity prediction model to predict the emotion intensity and obtain the preliminary emotion intensity.

[0076] Step S103: Adjust the parameters of the original emotion intensity prediction model based on the preliminary emotion intensity to obtain the target emotion intensity prediction model;

[0077] Step S104: Input the audio reference data into the preset emotion encoder for feature encoding to obtain the emotion feature vector;

[0078] Step S105: Input the audio reference data into the target emotion intensity prediction model to predict the emotion intensity and obtain the target emotion intensity;

[0079] Step S106: Perform speech synthesis on the emotion feature vector, target emotion intensity, and target text to obtain target speech data.

[0080] Steps S101 to S106 of this embodiment involve training the original emotion intensity prediction model into a target emotion intensity prediction model based on audio training data. The target emotion intensity prediction model is then used to predict the emotion intensity of the audio reference data, thus automatically identifying the emotion intensity of the audio reference data with greater precision. Next, the audio reference data is input into an emotion encoder for feature encoding to obtain an emotion feature vector. The emotion feature vector, the target emotion intensity, and the target text are then combined to perform speech synthesis to obtain target speech data. This generates target speech data with finer-grained emotion intensity control, eliminating the need for manual emotion intensity identification during speech synthesis, saving labor costs and improving the efficiency of speech synthesis.

[0081] In step S101 of some embodiments, the speech synthesis reference data includes: audio training data, target text, and audio reference data. The audio training data can be directly extracted from a locally preset training database. It can also be obtained from the cloud through other means; there are no specific restrictions on the method of obtaining the audio training data. Specifically, to achieve fine-grained control of the emotional intensity of speech in speech synthesis, audio training data for multiple emotional categories is obtained, with at least two sets of audio training data for each emotional category. By setting audio training data for multiple emotional categories, the target emotional intensity prediction model is trained on the original emotional intensity prediction model based on the audio training data. Therefore, the target emotional intensity prediction model can achieve more fine-grained identification of emotional intensity. For example, if the application scenario is the insurance industry, audio training data is extracted from the question and response voice data of each insurance agent responding to a customer.

[0082] It should be noted that the target text is obtained through real-time input, and the input methods can include the following: text input, voice input, and image recognition. If text input is used, the target text is obtained directly; if voice input is used, the input voice data is subjected to voice content recognition to obtain the voice content, which is then used as the target text; if image recognition is used, image or video data containing text content is directly input, and the target text is obtained by performing OCR recognition on the image or video frame.

[0083] Audio reference data is audio data with emotional intensity. Users select a specific emotion category from a pop-up emotion intensity selection interface, and then choose the corresponding audio data as the audio reference data. For example, if the application is in the financial industry, such as insurance, and the target text is "You are very young, and we recommend you choose Class A medical insurance for your current age," choosing either "very happy" or "generally happy" audio data as the audio reference data will result in audio data with specific emotional intensity, allowing for more nuanced emotional intensity in the synthesized speech data.

[0084] In step S102 of some embodiments, the preset original emotion intensity prediction model is a neural network, and the original emotion intensity prediction model obtains the preliminary emotion intensity by predicting the emotion intensity of the audio training data in a sorting manner. The original emotion intensity prediction model includes an emotion intensity prediction function, which is f(x) t =Wx t ), x t The audio training data is denoted as W, which is the sentiment intensity matrix and includes weights for each sentiment category. The original sentiment intensity prediction model vectorizes the audio training data to obtain audio training vectors. These audio training vectors are then multiplied by the weights in the sentiment intensity matrix, and the summation yields the initial sentiment intensity. The initial sentiment intensity is set between 0 and 1, using a numerical value between 0 and 1 for a more intuitive representation of the initial sentiment intensity.

[0085] For example, if the audio training data consists of very happy audio, the initial emotional intensity prediction obtained by the original emotional intensity prediction model is 0.7; if the audio training data consists of moderately happy audio, the initial emotional intensity is 0.6; and if the audio training data consists of sad audio, the initial emotional intensity is 0.3. Therefore, the emotional intensity prediction is completed automatically by the original emotional intensity prediction model, without the need for manual emotional intensity labeling.

[0086] Please see Figure 2 In some embodiments, step S103 may include, but is not limited to, steps S201 to S205:

[0087] Step S201: Obtain the initial emotion category of the initial emotion intensity to obtain the initial emotion category; wherein, the initial emotion category includes: a neutral emotion category without emotion and a positive and negative emotion category with emotion;

[0088] Step S202: Obtain the preliminary emotional intensity of the neutral emotional category to obtain the reference emotional intensity; obtain the preliminary emotional intensity of the positive and negative emotional categories to obtain the current emotional intensity;

[0089] Step S203: If the current sentiment intensity is greater than or equal to the reference sentiment intensity, then positive and negative sentiment categories are used as candidate sentiment intensities.

[0090] Step S204: Calculate the sentiment intensity loss based on the candidate sentiment intensity to obtain the sentiment intensity loss dataset;

[0091] Step S205: Adjust the parameters of the original emotional intensity prediction model based on the emotional intensity loss dataset to obtain the target emotional intensity prediction model.

[0092] In step S201 of some embodiments, since the audio training data includes emotion categories, and the preliminary emotion intensity is obtained by predicting emotion intensity using the audio training data, the emotion category of the audio training data is used as the preliminary emotion category of the preliminary emotion intensity. Emotion categories include neutral emotion categories without emotion and positive and negative emotion categories with emotion. The positive and negative emotion categories include: happy, sad, angry, and fear. Furthermore, the happy, sad, angry, and fear categories can be further divided into multiple subcategories based on different degrees to classify emotion categories in more detail. Therefore, by determining the preliminary emotion category for each preliminary emotion intensity, the accuracy of the preliminary emotion intensity predicted by the original emotion intensity prediction model can be judged based on the preliminary emotion category.

[0093] In step S202 of some embodiments, the preliminary emotional intensity of the neutral emotional category is used as the reference emotional intensity, and the preliminary emotional intensity of the positive and negative emotional categories is used as the current emotional intensity, so as to divide the preliminary emotional intensity of the neutral emotional category and the positive and negative emotional categories.

[0094] In step S203 of some embodiments, the accuracy of the emotion intensity matrix in the original emotion intensity prediction model determines the accuracy of the emotion intensity prediction. The process of solving the emotion intensity matrix needs to satisfy two conditions. The first condition is that the initial emotion intensity of the neutral emotion category output by the original emotion intensity prediction model must be lower than the initial emotion intensity of the positive and negative emotion categories. If the current emotion intensity is greater than or equal to the reference emotion intensity, then the current emotion intensity is used as a candidate emotion intensity to select a more reasonable candidate emotion intensity. For example, if the reference emotion intensity is 0.2, and there are three current emotion intensities, and the three current emotion intensities are 0.1, 0.3, and 0.4, then the current emotion intensities of 0.3 and 0.4 are used as candidate emotion intensities.

[0095] In step S204 of some embodiments, after the candidate emotion intensities are selected, emotion intensity loss is calculated for each candidate emotion intensity. This calculation involves averaging the differences between candidate emotion intensities within the same positive or negative emotion subcategory to obtain emotion intensity loss data. Multiple emotion intensity loss data sets are then merged into an emotion intensity loss dataset, which is used to assess the accuracy of the original emotion intensity prediction model. For example, if the candidate emotion intensities for "generally happy" are 0.7, 0.75, and 0.72, the emotion intensity loss data is calculated as 0.033 by averaging the differences between these three "generally happy" candidate emotion intensities. Similarly, if the candidate emotion intensities for "very sad" are 0.31, 0.32, and 0.34, the emotion intensity loss data is calculated as 0.02 by averaging the differences between these three "very sad" candidate emotion intensities. The emotion intensity loss dataset is then synthesized from 0.033 and 0.02 to determine the accuracy of the original emotion intensity prediction model.

[0096] In step S205 of some embodiments, the parameters of the original emotion intensity prediction model are adjusted based on the emotion intensity loss dataset. Specifically, the target loss data is obtained by averaging or weighted averaging each loss data point in the emotion intensity loss dataset. The parameters of the original emotion intensity prediction model are adjusted based on the target loss data, that is, the emotion intensity matrix is ​​adjusted according to the target loss data until the emotion intensity matrix corresponding to the minimum target loss data is the target intensity matrix. The target intensity matrix is ​​substituted into the original emotion intensity prediction model to obtain the target emotion intensity prediction model, thereby constructing a more refined target emotion intensity prediction model. The generated target emotion intensity prediction model can predict the emotion intensity of financial intelligent customer service responses to different financial business questions, thus predicting the emotion intensity represented by each voice data point.

[0097] In steps S201 to S205 of the embodiments of this application, a preliminary emotion category is determined to classify the preliminary emotion intensity into neutral emotion categories and positive and negative emotion categories. Preliminary emotion intensities of neutral emotion categories greater than those of positive and negative emotion categories are filtered out, and the remaining preliminary emotion intensities of positive and negative emotion categories are used as candidate emotion intensities. Then, the candidate emotion intensities of the same positive and negative emotion category are averaged to obtain loss data, and an emotion intensity loss dataset is constructed based on this loss data. The parameters of the original emotion intensity model are adjusted based on the emotion intensity loss data until the emotion intensity loss dataset is minimized, completing the training of the original emotion intensity prediction model and obtaining the target emotion intensity prediction model. This constructs a more granular target emotion intensity prediction model for predicting emotion intensity.

[0098] Please see Figure 3 In some embodiments, the emotion encoder includes a feature extraction layer and an encoding layer; step S104 may include, but is not limited to, steps S301 to S302:

[0099] Step S301: Input the audio reference data into the feature extraction layer to extract emotional features and obtain audio emotional features;

[0100] Step S302: Input the audio emotional features into the coding layer for feature encoding processing to obtain the emotional feature vector.

[0101] In step S301 of some embodiments, since the audio reference data for positive and negative sentiment categories exhibits variations in speech rate, with the speech rate of the audio reference data for positive and negative sentiment categories being higher than that of the audio reference data for neutral sentiment categories, and the audio reference data for positive and negative sentiment categories often having larger amplitudes, the emotional features in the audio reference data are extracted through a feature extraction layer. That is, the pitch, energy, and duration of the audio in the audio reference data are obtained as audio emotional features. For example, if the audio reference data is happy audio data, then the pitch, energy, and duration of the happy audio data are obtained as audio emotional features.

[0102] In step S302 of some embodiments, after acquiring audio emotional features, the audio emotional features are input to the encoding layer. The encoding layer performs feature encoding processing on the audio emotional features, that is, converts the audio emotional features into vectors to obtain emotional feature vectors. By representing the audio emotional features in vector form, it is easier to embed the audio emotional features into the audio data of speech synthesis to generate target speech data with emotion.

[0103] In steps S301 to S302 as illustrated in the embodiments of this application, audio emotional features are extracted from the audio reference data, and then the audio emotional features are converted into an emotional feature vector in vector form. By constructing an emotional reference vector with emotion, target speech data with emotion can be generated based on the emotional feature vector during speech synthesis.

[0104] In step S105 of some embodiments, after constructing a target emotion intensity prediction model capable of fine-grained segmentation of emotion intensity, audio reference data is input into the target emotion intensity prediction model. The target emotion intensity prediction model vectorizes the audio reference data to obtain an audio reference vector, and then multiplies the audio reference vector by each weight in the target intensity matrix and sums them to obtain the target emotion intensity. The target emotion intensity value is between 0 and 1, allowing for further control of the emotion intensity during speech synthesis, thereby generating target speech data with more accurate emotion intensity control.

[0105] Please see Figure 4 In some embodiments, step S106 may include, but is not limited to, steps S401 to S405:

[0106] Step S401: Input the emotion feature vector, the target emotion intensity, and the target text into the preset target speech synthesis model; wherein, the target speech synthesis model includes: a phoneme coding network, a speech parameter predictor, a decoder, and a vocoder;

[0107] Step S402: The target text is phoneme encoded using a phoneme encoding network to obtain a phoneme vector;

[0108] Step S403: The phoneme vector, emotion feature vector and target emotion intensity are predicted by the speech parameter predictor to obtain the target speech parameters.

[0109] Step S404: The target speech parameters are decoded by the decoder to obtain speech spectrum data;

[0110] Step S405: The speech spectrum data is converted into audio using a vocoder to obtain the target speech data.

[0111] In step S401 of some embodiments, the target text, target emotional intensity, and emotional feature vector are input into the target speech synthesis model. The target speech synthesis model is a pre-trained speech synthesis model. It synthesizes speech from the target text, target emotional intensity, and emotional feature vector to generate target speech data with emotional intensity, achieving accurate control of emotional intensity and enriching the emotional content of the speech data. The target speech synthesis model includes a phoneme coding network, a speech parameter predictor, a decoder, and a vocoder. The phoneme coding network converts text into vectors; the phoneme parameter predictor predicts the duration, pitch, and energy of each phoneme to generate audio with nuanced emotional intensity control; the decoder converts vectors into spectral data; and the vocoder converts spectral data into speech data.

[0112] In step S402 of some embodiments, the target text is input into a phoneme encoding network for phoneme embedding and encoding processing to convert the target text into a phoneme vector. The phoneme vector includes a sub-vector for each phoneme, representing each phoneme of the target text in vector form to facilitate speech synthesis.

[0113] In step S403 of some embodiments, the phoneme vector, target emotion intensity, and emotion feature vector are input to the speech parameter predictor. The speech parameter predictor embeds the emotion feature vector into the phoneme vector to obtain candidate vectors. Then, the candidate vectors are adjusted according to the target emotion intensity to obtain the target speech parameters. The target speech parameters include the duration, pitch, and energy of each phoneme, and the duration, pitch, and energy of each phoneme are represented by vectors. Therefore, the target speech parameters are represented in the form of a vector set.

[0114] In step S404 of some embodiments, after the target speech parameter prediction is completed, the duration, pitch, and energy of each phoneme are determined. The target speech parameters are then input to the decoder for decoding processing, that is, the target speech parameters are converted from vectors into spectral data to obtain speech spectral data. The spectrum in the speech spectral data is a Mel spectrum.

[0115] In step S405 of some embodiments, after being converted into speech spectrum data, the speech spectrum data is converted into target speech data using a vocoder. The vocoder can be either HiFi-GAN or WaveNet.

[0116] For example, in the financial application scenario of the insurance industry, when a customer asks on the insurance software interface, "I am 26 years old this year, which type of medical insurance can I choose?", the insurance software automatically matches the target text based on the customer's question. The target text is "You are very young, and for your current age, we recommend you choose Class A medical insurance." This target text, along with the emotional feature vector of happiness and the target emotional intensity, is input into the target speech synthesis model. A phoneme coding network converts each character into a phoneme sub-vector to obtain a phoneme vector. The speech parameter predictor adjusts the phoneme vector based on the emotional feature vector of happiness and the target emotional intensity to obtain the target speech parameters, which are represented as vectors. A decoder decodes the target speech parameters to obtain speech spectrum data. A vocoder then converts the speech spectrum data into target speech data containing the happy sentiment and the content "You are very young, and for your current age, we recommend you choose Class A medical insurance." After the target voice data is generated, the target voice data with the intensity of happy emotions is played through the financial intelligent customer service. This makes it easier for customers to choose medical insurance based on the voice data of the intelligent response from the financial intelligent customer service, so as to provide customers with a "real person" service experience and improve their experience when consulting about insurance.

[0117] In steps S401 to S405 of the embodiments of this application, a target speech synthesis model is constructed, including a phoneme coding network, a speech parameter predictor, a decoder, and a vocoder. The phoneme coding network converts phonemes in the target text into vectors to obtain phoneme vectors. Then, the speech parameter predictor predicts the duration, pitch, and energy of each phoneme from the target speech intensity, emotional feature vector, and phoneme vectors to obtain target speech parameters. Based on these target speech parameters, target speech data with accurate emotional intensity is generated.

[0118] Please see Figure 5 In some embodiments, the phoneme coding network includes: a phoneme conversion layer and an encoder; step S402 may include, but is not limited to, steps S501 to S502:

[0119] Step S501: The target text is converted into phonemes through the phoneme conversion layer to obtain text phoneme information;

[0120] Step S502: The text phoneme information is encoded by an encoder to obtain a phoneme vector.

[0121] In step S501 of some embodiments, each character in the target text is extracted through a phoneme conversion layer, and then each character is replaced with its corresponding phoneme sequence. The phoneme sequence of the entire target text is then merged into text phoneme information. Therefore, by representing the text content through phonemes, speech synthesis is facilitated.

[0122] In step S502 of some embodiments, the encoder is an encoder that can encode data or information, converting the input information into an intermediate semantic representation to obtain a phoneme vector through nonlinear transformation. Therefore, converting the target text into a phoneme vector makes speech synthesis using phoneme vectors easier.

[0123] In steps S501 to S502 of the embodiments of this application, each character in the target text is first converted into a phoneme to obtain text phoneme information, and then the text phoneme information is vectorized to obtain a phoneme vector. By converting the target text into a phoneme vector, the target text is represented in vector form, making speech synthesis operations through phoneme vectors easier.

[0124] In some embodiments, prior to step S401, the speech synthesis method further includes: constructing a target speech synthesis model.

[0125] It should be noted that in order to improve the accuracy of the speech data output by the target speech synthesis model, it is necessary to train the original speech synthesis model in advance to obtain the target speech synthesis model that can accurately perform speech synthesis.

[0126] Please see Figure 6In some embodiments, constructing the target speech synthesis model includes, but is not limited to, steps S601 to S603:

[0127] Step S601: Obtain speech synthesis training data; wherein, the speech synthesis training data includes: training feature vectors, training emotion intensity, and training text;

[0128] Step S602: Input the training feature vector, training emotion intensity and training text into the preset original speech synthesis model to perform speech synthesis and obtain speech prediction data;

[0129] Step S603: Adjust the parameters of the original speech synthesis model based on the speech prediction data and the preset speech reference data to obtain the target speech synthesis model.

[0130] In step S601 of some embodiments, audio training data is acquired and input into an emotion encoder for emotion feature encoding processing to obtain a training feature vector. Simultaneously, an original emotion intensity prediction model is trained using the audio training data to obtain a target emotion intensity prediction model, and then the audio training data is input into the target emotion intensity prediction model to obtain the training emotion intensity.

[0131] In step S602 of some embodiments, the training feature vector, training emotion intensity, and training text are all input into a preset original speech synthesis model. The specific structure of the original speech synthesis model is the same as that of the target speech synthesis model, and the speech synthesis operations performed by the original speech synthesis model are consistent with those performed by the target speech synthesis model; these details will not be elaborated further here.

[0132] It should be noted that the original speech synthesis model is used to synthesize speech from the training feature vector, training emotional intensity, and training text to obtain speech prediction data with specific emotional intensity.

[0133] In step S603 of some embodiments, speech reference data is used as reference speech for speech prediction data. For example, if the application scenario is the insurance industry, and the pre-set training text is "Your current experience report shows you are very healthy, and we recommend you choose ordinary medical insurance," and the audio training data is very happy audio data, then the speech reference data is speech data with the emotion of "very happy" and the speech content "Your current experience report shows you are very healthy, and we recommend you choose ordinary medical insurance." If the generated speech prediction data is speech data with the emotion of "generally happy" and the speech content "Your current experience report shows you are very healthy, and we recommend you choose grape insurance," then it indicates that the accuracy of the original speech synthesis model in speech synthesis is insufficient. Therefore, the model parameters of the original speech synthesis model are adjusted according to the speech prediction data and the speech reference data to make the speech prediction data output by the original speech synthesis model approximate the speech reference data, so as to obtain the target speech synthesis model.

[0134] In steps S601 to S603 of the embodiments of this application, speech prediction data is obtained by inputting training feature vectors, training emotional intensity, and training text into the original speech synthesis model. Then, the parameters of the original speech synthesis model are adjusted according to the speech reference data and the speech prediction data to make the speech prediction data approximate the speech reference data, thus obtaining the target speech synthesis model. Therefore, a target speech synthesis model that can output accurate speech data with emotional intensity is constructed, improving the accuracy of speech synthesis. For example, setting the target speech synthesis model in insurance software can enable intelligent customers to output speech data with different emotional intensities, which can improve the customer's experience during insurance business consultation.

[0135] Please see Figure 7 In some embodiments, step S603 may include, but is not limited to, steps S701 to S702:

[0136] Step S701: Perform loss calculation on the speech prediction data and speech reference data to obtain synthetic loss data;

[0137] Step S702: Adjust the parameters of the original speech synthesis model based on the synthesis loss data to obtain the target speech synthesis model.

[0138] In step S701 of some embodiments, speech prediction data is converted into spectrum prediction data, and then speech reference data is converted into spectrum reference data. A loss calculation is performed on the spectrum prediction data and spectrum reference data using a preset loss function to obtain synthetic loss data. The loss function includes any of the following: absolute value loss function, squared loss function, exponential loss function, and cross-entropy loss function.

[0139] In step S702 of some embodiments, the parameters of the original speech synthesis model are adjusted according to the synthesis loss data until the synthesis loss data converges to complete the parameter adjustment of the original speech synthesis model and obtain the target speech synthesis model.

[0140] In steps S701 to S702 as shown in the embodiments of this application, the synthesis loss data is obtained by performing loss calculation based on the speech prediction data and the speech reference data. Then, the parameters of the original speech synthesis model are adjusted based on the synthesis loss data until the synthesis loss data converges to obtain the target speech synthesis model, so as to construct a target speech synthesis model that can achieve higher speech synthesis accuracy.

[0141] This application embodiment obtains audio training data, inputs it into an original emotion intensity prediction model to obtain preliminary emotion intensity, and obtains the emotion category of the audio training data to obtain the emotion category of the preliminary emotion intensity. The emotion category includes a neutral emotion category (no emotion) and positive / negative emotion categories (emotional). The preliminary emotion intensity of the neutral emotion category is used as a reference emotion intensity, and the positive / negative emotion intensity is used as the current emotion intensity. Current emotion intensities greater than or equal to the reference emotion intensity are used as candidate emotion intensities. Loss calculation is then performed on the candidate emotion intensities of the same positive / negative emotion category to obtain an emotion intensity loss dataset. The parameters of the original emotion intensity prediction model are adjusted based on the emotion intensity loss dataset until the emotion intensity loss dataset converges to obtain the target emotion intensity prediction model. The audio training data is input into an encoding layer for emotion feature encoding to obtain a training feature vector. The audio training data is then input into the target emotion intensity prediction model to predict the emotion intensity and obtain the training emotion intensity. The training emotion intensity, training feature vector, and training text are input into an original speech synthesis model for speech synthesis to obtain speech prediction data. Loss calculation is then performed on the speech prediction data and the speech reference data to obtain the synthesis loss data. The parameters of the original speech synthesis model are adjusted based on the synthesis loss data until the synthesis loss data converges to obtain the target speech synthesis model. Audio reference data is input to the feature extraction layer to extract emotional features, which are then input to the encoding layer for encoding to obtain an emotional feature vector. The audio reference data table is input to the target emotional intensity prediction model to predict the target emotional intensity. The emotional feature vector, target emotional intensity, and target text are input to the target speech synthesis model, which includes a phoneme conversion layer, encoder, speech parameter predictor, decoder, and vocoder. The phoneme conversion layer converts the target text into phoneme information, which is then encoded by the encoder to obtain a phoneme vector. The speech parameter predictor predicts the target speech parameters from the phoneme vector, emotional feature vector, and target emotional intensity, where the target speech parameters are the duration, pitch, and energy of each phoneme. The decoder decodes the target speech parameters to obtain speech spectrum data, which is then converted into audio data by the vocoder to obtain the target speech data. Therefore, by automatically constructing a target emotion intensity prediction model capable of predicting emotion intensity, there is no need for manual pre-labeling of the data, saving manpower in emotion intensity labeling. Simultaneously, a target speech synthesis model performs speech synthesis based on the target emotion intensity, target text, and emotion feature vectors to obtain target speech data with more granular emotion intensity.

[0142] For example, in the insurance industry, after constructing a target emotion intensity prediction model and a target speech synthesis model, these models are loaded into insurance software. Each time an intelligent customer service representative engages in insurance business consultation with a customer, the target emotion intensity prediction model and the target speech synthesis model are invoked to automatically generate and output speech data with emotional intensity. Furthermore, the system can output speech data with different emotional intensities to address different insurance questions, thereby improving the customer's experience during insurance business consultations.

[0143] Please see Figure 8 This application also provides a speech synthesis apparatus that can implement the above-described speech synthesis method. The apparatus includes:

[0144] The data acquisition module 801 is used to acquire speech synthesis reference data; wherein, the speech synthesis reference data includes: audio training data, target text and audio reference data;

[0145] The preliminary emotion intensity prediction module 802 is used to input audio training data into a preset original emotion intensity prediction model to predict the emotion intensity and obtain the preliminary emotion intensity.

[0146] The parameter adjustment module 803 is used to adjust the parameters of the original emotion intensity prediction model based on the preliminary emotion intensity to obtain the target emotion intensity prediction model.

[0147] The feature encoding module 804 is used to input audio reference data into a preset emotion encoder for feature encoding to obtain an emotion feature vector;

[0148] The target emotion intensity prediction module 805 is used to input audio reference data into the target emotion intensity prediction model to predict the emotion intensity and obtain the target emotion intensity.

[0149] The speech synthesis module 806 is used to synthesize speech data from the emotion feature vector, the target emotion intensity, and the target text.

[0150] The specific implementation of this speech synthesis device is basically the same as the specific implementation of the speech synthesis method described above, and will not be repeated here.

[0151] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described speech synthesis method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0152] Please see Figure 9 , Figure 9 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes:

[0153] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0154] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the speech synthesis method of the embodiments of this application.

[0155] The input / output interface 903 is used to implement information input and output;

[0156] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0157] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0158] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0159] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech synthesis method.

[0160] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0161] The speech synthesis method, apparatus, computer device, and storage medium provided in this application train an original emotion intensity prediction model into a target emotion intensity prediction model based on audio training data. Therefore, the target emotion intensity prediction model enables automatic prediction of emotion intensity with higher granularity, eliminating the need for manual pre-labeling of emotion intensity in the data and saving labor costs associated with manual labeling. Furthermore, by synthesizing speech data with emotion intensity using the target text, target emotion intensity, and emotion feature vector, the speech synthesis effect can be enriched.

[0162] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0163] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0164] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0165] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0166] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0167] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0169] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0171] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: Acquire speech synthesis reference data; wherein, the speech synthesis reference data includes: audio training data, target text, and audio reference data; The audio training data is input into a preset original emotion intensity prediction model to predict the emotion intensity and obtain a preliminary emotion intensity. Based on the preliminary emotional intensity, the parameters of the original emotional intensity prediction model are adjusted to obtain the target emotional intensity prediction model; The audio reference data is input into a preset emotion encoder for feature encoding to obtain an emotion feature vector; The audio reference data is input into the target emotion intensity prediction model to predict the emotion intensity, thereby obtaining the target emotion intensity. The emotional feature vector, the target emotional intensity, and the target text are combined to synthesize speech data to obtain target speech data. The step of adjusting the parameters of the original emotion intensity prediction model based on the preliminary emotion intensity to obtain the target emotion intensity prediction model includes: The initial emotion category is obtained by acquiring the emotion category of the initial emotion intensity; wherein, the initial emotion category includes: a neutral emotion category without emotion and a positive and negative emotion category with emotion, and the positive and negative emotion category includes at least one positive and negative emotion subcategory; Obtain the preliminary emotional intensity of the neutral emotional category to obtain the reference emotional intensity; obtain the preliminary emotional intensity of the positive and negative emotional categories to obtain the current emotional intensity; If the current emotional intensity is greater than or equal to the reference emotional intensity, then the current emotional intensity is used as the candidate emotional intensity. Based on the candidate emotion intensities, emotion intensity loss is calculated to obtain the emotion intensity loss dataset; The original emotional intensity prediction model is adjusted according to the emotional intensity loss dataset to obtain the target emotional intensity prediction model; The step of calculating the emotional intensity loss based on the candidate emotional intensity to obtain the emotional intensity loss dataset includes: The average difference between the candidate sentiment intensities for the same positive and negative sentiment sub-category is used as the sentiment intensity loss data; Multiple emotional intensity loss data are merged into the emotional intensity loss dataset.

2. The method according to claim 1, characterized in that, The emotion encoder includes a feature extraction layer and an encoding layer; the step of inputting the audio reference data into the preset emotion encoder for feature encoding to obtain an emotion feature vector includes: The audio reference data is input into the feature extraction layer to extract emotional features, thereby obtaining audio emotional features; The audio emotional features are input into the encoding layer for feature encoding processing to obtain the emotional feature vector.

3. The method according to any one of claims 1 to 2, characterized in that, The step of synthesizing the emotional feature vector, the target emotional intensity, and the target text to obtain target speech data includes: The emotional feature vector, the target emotional intensity, and the target text are input into a preset target speech synthesis model; wherein, the target speech synthesis model includes: a phoneme coding network, a speech parameter predictor, a decoder, and a vocoder; The target text is phonetically encoded using the phoneme encoding network to obtain a phoneme vector. The speech parameter predictor performs speech parameter prediction on the phoneme vector, the emotion feature vector and the target emotion intensity to obtain the target speech parameters. The target speech parameters are decoded by the decoder to obtain speech spectrum data; The vocoder performs audio conversion on the speech spectrum data to obtain the target speech data.

4. The method according to claim 3, characterized in that, The phoneme encoding network includes a phoneme conversion layer and an encoder; the process of phoneme encoding the target text through the phoneme encoding network to obtain a phoneme vector includes: The target text is converted into phonemes through the phoneme conversion layer to obtain text phoneme information; The encoder encodes the text phoneme information to obtain the phoneme vector.

5. The method according to claim 3, characterized in that, Before inputting the emotion feature vector, the target emotion intensity, and the target text into a preset target speech synthesis model, the method further includes: Constructing the target speech synthesis model specifically includes: Acquire speech synthesis training data; wherein, the speech synthesis training data includes: training feature vectors, training emotion intensity, and training text; The training feature vector, the training emotion intensity, and the training text are input into a preset original speech synthesis model to perform speech synthesis and obtain speech prediction data. The parameters of the original speech synthesis model are adjusted based on the speech prediction data and the preset speech reference data to obtain the target speech synthesis model.

6. The method according to claim 5, characterized in that, The step of adjusting the parameters of the original speech synthesis model based on the speech prediction data and preset speech reference data to obtain the target speech synthesis model includes: The speech prediction data and the speech reference data are subjected to loss calculation to obtain synthetic loss data; The parameters of the original speech synthesis model are adjusted based on the synthesis loss data to obtain the target speech synthesis model.

7. A speech synthesis device, characterized in that, The device includes: The data acquisition module is used to acquire speech synthesis reference data; wherein, the speech synthesis reference data includes: audio training data, target text, and audio reference data; The preliminary emotion intensity prediction module is used to input the audio training data into a preset original emotion intensity prediction model to predict the emotion intensity and obtain the preliminary emotion intensity. The parameter adjustment module is used to adjust the parameters of the original emotion intensity prediction model based on the preliminary emotion intensity to obtain the target emotion intensity prediction model. The feature encoding module is used to input the audio reference data into a preset emotion encoder for feature encoding to obtain an emotion feature vector; The target emotion intensity prediction module is used to input the audio reference data into the target emotion intensity prediction model to predict the emotion intensity and obtain the target emotion intensity. The speech synthesis module is used to synthesize the emotion feature vector, the target emotion intensity, and the target text to obtain target speech data. The step of adjusting the parameters of the original emotion intensity prediction model based on the preliminary emotion intensity to obtain the target emotion intensity prediction model includes: The initial emotion category is obtained by acquiring the emotion category of the initial emotion intensity; wherein, the initial emotion category includes: a neutral emotion category without emotion and a positive and negative emotion category with emotion, and the positive and negative emotion category includes at least one positive and negative emotion subcategory; Obtain the preliminary emotional intensity of the neutral emotional category to obtain the reference emotional intensity; obtain the preliminary emotional intensity of the positive and negative emotional categories to obtain the current emotional intensity; If the current emotional intensity is greater than or equal to the reference emotional intensity, then the current emotional intensity is used as the candidate emotional intensity. Based on the candidate emotion intensities, emotion intensity loss is calculated to obtain the emotion intensity loss dataset; The original emotional intensity prediction model is adjusted according to the emotional intensity loss dataset to obtain the target emotional intensity prediction model; The step of calculating the emotional intensity loss based on the candidate emotional intensity to obtain the emotional intensity loss dataset includes: The average difference between the candidate sentiment intensities for the same positive and negative sentiment sub-category is used as the sentiment intensity loss data; Multiple emotional intensity loss data are merged into the emotional intensity loss dataset.

8. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method according to any one of claims 1 to 6.

Images