Speech synthesis method, speech synthesis system, electronic device, and storage medium

Abstract

Embodiments of the present application provide a speech synthesis method, a speech synthesis system, an electronic device and a storage medium, and belong to the technical field of financial technology. The method comprises: obtaining a sample phoneme sequence of a sample speech text; encoding the sample phoneme sequence through a text coding module to obtain phoneme coding features; obtaining global emotional features of the sample phoneme sequence according to a global recognition module; obtaining emotional transformation features of a sample initial speech according to a sentence-level recognition module; determining target sample features according to the phoneme coding features, the global emotional features and the emotional transformation features; performing speech synthesis on the target sample features through a speech synthesis module to obtain predicted synthesized speech; adjusting parameters of a model according to the sample initial speech and the predicted synthesized speech to obtain an emotional speech synthesis model; and inputting a target speech text into the emotional speech synthesis model to synthesize target synthesized speech. The embodiments of the present application can generate high-quality synthesized speech with richer emotional performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of financial technology, and in particular to a speech synthesis method, a speech synthesis system, an electronic device and a storage medium. BACKGROUND

[0002] Speech synthesis is a technology that can convert a specified text into synthesized speech of a target speaker. With the wide use of intelligent speech technology in the task scenarios of voice interaction, information broadcasting, voice reading, intelligent sales and the like in financial technology, the target object has higher and higher requirements for the effect of speech synthesis. With the rapid development of deep learning technology, the naturalness and sound quality of speech synthesis have been greatly improved. However, human speech is rich in expressiveness and emotion, and how to make the synthesized speech better simulate the emotional expression of human speech and have characteristics such as more natural and fluent and higher degree of simulation is crucial to improving the application of speech synthesis technology. At present, the speech synthesis method of the related technology usually uses explicit emotion category labels as conditions to generate synthesized speech containing emotions from the original text. However, the synthesized speech obtained by this method only learns an average emotional expression and cannot convey subtle style changes in emotions in the speech, so as to generate high-quality synthesized speech with richer emotional expression. Therefore, how to provide a speech synthesis method capable of deeply mining subtle emotional information contained in the text to generate high-quality synthesized speech with richer emotional expression has become a technical problem to be solved. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a speech synthesis method, a speech synthesis system, an electronic device and a storage medium, which can deeply mine subtle emotional information contained in the text to generate high-quality synthesized speech with richer emotional expression.

[0004] To achieve the above purpose, a first aspect of the embodiments of the present application provides a speech synthesis method, which comprises:

[0005] obtaining sample data, wherein the sample data comprises sample speech text and sample initial speech of the sample speech text;

[0006] performing text conversion on the sample speech text to obtain a sample phoneme sequence;

[0007] inputting the sample data into a preset initial speech synthesis model, wherein the initial speech synthesis model comprises a text encoding module, a global recognition module, a sentence-level recognition module and a speech synthesis module;

[0008] performing encoding processing on the sample phoneme sequence by the text encoding module to obtain phoneme encoding features;

[0009] The global recognition module performs emotion recognition processing on the sample phoneme sequence to obtain global emotion features.

[0010] The sentence-level recognition module extracts emotional features from the initial speech sample to obtain emotional transformation features.

[0011] The phoneme encoding features, the global sentiment features, and the sentiment transformation features are concatenated to obtain the target sample features;

[0012] The speech synthesis module performs speech synthesis processing on the target sample features to obtain predicted synthesized speech;

[0013] Based on the initial sample speech and the predicted synthesized speech, the parameters of the initial speech synthesis model are adjusted to obtain the emotional speech synthesis model.

[0014] The target speech text to be processed is input into the emotional speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

[0015] In some embodiments, the step of performing emotion recognition processing on the sample phoneme sequence through the global recognition module to obtain global emotion features includes:

[0016] The global recognition module performs emotion recognition processing on the sample phoneme sequence to obtain emotion category labels and classification prediction values ​​for the emotion category labels;

[0017] The emotion embedding vector of the emotion category tag is obtained by searching a preset emotion vector lookup table based on the emotion category tag.

[0018] The global sentiment feature is obtained by weighting the classification prediction value and the sentiment embedding vector.

[0019] In some embodiments, the global recognition module includes a pre-trained model and an emotion classifier. The step of performing emotion recognition processing on the sample phoneme sequence based on the global recognition module to obtain an emotion category label and a classification prediction value for the emotion category label includes:

[0020] Based on the pre-trained model, the emotional features of the sample phoneme sequence are extracted to obtain the sample prediction features;

[0021] Based on the sentiment classifier, the predicted features of the sample are used to perform sentiment classification prediction to obtain the classification prediction features;

[0022] The classification prediction features are delinearized according to a preset activation function to obtain the sentiment category label and the classification prediction value of the sentiment category label.

[0023] In some embodiments, the sentiment classifier includes a multi-head attention unit and a global convolutional unit. The step of performing sentiment classification prediction on the sample prediction features based on the sentiment classifier to obtain classification prediction features includes:

[0024] The sample prediction features are obtained by performing self-attention processing on the multi-head attention unit;

[0025] The sample prediction features and the attention features are fused to obtain the attention fused features;

[0026] The attention fusion features are normalized to obtain the first prediction features;

[0027] Global feature extraction is performed on the first predicted feature based on the global convolutional unit to obtain global convolutional features;

[0028] The first predicted feature and the global convolutional feature are fused to obtain the second predicted feature;

[0029] The second prediction feature is normalized to obtain the classification prediction feature.

[0030] In some embodiments, the sentence-level recognition module includes a sentence-level encoder, and the step of extracting emotional features from the initial sample speech through the sentence-level recognition module to obtain emotional transformation features includes:

[0031] The initial speech of the sample is converted into audio to obtain the sample Mel spectrum;

[0032] The sentence-level encoder extracts sentiment features from the sample Mel spectrum to obtain sentence-level hidden features;

[0033] The sentence-level hidden features are transformed to obtain sentiment transformation features.

[0034] In some embodiments, the sentence-level recognition module further includes a sentence-level convolutional unit, a correction unit, and a feature mapping unit, and the method further includes: training the sentence-level encoder, specifically including:

[0035] Sentence-level feature extraction is performed on the phoneme encoding features based on the sentence-level convolutional unit to obtain sentence-level convolutional features;

[0036] The sentence-level convolutional features are modified according to the modification unit to obtain the modified features;

[0037] The modified features are processed by the feature mapping unit to obtain sentence-level prediction features.

[0038] The sentence-level prediction loss value is obtained by calculating the loss of the sentiment transformation feature and the sentence-level prediction feature according to the preset loss function.

[0039] The sentence-level encoder is obtained by adjusting the parameters of the preset initial encoder based on the sentence-level prediction loss value.

[0040] In some embodiments, the speech synthesis module includes a prior encoder, a posterior encoder, a duration predictor, and a decoder. The step of performing speech conversion processing on the target sample features through the speech synthesis module to obtain predicted synthesized speech includes:

[0041] The target sample features are encoded using the prior encoder to obtain prior encoded features.

[0042] Perform a short-time Fourier transform on the initial speech sample to obtain the linear spectrum of the sample;

[0043] Latent variable extraction is performed on the linear spectrum of the sample based on the posterior encoder to obtain the latent variable features of the sample.

[0044] Based on the duration predictor, the phoneme duration of the target sample is extracted to obtain the sample phoneme duration;

[0045] Based on the duration of the sample phonemes, a monotonic alignment search is performed on the latent variable features of the samples and the features of the target samples to determine the target alignment matrix;

[0046] The target sample features are decoded using the target alignment matrix and the decoder to obtain the predicted synthesized speech.

[0047] To achieve the above objectives, a second aspect of this application provides a speech synthesis system, the system comprising:

[0048] A speech sample acquisition module is used to acquire sample data, which includes sample speech text and the initial speech of the sample speech text.

[0049] The text conversion module is used to convert the sample speech text into a sample phoneme sequence.

[0050] The model input module is used to input the sample data into a preset initial speech synthesis model, wherein the initial speech synthesis model includes a text encoding module, a global recognition module, a sentence-level recognition module, and a speech synthesis module;

[0051] The encoding module is used to encode the sample phoneme sequence through the text encoding module to obtain phoneme encoding features;

[0052] A global emotion recognition module is used to perform emotion recognition processing on the sample phoneme sequence through the global recognition module to obtain global emotion features;

[0053] The sentence-level feature extraction module is used to extract emotional features from the initial speech of the sample through the sentence-level recognition module to obtain emotional transformation features;

[0054] The feature concatenation module is used to concatenate the phoneme encoding features, the global sentiment features, and the sentiment transformation features to obtain the target sample features.

[0055] The speech conversion module is used to perform speech synthesis processing on the target sample features through the speech synthesis module to obtain predicted synthesized speech;

[0056] The parameter adjustment module is used to adjust the parameters of the initial speech synthesis model based on the sample initial speech and the predicted synthesized speech to obtain the emotional speech synthesis model.

[0057] The speech synthesis module is used to input the target speech text to be processed into the emotional speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

[0058] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any of the first aspects of the present application.

[0059] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the first aspects of the present application.

[0060] The speech synthesis method, speech synthesis system, electronic device, and storage medium proposed in this application first acquire sample data, which includes sample speech text and initial sample speech of the sample speech text. The initial sample speech contains the speech emotion information to be synthesized from the sample speech text. The sample speech text is then converted into a text sequence to obtain a sample phoneme sequence. The sample data is input into a preset initial speech synthesis model, which includes a text encoding module, a global recognition module, a sentence-level recognition module, and a speech synthesis module. The text encoding module encodes the sample phoneme sequence to obtain phoneme encoding features; the global recognition module performs emotion recognition processing on the sample phoneme sequence to obtain global emotion features; and the sentence-level recognition module extracts emotion features from the initial sample speech to obtain emotion transformation features. Then, the phoneme encoding features, global emotion features, and emotion transformation features are concatenated to obtain target sample features. The speech synthesis module performs speech synthesis processing on the target sample features to obtain predicted synthesized speech, which represents synthesized speech with the same speech emotion information as the initial sample speech. This application embodiment constructs an initial speech synthesis model including a text encoding module, a global recognition module, a sentence-level recognition module, and a speech synthesis module. This enables the deep mining of subtle emotional information contained in the text when performing speech synthesis on sample data based on the initial speech synthesis model. By adjusting the parameters of the initial speech synthesis model using sample initial speech and predicted synthesized speech, an emotional speech synthesis model with the same structure as the initial speech synthesis model is obtained, but capable of generating high-quality synthesized speech with richer emotional expression. Therefore, when performing speech synthesis processing on target speech text using the emotional speech synthesis model provided in this application embodiment, it is possible to deeply mine the subtle emotional information contained in the text to generate high-quality synthesized speech with richer emotional expression. Attached Figure Description

[0061] Figure 1 This is a first flowchart of the speech synthesis method provided in the embodiments of this application;

[0062] Figure 2 yes Figure 1 A flowchart illustrating the specific method of step S1050 in the middle section;

[0063] Figure 3 yes Figure 2 A flowchart illustrating the specific method of step S210;

[0064] Figure 4 yes Figure 3 A flowchart illustrating the specific method of step S320;

[0065] Figure 5 yes Figure 1 A flowchart illustrating the specific method of step S1060;

[0066] Figure 6 This is a second flowchart of the speech synthesis method provided in the embodiments of this application;

[0067] Figure 7 yes Figure 1 A flowchart illustrating the specific method of step S1080;

[0068] Figure 8 This is a block diagram of the module structure of the speech synthesis system provided in the embodiments of this application;

[0069] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0071] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0073] First, let's analyze some of the terms used in this application:

[0074] Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0075] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0076] Text-to-Speech (TTS): TTS is a technology that converts text into speech. TTS generally includes two steps: the first step is text processing, which mainly converts the text into a phoneme sequence and marks the start and end times, frequency changes, and other information of each phoneme; the second step is speech synthesis, which mainly generates speech based on the phoneme sequence (and the marked start and end times, frequency changes, and other information).

[0077] Transformer models are widely used in the field of natural language processing, such as machine translation, question answering systems, text summarization, and speech recognition.

[0078] BERT (Bidirectional Encoder Representation from Transformers) model: It is used to further enhance the generalization ability of word vector models, fully describe character-level, word-level, sentence-level and even sentence-level relationship features, and is built on Transformer.

[0079] Speech synthesis is a technology that converts specified text into synthesized speech from a target speaker. It is a core technology serving tasks such as voice interaction, information broadcasting, and audio reading. With the rapid development of deep learning technology, the naturalness and sound quality of synthesized speech have been greatly improved. However, human speech is rich in expressiveness and emotion, and how to present appropriate emotions in synthesized speech is crucial for building diverse speech generation systems.

[0080] Speech synthesis is a technology that converts specified text into synthesized speech from a target speaker. With the widespread use of intelligent voice technology in tasks such as voice interaction, information broadcasting, audio reading, and intelligent sales in fintech, the demands on the quality of synthesized speech are increasing. With the rapid development of deep learning technology, the naturalness and sound quality of synthesized speech have greatly improved. However, human speech is rich in expressiveness and emotion. How to enable synthesized speech to better simulate the emotional expression of human speech, and possess characteristics such as naturalness, fluency, and high realism, is crucial for improving the application of speech synthesis technology. Currently, related speech synthesis methods typically use explicit emotion category labels as conditions to generate synthesized speech containing emotion from the original text. However, the synthesized speech obtained by this method only learns an average emotional expression and cannot convey the subtle stylistic changes in emotion within the speech, thus failing to generate high-quality synthesized speech with richer emotional expression. Therefore, how to provide a method that can deeply mine the subtle emotional information contained in the text to generate high-quality synthesized speech with richer emotional expression has become an urgent technical problem to be solved.

[0081] Based on this, the speech synthesis method, speech synthesis system, electronic device and storage medium provided in the embodiments of this application can deeply mine the subtle emotional information contained in the text in order to generate high-quality synthesized speech with richer emotional expression.

[0082] This application provides a speech synthesis method, a speech synthesis system, an electronic device, and a storage medium, which are specifically described through the following embodiments. First, the speech synthesis method in the embodiments of this application is described.

[0083] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0084] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0085] The speech synthesis method provided in this application relates to the field of artificial intelligence technology, and particularly to the field of animation processing technology. The speech synthesis method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc.; the server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; the software can be an application implementing the speech synthesis method, but is not limited to the above forms.

[0086] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0087] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0088] Please refer to Figure 1 , Figure 1This is an optional flowchart of the speech synthesis method provided in the embodiments of this application. In some embodiments of this application, the speech synthesis method includes, but is not limited to, steps S1010 to S1100. The following is a detailed description... Figure 1 These ten steps will be explained in detail.

[0089] Step S1010: Obtain sample data, which includes sample speech text and the initial speech of the sample speech text;

[0090] Step S1020: Perform text conversion on the sample speech text to obtain the sample phoneme sequence;

[0091] Step S1030: Input the sample data into the preset initial speech synthesis model, wherein the initial speech synthesis model includes a text encoding module, a global recognition module, a sentence-level recognition module, and a speech synthesis module;

[0092] Step S1040: The sample phoneme sequence is encoded using the text encoding module to obtain phoneme encoding features;

[0093] Step S1050: Perform emotion recognition processing on the sample phoneme sequence through the global recognition module to obtain global emotion features;

[0094] Step S1060: Extract emotional features from the initial speech sample using the sentence-level recognition module to obtain emotional transformation features;

[0095] Step S1070: The phoneme coding features, global sentiment features, and sentiment transformation features are concatenated to obtain the target sample features;

[0096] Step S1080: The target sample features are processed by the speech synthesis module to obtain the predicted synthesized speech.

[0097] Step S1090: Adjust the parameters of the initial speech synthesis model based on the initial sample speech and the predicted synthesized speech to obtain the emotional speech synthesis model;

[0098] Step S1100: Input the target speech text to be processed into the emotional speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

[0099] It should be noted that, in practical application environments, the speech synthesis method provided in this application embodiment can be executed by a terminal or a server, or by a terminal and a server working together. Specifically, the terminal communicates with the server via a network. For example, taking the execution of the synthesized speech detection method on a corresponding terminal as an example, the process is as follows: the terminal or server obtains the target speech text to be processed locally, obtains the corresponding target phoneme sequence based on the text, and performs speech synthesis processing on the target phoneme sequence using a pre-trained emotional speech synthesis model to obtain target synthesized speech containing the emotion of the target speech text. This emotional speech synthesis model is trained by the terminal or server based on the sample phoneme sequence of the sample speech text and the corresponding initial sample speech, and is deployed on the terminal. The speech synthesis method can also be deployed on a server so that the server can also implement the steps of the above speech synthesis method.

[0100] In step S1010 of some embodiments, a training sample set is obtained, which includes at least one sample data, including sample speech text and sample initial speech of the sample speech text. The sample data also includes pre-labeled sample sentiment tags for the sample speech text.

[0101] It should be noted that the sample data can be obtained by recording the obtained sample speech text to determine the corresponding initial sample text; or by converting the obtained initial sample speech into text to determine the corresponding sample speech text. The initial sample speech represents the actual speech of the sample speech text, and the initial sample speech and the sample speech text share the same sentiment label.

[0102] It should be noted that the preset emotional category tags may include, for example, questioning, happiness, or coquetry. The sample emotional tags in this application are any of the preset emotional category tags.

[0103] It should be noted that the sample speech text obtained in the embodiments of this application can relate to various fields, such as science and technology, sports, leisure and entertainment, food, and literature. In other words, the speech synthesis method proposed in this application can be applied to different fields. For example, in the voice interaction scenario of fintech, it can synthesize emotionally charged speech to intelligent interactive robots, such as "happy," "coquettish," and "sorry," thereby giving the robot's emotional expression more vitality and improving the human-computer interaction experience.

[0104] It should be noted that the initial audio sample in this application may be in MP3, CDA, WAV, WMA, RA, MIDI, OGG, APE, or AAC format, etc., and this application does not limit it.

[0105] In step S1020 of some embodiments, in order to make the synthesized speech obtained more conform to the real vocal state, the sample speech text is subjected to text conversion to obtain a sample phoneme sequence. Specifically, a pre-trained acoustic model can be used to perform text conversion on the sample speech text, that is, the sample speech text is segmented to obtain the text segments to be synthesized; each text segment is subjected to phoneme conversion to obtain the phonemes of the segment to be synthesized; the obtained segment phonemes are combined to obtain the sample phoneme sequence of the sample speech text. Herein, a phoneme can refer to the pronunciation phoneme of a character or word in the text to be synthesized, such as the initial consonant and final vowel in Chinese characters. Correspondingly, a sample phoneme sequence can refer to a sequence composed of multiple phonemes. For example, if the sample speech text is "Which one do you like", the corresponding text segments are "you", "like", "which", "one", then when obtaining the segment phonemes corresponding to these text segments, the segment phonemes corresponding to these text segments are combined to obtain the sample phoneme sequence of the sample speech text as [ninxihuannayige].

[0106] In step S1030 of some embodiments, an initial speech synthesis model is constructed. The initial speech synthesis model includes a text encoding module, a global recognition module, a sentence-level recognition module, and a speech synthesis module. Among them, the global recognition module is used to learn the emotional category to which the text belongs from the text globally, and the sentence-level recognition module is used to learn the emotional change trend of the intonation within the sentence. Inputting the sample data into the preset initial speech synthesis model can deeply挖掘 the subtle emotional information contained in the text.

[0107] In step S1040 of some embodiments, after the sample data is input into the preset initial speech synthesis model, the text encoding module performs encoding processing on the sample phoneme sequence to obtain a phoneme encoding feature. It should be noted that the text encoding module can be constructed using a transformer model, and the encoding part of the transformer model can be stacked by n encoder layers, and each encoder layer is composed of two sub-layer connection structures. Among them, the first sub-layer connection structure can include a multi-head attention sub-layer, a normalization layer, and a residual connection; the second sub-layer connection structure can include a feed-forward fully connected sub-layer, a normalization layer, and a residual connection.

[0108] In step S1050 of some embodiments, in order to more accurately identify the emotional category of the sample speech text, the global recognition module performs emotional recognition processing on the sample phoneme sequence to obtain a global emotional feature.

[0109] Specifically, please refer to Figure 2 , Figure 2This is a flowchart illustrating the specific method of step S1050 provided in an embodiment of this application. In some embodiments of this application, step S1050 may specifically include, but is not limited to, steps S210 to S230, as described below. Figure 2 These three steps will be explained in detail.

[0110] Step S210: Perform emotion recognition processing on the sample phoneme sequence according to the global recognition module to obtain the emotion category label and the classification prediction value of the emotion category label;

[0111] Step S220: Search the preset emotion vector lookup table according to the emotion category label to obtain the emotion embedding vector of the emotion category label;

[0112] Step S230: Perform a weighted calculation based on the classification prediction value and the sentiment embedding vector to obtain the global sentiment features.

[0113] In step S210 of some embodiments, in order to predict the sentiment category of the sample text statement, the sample phoneme sequence is input into the global recognition module, so that the global recognition module performs sentiment recognition processing on the sample phoneme sequence to obtain the sentiment category label and the classification prediction value of the sentiment category label, that is, the probability value corresponding to each preset sentiment category label to which the sample text statement belongs.

[0114] In step S220 of some embodiments, the global recognition module of this application includes a global sentiment feature extractor. This global sentiment feature extractor can train and construct a sentiment vector lookup table based on the sentiment labels of each sample speech text in the training sample set. This sentiment vector lookup table stores the mapping relationship between sentiment category labels and sentiment embedding vectors, such that sample data belonging to the same sentiment category label correspond to the same sentiment embedding vector. Therefore, the sentiment embedding vector can represent the information of that sentiment category. Specifically, the sentiment embedding vector for each sentiment category label is determined by searching the preset sentiment vector lookup table according to the sentiment category label. This sentiment embedding vector is used to represent the features of the embedding vector corresponding to the sentiment category label.

[0115] It should be noted that the search of the sentiment vector lookup table satisfies the condition shown in Formula (1) to determine the sentiment embedding vector of each sentiment category label based on the sentiment vector lookup table.

[0116] h glo =f1(e i (1)

[0117] Where f1 represents the search function for the sentiment vector lookup table, e i h is used to represent the true i-th emotion category label. glo Used to indicate the emotion category label e iThe corresponding sentiment embedding vector.

[0118] In step S230 of some embodiments, to avoid inaccurate emotional expression due to incorrect emotional category prediction, this embodiment uses weighted calculation of emotional embedding to perform global emotional recognition on the sample phoneme sequence. Specifically, as shown in formula (2), a weighted calculation is performed based on the classification prediction value and the emotional embedding vector to obtain the global emotional feature, which is denoted as...

[0119]

[0120] Where M represents the total number of preset emotion category labels, p i This indicates that the sample phoneme sequence prediction belongs to the sentiment category label e. i The classification prediction value.

[0121] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating the specific method of step S210 provided in the embodiments of this application. In some embodiments of this application, the global recognition module further includes a pre-trained model and a sentiment classifier. Step S210 may specifically include, but is not limited to, steps S310 to S330. The following describes the process in conjunction with... Figure 3 These three steps will be explained in detail.

[0122] Step S310: Extract emotional features from the sample phoneme sequence based on the pre-trained model to obtain the sample predicted features;

[0123] Step S320: Perform sentiment classification prediction on the sample prediction features based on the sentiment classifier to obtain the classification prediction features;

[0124] Step S330: Perform delinearization processing on the classification prediction features according to the preset activation function to obtain the sentiment category label and the classification prediction value of the sentiment category label.

[0125] In step S310 of some embodiments, to improve the accuracy of sentiment prediction by the global recognition module, sentiment features are extracted from the sample phoneme sequences based on the pre-trained model to obtain the sample prediction features corresponding to the sample phoneme sequences. The training process of the pre-trained model involves using an initial network model built based on the BERT model structure. To prevent overfitting during the processing of sample data, a pre-training sample set is obtained. This pre-training sample set includes multiple pre-trained text sentiment samples and their corresponding sentiment classification labels. The model parameters of the initial network model are fine-tuned based on this pre-training sample set to obtain the pre-trained model, which effectively improves the computational speed and classification accuracy of the pre-trained model.

[0126] It should be noted that the embodiments of this application can use data such as NLPCC2013 and NLPCC2014 to fine-tune the initial network model based on BERT, and the sentiment categories included in the pre-training sample set are consistent with the sentiment categories included in the training sample set of the overall model.

[0127] In steps S320 and S330 of some embodiments, sentiment classification prediction is performed on the sample prediction features based on the sentiment classifier to obtain classification prediction features, which are used to characterize the feature information under each sentiment category label. Then, the classification prediction features are delinearized according to a preset activation function, that is, the classification output result is mapped to the interval [0,1], so that the classification prediction value corresponding to each sentiment category label can be obtained.

[0128] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the specific method of step S320 provided in the embodiments of this application. In some embodiments of this application, the sentiment classifier includes a multi-head attention unit and a global convolutional unit. Step S320 may specifically include, but is not limited to, steps S410 to S460. The following describes the process in conjunction with... Figure 4 These six steps will be explained in detail.

[0129] Step S410: Perform self-attention processing on the sample prediction features based on the multi-head attention unit to obtain attention features;

[0130] Step S420: Perform feature fusion on the sample prediction features and attention features to obtain attention fusion features;

[0131] Step S430: Normalize the attention fusion features to obtain the first prediction feature;

[0132] Step S440: Extract global features from the first predicted features based on the global convolutional units to obtain global convolutional features;

[0133] Step S450: Perform feature fusion on the first predicted feature and the global convolutional feature to obtain the second predicted feature;

[0134] Step S460: Normalize the second prediction feature to obtain the classification prediction feature.

[0135] In steps S410 to S460 of some embodiments, to improve the classification accuracy of the sentiment classifier, the sentiment classifier of this application embodiment may include a multi-head attention unit and a global convolutional unit, which can help the sentiment classifier capture richer feature information and effectively avoid model overfitting. Specifically, the sample prediction features are processed by self-attention using the multi-head attention unit to obtain attention features. Then, the sample prediction features and attention features are fused using residual connections to obtain attention fusion features. To limit the obtained feature data to a certain range and reduce the adverse effects caused by singular sample data, the attention fusion features are normalized to obtain the first prediction features. Then, to improve the model's computation speed, global feature extraction is performed on the first prediction features using the global convolutional unit to obtain global convolutional features. The global convolutional unit can operate in the form of one-dimensional convolution. The first prediction features and global convolutional features are fused using residual connections to obtain the second prediction features. The second prediction features are then normalized to obtain classification prediction features.

[0136] In step S1060 of some embodiments, in order to learn the prosodic information unique to each sentence in the text, such as intra-sentence emotional change information, intonation change information, etc., the embodiments of this application extract emotional features from the initial speech of the sample according to the set sentence-level recognition model to obtain emotional change features.

[0137] Specifically, please refer to Figure 5 , Figure 5 This is a flowchart illustrating the specific method of step S1060 provided in an embodiment of this application. In some embodiments of this application, the sentence-level recognition module includes a sentence-level encoder, and step S1060 may specifically include, but is not limited to, steps S510 to S530. The following describes the process in conjunction with... Figure 5 These three steps will be explained in detail.

[0138] Step S510: Perform audio conversion on the initial speech of the sample to obtain the sample Mel spectrum;

[0139] Step S520: Extract sentiment features from the sample Mel spectrum using a sentence-level encoder to obtain sentence-level hidden features;

[0140] Step S530: Perform feature transformation on the sentence-level hidden features to obtain sentiment transformation features.

[0141] In steps S510 and S520 of some embodiments, in order to extract intra-sentence sentiment change information, intonation change information, etc., of each sentence, audio conversion is performed on the initial sample speech to obtain the sample Mel spectrum corresponding to the initial sample speech. Sentiment features are extracted from the sample Mel spectrum by a preset sentence-level encoder to obtain sentence-level hidden features, which are used to represent the sentiment change information at the sentence level.

[0142] In step S530 of some embodiments, in order to retain the feature information that effectively predicts speech emotion changes, this application can connect a Long Short-Term Memory (LSTM) network to the output of the sentence-level encoder. That is, the sentence-level hidden features are transformed according to the LSTM network, and the emotion transformation feature is obtained by using the output of the last time step of the LSTM. This emotion transformation feature is a fixed-length feature vector, which can be denoted as h. utt .

[0143] Please refer to Figure 6 , Figure 6 This is another optional flowchart of the speech synthesis method provided in the embodiments of this application. In some embodiments of this application, the sentence-level recognition module further includes a sentence-level convolutional unit, a correction unit, and a feature mapping unit. After step S520, the speech synthesis method provided in the embodiments of this application may further include the step of training a sentence-level encoder, which may specifically include, but is not limited to, steps S610 to S650. The following describes the steps in conjunction with... Figure 6 These five steps will be explained in detail.

[0144] Step S610: Extract sentence-level features from phoneme encoding features using sentence-level convolutional units to obtain sentence-level convolutional features;

[0145] Step S620: Correct the sentence-level convolutional features according to the correction unit to obtain the corrected features;

[0146] Step S630: Perform feature mapping processing on the modified features according to the feature mapping unit to obtain sentence-level predicted features;

[0147] Step S640: Calculate the loss of sentiment transformation features and sentence-level prediction features according to the preset loss function to obtain the sentence-level prediction loss value;

[0148] Step S650: Adjust the parameters of the preset initial encoder according to the sentence-level prediction loss value to obtain the sentence-level encoder.

[0149] In steps S610 to S650 of some embodiments, to improve the encoding capability of the sentence-level encoder and obtain feature information that accurately reflects the overall emotional changes within a sentence, firstly, sentence-level feature extraction is performed on the phoneme encoding features using sentence-level convolutional units to obtain sentence-level convolutional features. These sentence-level convolutional units can employ one-dimensional or two-dimensional convolution, etc. To limit the obtained feature data to a certain range and reduce the adverse effects caused by singular sample data, the sentence-level convolutional features are corrected using a correction unit to obtain corrected features. To enhance the model's generalization ability, the corrected features are mapped using a feature mapping unit to obtain sentence-level predicted features. The feature mapping unit can be constructed using a dropout network. Then, loss calculation is performed on the emotional transformation features and sentence-level predicted features according to a preset loss function to obtain sentence-level predicted loss values. The parameters of the preset initial encoder are adjusted based on these sentence-level predicted loss values ​​until the sentence-level predicted loss value of the initial encoder reaches the model termination condition, thus obtaining the sentence-level encoder.

[0150] In step S1070 of some embodiments, in order to achieve multi-scale emotion transfer, the phoneme coding features, global emotion features and emotion transformation features are concatenated to obtain target sample features, and the target sample features are input into the speech synthesis module for training.

[0151] In step S1080 of some embodiments, in order to make the synthesized speech more expressive, the embodiments of this application adopt a structure based on conditional variational autoencoder to construct a speech synthesis model, and perform speech synthesis processing on the target sample features through a speech synthesis module to obtain predicted synthesized speech. This predicted synthesized speech is used to represent the synthesized speech predicted by the initial speech synthesis model constructed by this application.

[0152] Please refer to Figure 7 , Figure 7 This is a flowchart illustrating the specific method of step S1080 provided in an embodiment of this application. In some embodiments of this application, the speech synthesis module includes a prior encoder, a posterior encoder, a duration predictor, and a decoder. Step S1080 may specifically include, but is not limited to, steps S710 to S760. The following describes the process in conjunction with... Figure 7 These six steps will be explained in detail.

[0153] Step S710: Perform feature encoding processing on the target sample features according to the prior encoder to obtain the prior encoded features;

[0154] Step S720: Perform a short-time Fourier transform on the initial speech sample to obtain the linear spectrum of the sample.

[0155] Step S730: Extract latent variables from the linear spectrum of the sample based on the posterior encoder to obtain the latent variable features of the sample;

[0156] Step S740: Extract phoneme duration from the target sample features using the duration predictor to obtain the sample phoneme duration;

[0157] Step S750: Perform a monotonic alignment search on the latent variable features of the samples and the features of the target samples based on the duration of the sample phonemes to determine the target alignment matrix;

[0158] Step S760: Decode the target sample features based on the target alignment matrix and the decoder to obtain the predicted synthesized speech.

[0159] In steps S710 to S760 of some embodiments, to generate high-quality synthesized speech with richer emotional expression, firstly, the features of the concatenated target sample are encoded using a prior encoder to obtain prior encoded features. A short-time Fourier transform is performed on the initial sample speech to obtain the sample linear spectrum. Latent variables are extracted from the sample linear spectrum using a posterior encoder to obtain sample latent variable features. The posterior encoder can employ the non-causal WaveNet residual module from WaveGlow and Glow-TTS. Phoneme durations are extracted from the target sample features using a duration predictor to obtain sample phoneme durations, thus estimating the phoneme duration distribution. A monotonic alignment search is performed on the sample latent variable features and target sample features based on the sample phoneme durations to determine the target alignment matrix. The target sample features are then decoded using the target alignment matrix and the decoder to obtain the predicted synthesized speech.

[0160] In step S1090 of some embodiments, the parameters of the initial speech synthesis model are adjusted according to the initial sample speech and the predicted synthesized speech. That is, the model loss value is determined according to the initial sample speech and the predicted synthesized speech, and the model parameters in the global recognition module, sentence-level recognition module and speech synthesis module are adjusted according to the model loss value. When the initial speech synthesis model meets the preset training termination condition, the emotional speech synthesis model is obtained.

[0161] It should be noted that the preset training termination condition can be when the model loss value of the initial speech synthesis model is less than or equal to the preset loss value threshold, or when the synthesis accuracy of the initial speech synthesis model is greater than or equal to the preset accuracy threshold.

[0162] In step S1100 of some embodiments, after training an emotional speech synthesis model that can deeply mine the subtle emotional information contained in the text and generate high-quality synthesized speech with richer emotional expression, the target speech text to be processed is input into the emotional speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

[0163] Specifically, a text-to-speech (TTS) speech synthesis system can be installed on the terminal, and this system deploys an emotion-based speech synthesis model. Therefore, upon detecting a TTS operation, the terminal generates a speech synthesis service request and sends it to the speech synthesis system. In response to the request, the terminal uses the speech synthesis system to extract the target speech text to be processed and then uses the emotion-based speech synthesis model to synthesize a target synthesized speech containing the implicit emotions within the target speech text.

[0164] It should be noted that in practical applications, when a target user needs speech synthesis on a terminal, they can select the text content to be synthesized on the terminal page, which will then display a pop-up window. The target user can then touch the synthesized speech button in the pop-up window, at which point a speech synthesis service request is sent to the speech synthesis system for processing. Afterwards, the synthesized speech is played back through the terminal's speaker, allowing the target user to hear more emotionally expressive synthesized speech.

[0165] For example, in a fintech voice interaction scenario, target speech text from different scenarios is synthesized using the speech synthesis method provided in this application. The synthesized speech is then input into an intelligent interactive robot, allowing the robot to select the appropriate synthesized speech based on the specific scenario and dialogue. For instance, when a target begins a voice interaction with the intelligent interactive robot, the robot can choose to play a voice with a "questioning" emotional category, such as "How can I help you?", to determine the customer's needs. When the target ends their voice interaction with the intelligent interactive robot, the robot can play a voice with a "coquettish" emotional category, such as "Looking forward to communicating with you again!" This not only gives the robot's emotional expression more vitality but also enhances the human-computer interaction experience.

[0166] Please refer to Figure 8 , Figure 8 This is a schematic diagram of the module structure of the speech synthesis system provided in an embodiment of this application. In some embodiments of this application, the speech synthesis system includes a speech sample acquisition module 8010, a text conversion module 8020, a model input module 8030, an encoding module 8040, a global emotion recognition module 8050, a sentence-level feature extraction module 8060, a feature concatenation module 8070, a speech conversion module 8080, a parameter adjustment module 8090, and a speech synthesis module 8100.

[0167] The speech sample acquisition module 8010 is used to acquire sample data, which includes sample speech text and the initial speech of the sample speech text.

[0168] The text conversion module 8020 is used to convert sample speech text into sample phoneme sequences.

[0169] The model input module 8030 is used to input sample data into a preset initial speech synthesis model, wherein the initial speech synthesis model includes a text encoding module, a global recognition module, a sentence-level recognition module, and a speech synthesis module;

[0170] The encoding module 8040 is used to encode the sample phoneme sequence through the text encoding module to obtain phoneme encoding features;

[0171] The global emotion recognition module 8050 is used to perform emotion recognition processing on the sample phoneme sequence through the global recognition module to obtain global emotion features;

[0172] The sentence-level feature extraction module 8060 is used to extract emotional features from the initial speech of the sample through the sentence-level recognition module to obtain emotional transformation features;

[0173] The feature concatenation module 8070 is used to concatenate phoneme coding features, global sentiment features, and sentiment transformation features to obtain the target sample features.

[0174] The speech conversion module 8080 is used to perform speech synthesis processing on the features of the target sample through the speech synthesis module to obtain the predicted synthesized speech.

[0175] The parameter adjustment module 8090 is used to adjust the parameters of the initial speech synthesis model based on the initial sample speech and the predicted synthesized speech to obtain the emotional speech synthesis model.

[0176] The speech synthesis module 8100 is used to input the target speech text to be processed into the emotional speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

[0177] It should be noted that the speech synthesis system in this application embodiment is used to execute the above-described speech synthesis method, and the speech synthesis system in this application embodiment corresponds to the aforementioned speech synthesis method. For the specific training process, please refer to the aforementioned speech synthesis method, which will not be described in detail here.

[0178] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the speech synthesis method described in the embodiments of this application.

[0179] Electronic devices can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0180] The following is combinedFigure 9 The electronic devices described in the embodiments of this application will be described in detail.

[0181] Please refer to Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0182] The processor 910 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0183] The memory 920 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 920 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910 using the speech synthesis method of the embodiments of this application.

[0184] The input / output interface 930 is used to implement information input and output;

[0185] The communication interface 940 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0186] Bus 950 transmits information between various components of the device (e.g., processor 910, memory 920, input / output interface 930, and communication interface 940);

[0187] The processor 910, memory 920, input / output interface 930 and communication interface 940 are connected to each other within the device via bus 950.

[0188] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech synthesis method described in the embodiments of this application.

[0189] This application provides a speech synthesis method, speech synthesis system, electronic device, and storage medium. The method involves acquiring sample data, including sample speech text and initial sample speech text containing the emotional information to be synthesized from the sample speech text. The sample speech text is then converted into a text sequence to obtain a sample phoneme sequence. The sample data is input into a preset initial speech synthesis model, which includes a text encoding module, a global recognition module, a sentence-level recognition module, and a speech synthesis module. The text encoding module encodes the sample phoneme sequence to obtain phoneme encoding features; the global recognition module performs emotional recognition processing on the sample phoneme sequence to obtain global emotional features; and the sentence-level recognition module extracts emotional features from the initial sample speech to obtain emotional transformation features. Then, the phoneme encoding features, global emotional features, and emotional transformation features are concatenated to obtain target sample features. The speech synthesis module performs speech synthesis processing on the target sample features to obtain predicted synthesized speech, which represents synthesized speech with the same emotional information as the initial sample speech. This application embodiment constructs an initial speech synthesis model including a text encoding module, a global recognition module, a sentence-level recognition module, and a speech synthesis module. By building the model from different levels of speech emotion representation, it enables in-depth mining of subtle emotional information contained in the text when performing speech synthesis on sample data based on the initial speech synthesis model. This application embodiment can extract emotional information from the initial sample speech and predict emotional information from the sample speech text, realizing multi-scale emotion transfer in the emotional speech synthesis task. By adjusting the parameters of the initial speech synthesis model using the initial sample speech and the predicted synthesized speech, an emotional speech synthesis model with the same structure as the initial speech synthesis model is obtained, but capable of generating diverse and high-fidelity synthesized speech. Therefore, when performing speech synthesis processing on target speech text using the emotional speech synthesis model provided in this application embodiment, it is possible to deeply mine the subtle emotional information contained in the text to generate high-quality synthesized speech with richer emotional expression.

[0190] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0191] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0192] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0193] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0195] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0196] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0197] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0198] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0199] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0200] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0201] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: Acquire sample data, which includes sample speech text and the initial sample speech of the sample speech text; The sample speech text is converted into a text sequence to obtain a sample phoneme sequence; The sample data is input into a preset initial speech synthesis model, wherein the initial speech synthesis model includes a text encoding module, a global recognition module, a sentence-level recognition module, and a speech synthesis module; The sample phoneme sequence is encoded using the text encoding module to obtain phoneme encoding features; The global recognition module performs emotion recognition processing on the sample phoneme sequence to obtain global emotion features. The sentence-level recognition module extracts emotional features from the initial speech sample to obtain emotional transformation features. The phoneme encoding features, the global sentiment features, and the sentiment transformation features are concatenated to obtain the target sample features; The speech synthesis module performs speech synthesis processing on the target sample features to obtain predicted synthesized speech; Based on the initial sample speech and the predicted synthesized speech, the parameters of the initial speech synthesis model are adjusted to obtain the emotional speech synthesis model. The target speech text to be processed is input into the emotional speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

2. The method according to claim 1, characterized in that, The step of performing emotion recognition processing on the sample phoneme sequence through the global recognition module to obtain global emotion features includes: The global recognition module performs emotion recognition processing on the sample phoneme sequence to obtain emotion category labels and classification prediction values ​​for the emotion category labels; The emotion embedding vector of the emotion category tag is obtained by searching a preset emotion vector lookup table based on the emotion category tag. The global sentiment feature is obtained by weighting the classification prediction value and the sentiment embedding vector.

3. The method according to claim 2, characterized in that, The global recognition module includes a pre-trained model and an emotion classifier. The step of performing emotion recognition processing on the sample phoneme sequence based on the global recognition module to obtain emotion category labels and classification prediction values ​​for the emotion category labels includes: Based on the pre-trained model, the emotional features of the sample phoneme sequence are extracted to obtain the sample prediction features; Based on the sentiment classifier, the predicted features of the sample are used to perform sentiment classification prediction to obtain the classification prediction features; The classification prediction features are delinearized according to a preset activation function to obtain the sentiment category label and the classification prediction value of the sentiment category label.

4. The method according to claim 3, characterized in that, The sentiment classifier includes a multi-head attention unit and a global convolutional unit. The step of performing sentiment classification prediction on the sample prediction features based on the sentiment classifier to obtain classification prediction features includes: The sample prediction features are obtained by performing self-attention processing on the multi-head attention unit; The sample prediction features and the attention features are fused to obtain the attention fused features; The attention fusion features are normalized to obtain the first prediction features; Global feature extraction is performed on the first predicted feature based on the global convolutional unit to obtain global convolutional features; The first predicted feature and the global convolutional feature are fused to obtain the second predicted feature; The second prediction feature is normalized to obtain the classification prediction feature.

5. The method according to any one of claims 1 to 4, characterized in that, The sentence-level recognition module includes a sentence-level encoder. The step of extracting emotional features from the initial sample speech using the sentence-level recognition module to obtain emotional transformation features includes: The initial speech of the sample is converted into audio to obtain the sample Mel spectrum; The sentence-level encoder extracts sentiment features from the sample Mel spectrum to obtain sentence-level hidden features; The sentence-level hidden features are transformed to obtain sentiment transformation features.

6. The method according to claim 5, characterized in that, The sentence-level recognition module further includes a sentence-level convolutional unit, a correction unit, and a feature mapping unit. The method further includes: training the sentence-level encoder, specifically including: Sentence-level feature extraction is performed on the phoneme encoding features based on the sentence-level convolutional unit to obtain sentence-level convolutional features; The sentence-level convolutional features are modified according to the modification unit to obtain the modified features; The modified features are processed by the feature mapping unit to obtain sentence-level prediction features. The sentence-level prediction loss value is obtained by calculating the loss of the sentiment transformation feature and the sentence-level prediction feature according to the preset loss function. The sentence-level encoder is obtained by adjusting the parameters of the preset initial encoder based on the sentence-level prediction loss value.

7. The method according to any one of claims 1 to 4, characterized in that, The speech synthesis module includes a prior encoder, a posterior encoder, a duration predictor, and a decoder. The step of performing speech conversion processing on the target sample features through the speech synthesis module to obtain predicted synthesized speech includes: The target sample features are encoded using the prior encoder to obtain prior encoded features. Perform a short-time Fourier transform on the initial speech sample to obtain the linear spectrum of the sample; Latent variable extraction is performed on the linear spectrum of the sample based on the posterior encoder to obtain the latent variable features of the sample. Based on the duration predictor, the phoneme duration of the target sample is extracted to obtain the sample phoneme duration; Based on the duration of the sample phonemes, a monotonic alignment search is performed on the latent variable features of the samples and the features of the target samples to determine the target alignment matrix; The target sample features are decoded using the target alignment matrix and the decoder to obtain the predicted synthesized speech.

8. A speech synthesis system, characterized in that, The system includes: A speech sample acquisition module is used to acquire sample data, which includes sample speech text and the initial speech of the sample speech text. The text conversion module is used to convert the sample speech text into a sample phoneme sequence. The model input module is used to input the sample data into a preset initial speech synthesis model, wherein the initial speech synthesis model includes a text encoding module, a global recognition module, a sentence-level recognition module, and a speech synthesis module; The encoding module is used to encode the sample phoneme sequence through the text encoding module to obtain phoneme encoding features; A global emotion recognition module is used to perform emotion recognition processing on the sample phoneme sequence through the global recognition module to obtain global emotion features; The sentence-level feature extraction module is used to extract emotional features from the initial speech of the sample through the sentence-level recognition module to obtain emotional transformation features; The feature concatenation module is used to concatenate the phoneme encoding features, the global sentiment features, and the sentiment transformation features to obtain the target sample features. The speech conversion module is used to perform speech synthesis processing on the target sample features through the speech synthesis module to obtain predicted synthesized speech; The parameter adjustment module is used to adjust the parameters of the initial speech synthesis model based on the sample initial speech and the predicted synthesized speech to obtain the emotional speech synthesis model. The speech synthesis module is used to input the target speech text to be processed into the emotional speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images