Speech synthesis method, speech synthesis device, electronic device, and storage medium

Abstract

The application provides a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, and belongs to the technical field of financial technologies. The method comprises the following steps: inputting target text, reference audio and target phoneme data into a speech synthesis model, performing emotion feature extraction on the reference audio based on a mixed emotion control network to obtain a target emotion coding vector; performing phoneme coding on the target phoneme data based on a phoneme coding network to obtain a target phoneme coding vector; performing text coding on the target text based on a text coding network to obtain a target text coding vector; performing semantic coding on the target emotion coding vector, the target phoneme coding vector and the target text coding vector based on a semantic coding network to obtain a target semantic coding vector; and performing speech synthesis on the target emotion coding vector, the target semantic coding vector and the target text coding vector based on a speech synthesis network to obtain synthesized speech data. The application can improve the speech quality of the synthesized speech.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a speech synthesis method, speech synthesis device, electronic device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent voice interaction is being widely used in finance, logistics, customer service and other fields, improving the service level of enterprise customer service through functions such as intelligent marketing, intelligent debt collection and content navigation.

[0003] Currently, chatbots are frequently used in financial service scenarios such as intelligent customer service and shopping guidance to provide corresponding service support to various users. The dialogue voice used by these chatbots is often generated through speech synthesis.

[0004] In the process of speech synthesis, related technologies often require the control of emotional expression in dialogue to improve the quality of synthesized speech. However, most speech synthesis methods tend to produce synthesized speech containing only one fixed type of emotional information, resulting in a relatively singular emotional content and poor overall speech quality. Summary of the Invention

[0005] The main objective of this application is to provide a speech synthesis method, speech synthesis device, electronic device, and storage medium, which aim to improve the speech quality of synthesized speech.

[0006] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, the method comprising:

[0007] Obtain the target text and generate target phoneme data based on the target text;

[0008] The target text, the preset reference audio, and the target phoneme data are input into a preset speech synthesis model, which includes a hybrid emotion control network, a phoneme coding network, a semantic coding network, a text coding network, and a speech synthesis network.

[0009] Based on the hybrid emotion control network, emotion features are extracted from the reference audio to obtain the target emotion encoding vector;

[0010] The target phoneme data is phoneme encoded based on the phoneme encoding network to obtain the target phoneme encoding vector.

[0011] The target text is encoded using the text encoding network to obtain the target text encoding vector.

[0012] Based on the semantic coding network, the target emotion coding vector, the target phoneme coding vector, and the target text coding vector are semantically encoded to obtain the target semantic coding vector;

[0013] Based on the speech synthesis network, speech is synthesized from the target emotion encoding vector, the target semantic encoding vector, and the target text encoding vector to obtain synthesized speech data.

[0014] In some embodiments, the hybrid emotion control network includes an emotion embedder and an emotion difference module, and the extraction of emotion features from the reference audio based on the hybrid emotion control network includes:

[0015] The reference audio is embedded using the emotion embedder to obtain the target emotion embedding vector.

[0016] Based on the emotional difference module, the reference audio is subjected to emotional recognition to obtain a preliminary emotional attribute vector;

[0017] The initial emotional attribute vector is mapped to a preset vector space to obtain the target emotional attribute vector;

[0018] The target emotion embedding vector and the target emotion attribute vector are added together to obtain the target emotion encoding vector.

[0019] In some embodiments, the emotion embedder includes convolutional layers, GRU layers, and linear layers. The embedding process based on the emotion embedder on the reference audio to obtain the target emotion embedding vector includes:

[0020] Based on the convolutional layer, audio features are extracted from the reference audio to obtain preliminary audio features;

[0021] Based on the GRU layer, emotion embedding is performed on the preliminary audio features to obtain a preliminary emotion embedding vector;

[0022] The target emotion embedding vector is obtained by performing a linear transformation on the initial emotion embedding vector based on the linear layer.

[0023] In some embodiments, the semantic coding network includes a time prediction layer and an attention layer. The step of semantically coding the target sentiment coding vector, the target phoneme coding vector, and the target text coding vector based on the semantic coding network to obtain a target semantic coding vector includes:

[0024] Feature extraction is performed on the target phoneme encoding vector to obtain the first phoneme latent vector;

[0025] The first phoneme latent vector is pooled based on the target text encoding vector to obtain the second phoneme latent vector.

[0026] Based on the time prediction layer, the target text encoding vector is subjected to frame expansion processing to obtain the target word embedding vector;

[0027] The target semantic encoding vector is obtained by performing context encoding on the target word embedding vector, the target sentiment encoding vector, the first phoneme latent vector, and the second phoneme latent vector based on the attention layer.

[0028] In some embodiments, the speech synthesis network includes a Mel spectrum predictor and a vocoder. The process of synthesizing speech data by combining the target emotion encoding vector, the target semantic encoding vector, and the target text encoding vector using the speech synthesis network to obtain synthesized speech data includes:

[0029] The target emotion encoding vector, the target semantic encoding vector, and the target text encoding vector are vector-added to obtain the target audio representation vector;

[0030] Based on the Mel spectrum predictor, the target audio representation vector is predicted to obtain a predicted spectrum feature vector.

[0031] Based on the vocoder, speech synthesis is performed on the predicted spectral feature vector to obtain the synthesized speech data.

[0032] In some embodiments, before inputting the target text, the reference audio, and the target phoneme data into a preset speech synthesis model, the method further includes pre-training the speech synthesis model, specifically including:

[0033] Obtain sample audio, reference sentiment tags for the sample audio, and sample text; wherein, the sample text includes the audio content of the sample audio;

[0034] The sample audio, the reference sentiment tag, and the sample text are input into a preset original model;

[0035] Based on the sample audio and the reference sentiment label, the sample sentiment feature vector and sentiment loss value are obtained;

[0036] Phoneme encoding is performed on the sample phoneme data corresponding to the sample text to obtain the sample phoneme encoding vector;

[0037] The sample text is text encoded to obtain the sample text encoding vector;

[0038] Semantic encoding is performed on the sample sentiment feature vector, the sample phoneme encoding vector, and the sample text encoding vector to obtain the sample semantic encoding vector;

[0039] The sample emotion feature vector, the sample semantic encoding vector, and the sample text encoding vector are used to perform speech synthesis to obtain the sample synthesized speech;

[0040] Based on the synthesized speech and the sample audio, a model loss value is constructed;

[0041] The parameters of the original model are updated based on the model loss value and the sentiment loss value.

[0042] In some embodiments, obtaining the sample emotion feature vector and emotion loss value based on the sample audio and the reference emotion tag includes:

[0043] Based on the sample audio and the reference sentiment label, calculate the sample sentiment embedding vector and the sentiment embedding loss value;

[0044] Based on the sample audio and the reference sentiment label, calculate the sample sentiment attribute vector and sentiment difference degree;

[0045] Based on the sample sentiment embedding vector and the sample sentiment attribute vector, the sample sentiment feature vector is obtained;

[0046] The emotional loss value is obtained based on the emotional embedding loss value and the emotional difference.

[0047] To achieve the above objectives, a second aspect of this application provides a speech synthesis apparatus, the apparatus comprising:

[0048] The data acquisition module is used to acquire target text and generate target phoneme data based on the target text;

[0049] The input module is used to input the target text, the preset reference audio, and the target phoneme data into a preset speech synthesis model, wherein the speech synthesis model includes a hybrid emotion control network, a phoneme coding network, a semantic coding network, a text coding network, and a speech synthesis network.

[0050] The emotion feature extraction module is used to extract emotion features from the reference audio based on the hybrid emotion control network to obtain a target emotion encoding vector.

[0051] A phoneme encoding module is used to encode the target phoneme data based on the phoneme encoding network to obtain a target phoneme encoding vector.

[0052] The text encoding module is used to encode the target text based on the text encoding network to obtain the target text encoding vector;

[0053] The semantic encoding module is used to perform semantic encoding on the target emotion encoding vector, the target phoneme encoding vector, and the target text encoding vector based on the semantic encoding network to obtain the target semantic encoding vector;

[0054] The speech synthesis module is used to perform speech synthesis on the target emotion encoding vector, the target semantic encoding vector, and the target text encoding vector based on the speech synthesis network to obtain synthesized speech data.

[0055] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0056] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0057] The speech synthesis method, speech synthesis device, electronic device, and storage medium proposed in this application acquire target text and generate target phoneme data based on the target text. The target text, preset reference audio, and target phoneme data are input into a preset speech synthesis model, which includes a hybrid emotion control network, a phoneme coding network, a semantic coding network, a text coding network, and a speech synthesis network. Based on the hybrid emotion control network, emotion features are extracted from the reference audio to obtain a target emotion coding vector, enabling hybrid emotion control in the speech synthesis process, allowing the target emotion coding vector to contain mixed emotion information. Furthermore, the target phoneme data is phoneme-coded based on the phoneme coding network to obtain a target phoneme coding vector; the target text is text-coded based on the text coding network to obtain a target text coding vector, which facilitates the extraction of phoneme feature information and text content information from the target text. Furthermore, semantic encoding is performed on the target emotion encoding vector, target phoneme encoding vector, and target text encoding vector using a semantic encoding network to obtain the target semantic encoding vector. Finally, speech synthesis is performed on the target emotion encoding vector, target semantic encoding vector, and target text encoding vector using a speech synthesis network to obtain synthesized speech data. This allows the synthesized speech data to contain mixed emotional information from the reference audio, as well as phoneme feature information and text content information from the target text, thereby effectively improving the speech quality of the synthesized speech data. Consequently, in intelligent dialogues involving insurance products, financial products, etc., the synthesized speech expressed by the chatbot can better match the dialogue style preferences of the dialogue object. By adopting dialogue methods and styles that are more interesting to the dialogue object, the quality and effectiveness of the dialogue are improved, enabling intelligent voice dialogue services and improving customer service quality and customer satisfaction. Attached Figure Description

[0058] Figure 1 This is a flowchart of the speech synthesis method provided in the embodiments of this application;

[0059] Figure 2 This is another flowchart of the speech synthesis method provided in the embodiments of this application;

[0060] Figure 3 yes Figure 2 The flowchart of step S203 in the process;

[0061] Figure 4 yes Figure 1 The flowchart of step S103 in the process;

[0062] Figure 5 yes Figure 4 The flowchart of step S401 in the process;

[0063] Figure 6 yes Figure 1The flowchart of step S106 in the process;

[0064] Figure 7 yes Figure 1 The flowchart of step S107 in the process;

[0065] Figure 8 This is a schematic diagram of the speech synthesis device provided in the embodiments of this application;

[0066] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0070] First, let's analyze some of the terms used in this application:

[0071] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0072] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0073] Information Extraction (NER) is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0074] Mel-Frequency Cipstal Coefficients (MFCCs) are a set of key coefficients used to construct a Mel-Frequency Cipstal spectrum. From a segment of a music signal, a set of cepstrum values ​​can be obtained that is sufficient to represent the music signal. The Mel-Frequency Cipstal Coefficients are the cepstrum values ​​derived from this cepstrum (i.e., the spectrum of the spectrum). Unlike a regular cepstrum, the most distinctive feature of the Mel-Frequency Cipstrum is that its frequency bands are uniformly distributed across the Mel scale. In other words, compared to the linear cepstrum representations commonly seen, this frequency band is closer to the non-linear human auditory system. For example, Mel-Frequency Cipstals are frequently used in audio compression techniques.

[0075] A phoneme is the smallest unit of speech determined by the natural properties of speech. It is analyzed based on the articulation actions within a syllable, with each action constituting a phoneme.

[0076] GRU (Gate Recurrent Unit): GRU is a type of Recurrent Neural Network (RNN), similar to LSTM (Long-Short Term Memory). It was proposed to solve problems such as long-term memory and gradients in backpropagation.

[0077] Speech synthesis refers to the synthesis of intelligible and natural speech from text, also known as text-to-speech (TTS).

[0078] Emotional TTS aims to add emotional effects to synthesized speech.

[0079] With the rapid development of artificial intelligence technology, intelligent voice interaction is being widely used in finance, logistics, customer service and other fields, improving the service level of enterprise customer service through functions such as intelligent marketing, intelligent debt collection and content navigation.

[0080] Currently, chatbots are frequently used in financial service scenarios such as intelligent customer service and shopping guidance to provide corresponding service support to various users. The dialogue voice used by these chatbots is often generated through speech synthesis.

[0081] Taking insurance service robots as an example, it is often necessary to fuse the descriptive text of insurance products with the speaking style of a fixed subject to generate a voice description of the insurance product by that fixed subject. When the insurance service robot converses with some interested individuals, it automatically uses this descriptive voice to introduce the insurance product to those individuals. Related technologies often require the control of the subject's emotions during the speech synthesis process to improve the quality of the synthesized speech. However, most speech synthesis methods in these technologies tend to produce synthesized speech containing only one fixed type of emotional information, resulting in a relatively singular emotional content and poor speech quality.

[0082] Based on this, embodiments of this application provide a speech synthesis method, a speech synthesis apparatus, an electronic device, and a storage medium, aiming to improve the speech quality of synthesized speech.

[0083] The speech synthesis method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the speech synthesis method in this application is described.

[0084] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0085] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0086] The speech synthesis method provided in this application relates to the field of artificial intelligence technology. The speech synthesis method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech synthesis method, but is not limited to the above forms.

[0087] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0088] Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiments of this application. Figure 1The method may include, but is not limited to, steps S101 to S107.

[0089] Step S101: Obtain the target text and generate target phoneme data based on the target text;

[0090] Step S102: Input the target text, the preset reference audio, and the target phoneme data into the preset speech synthesis model. The speech synthesis model includes a hybrid emotion control network, a phoneme coding network, a semantic coding network, a text coding network, and a speech synthesis network.

[0091] Step S103: Extract emotional features from the reference audio based on the hybrid emotion control network to obtain the target emotion encoding vector;

[0092] Step S104: Phoneme encoding is performed on the target phoneme data based on the phoneme encoding network to obtain the target phoneme encoding vector;

[0093] Step S105: Encode the target text using a text encoding network to obtain the target text encoding vector;

[0094] Step S106: Semantically encode the target sentiment encoding vector, target phoneme encoding vector, and target text encoding vector based on the semantic encoding network to obtain the target semantic encoding vector;

[0095] Step S107: Based on the speech synthesis network, speech is synthesized from the target emotion encoding vector, target semantic encoding vector, and target text encoding vector to obtain synthesized speech data.

[0096] Steps S101 to S107, as illustrated in this embodiment, involve acquiring target text and generating target phoneme data based on the target text. The target text, a preset reference audio, and the target phoneme data are then input into a preset speech synthesis model. This speech synthesis model includes a hybrid emotion control network, a phoneme coding network, a semantic coding network, a text coding network, and a speech synthesis network. Emotional features are extracted from the reference audio using the hybrid emotion control network to obtain a target emotion coding vector. This enables hybrid emotion control during speech synthesis, allowing the target emotion coding vector to contain mixed emotional information. Furthermore, the target phoneme data is phoneme-coded using the phoneme coding network to obtain a target phoneme coding vector; the target text is text-coded using the text coding network to obtain a target text coding vector. This facilitates the extraction of phoneme feature information and text content information from the target text. Furthermore, semantic encoding is performed on the target emotion encoding vector, target phoneme encoding vector, and target text encoding vector using a semantic encoding network to obtain the target semantic encoding vector. Finally, speech synthesis is performed on the target emotion encoding vector, target semantic encoding vector, and target text encoding vector using a speech synthesis network to obtain synthesized speech data. This enables the synthesized speech data to contain mixed emotional information from the reference audio as well as phoneme feature information and text content information of the target text, thereby effectively improving the speech quality of the synthesized speech data.

[0097] In step S101 of some embodiments, obtaining the target text includes, but is not limited to, the following methods:

[0098] (1) The target text is obtained by purposefully crawling the preset data source through web crawler. The preset data source includes a preset database or other network platforms that can provide text materials for speech synthesis.

[0099] (2) Obtain the raw text from a public dataset. The public dataset can be the LJSpeech dataset, which contains multiple English speech data recorded by the speakers and the corresponding text data.

[0100] Furthermore, when generating target phoneme data based on target text, the target text can be converted using a pre-defined text conversion model and a reference dictionary to obtain the target phoneme data. Specifically, the pre-defined text conversion model can be an open-source text-to-phoneme model, such as the g2p-seq2seq model, etc., and the reference dictionary can be the CMUSphinx data dictionary, which contains multiple words. A word list can be constructed using the text conversion model and the reference dictionary. The word list is formatted as one word or character per line. This word list allows for data conversion of the text content in the target text, transforming the corresponding words or characters into phoneme sequences, thereby forming the target phoneme data corresponding to the target text.

[0101] It should be noted that the target text may contain financial terminology, financial business templates, product descriptions of insurance products and wealth management products, as well as commonly used dialogue phrases in the financial field.

[0102] The above methods can be used to obtain target text and target phoneme data relatively easily, and can extract the required information from multiple information channels, thus improving data acquisition efficiency.

[0103] Please see Figure 2 In some embodiments, prior to step S102, the speech synthesis method further includes a pre-trained speech synthesis model, which may include, but is not limited to, steps S201 to S209:

[0104] Step S201: Obtain the sample audio, the reference sentiment tag of the sample audio, and the sample text; wherein, the sample text includes the audio content of the sample audio.

[0105] Step S202: Input the sample audio, reference sentiment tags, and sample text into the preset original model;

[0106] Step S203: Based on the sample audio and reference sentiment labels, obtain the sample sentiment feature vector and sentiment loss value;

[0107] Step S204: Phoneme encoding is performed on the sample phoneme data corresponding to the sample text to obtain the sample phoneme encoding vector;

[0108] Step S205: Perform text encoding on the sample text to obtain the sample text encoding vector;

[0109] Step S206: Semantically encode the sample sentiment feature vector, sample phoneme encoding vector, and sample text encoding vector to obtain the sample semantic encoding vector;

[0110] Step S207: Perform speech synthesis on the sample emotion feature vector, sample semantic encoding vector, and sample text encoding vector to obtain the sample synthesized speech;

[0111] Step S208: Construct the model loss value based on the synthesized speech and sample audio.

[0112] Step S209: Update the parameters of the original model based on the model loss value and the sentiment loss value.

[0113] In step S201 of some embodiments, the sample audio may be extracted from a preset audio database, and a reference sentiment tag is used to characterize the sentiment type contained in the sample audio. This reference sentiment tag may be obtained based on manual annotation or other methods. Further, ASR speech recognition technology is used to recognize the sample audio into text form, resulting in sample text that includes the audio content of the sample audio.

[0114] In step S202 of some embodiments, sample audio, reference sentiment tags, and sample text can be directly input into a preset original model using a computer program. The preset original model can be built based on the SyntaSpeech network structure. Similar to the SyntaSpeech network structure, the original model includes a phoneme encoding network, a semantic encoding network, and a speech synthesis network. Unlike the SyntaSpeech network structure, the original model also includes a hybrid sentiment control network and a text encoding network.

[0115] In step S203 of some embodiments, based on the sample audio and the reference sentiment label, the sample sentiment embedding vector and the sentiment embedding loss value are calculated; based on the sample audio and the reference sentiment label, the sample sentiment attribute vector and the sentiment difference are calculated; based on the sample sentiment embedding vector and the sample sentiment attribute vector, the sample sentiment feature vector is obtained; based on the sentiment embedding loss value and the sentiment difference value, the sentiment loss value is obtained.

[0116] In step S204 of some embodiments, sample phoneme data corresponding to the sample text is first extracted. This process is similar to the specific process of extracting target phoneme data from the target text in step S101 above, and will not be repeated here. Further, the phoneme sequence information of the sample phoneme data is extracted using a phoneme coding network to obtain the sample phoneme coding vector.

[0117] In step S205 of some embodiments, the RoBERTa model is used as a text encoding network to encode the sample text, extract the context information of the sample text, and obtain the sample text encoding vector.

[0118] In step S206 of some embodiments, semantic encoding can be performed on the sample sentiment feature vector, sample phoneme encoding vector, and sample text encoding vector based on a semantic encoding network to obtain a sample semantic encoding vector. Specifically, the semantic encoding network includes a temporal prediction layer and an attention layer. It extracts features from the sample phoneme encoding vector to obtain a first sample phoneme latent vector; it performs pooling processing on the first sample phoneme latent vector based on the sample text encoding vector to obtain a second sample phoneme latent vector; it performs frame expansion processing on the sample text encoding vector based on the temporal prediction layer to obtain a sample word embedding vector; and it performs context encoding on the sample word embedding vector, sample sentiment encoding vector, first sample phoneme latent vector, and second sample phoneme latent vector based on the attention layer to obtain the sample semantic encoding vector. This process is essentially the same as the specific implementation of step S106, and will be described in detail when step S106 is specifically described.

[0119] In step S207 of some embodiments, speech synthesis can be performed on the sample emotion feature vector, sample semantic encoding vector, and sample text encoding vector based on a speech synthesis network to obtain sample synthesized speech. The specific implementation of this process is basically the same as that of step S107, and will be described in detail when step S107 is specifically described.

[0120] In step S208 of some embodiments, when constructing the model loss value based on the sample synthesized speech and sample audio, preset cosine similarity algorithms, Euclidean distance, Manhattan distance, etc., can be used to calculate the similarity between the sample synthesized speech and sample audio, and the model loss value is obtained based on the calculation result. Taking the cosine similarity algorithm as an example, the sample synthesized speech and sample audio are first converted into vector form, and then the cosine similarity algorithm is used to calculate the similarity between the vector form sample synthesized speech and sample audio, and the calculated similarity is used as the model loss value.

[0121] In step S209 of some embodiments, since the synthesized speech data is generated based on sample audio and sample text containing audio content, the closer the synthesized speech data is to the sample audio, the better the speech synthesis ability of the speech synthesis model. That is, the smaller the model loss value, the better the training effect of the speech synthesis model, and the stronger its ability to learn mixed information of various emotions, resulting in better speech synthesis capabilities. Based on this, when updating the parameters of the original model based on the model loss value and the emotion loss value, the model parameters of the hybrid emotion control network in the original model are continuously adjusted to minimize the model loss value and optimize the emotion loss value. When the model loss value satisfies the minimization condition and the emotion loss value satisfies the optimization condition, the adjustment of the model parameters is stopped, and the original model at this point is used as the trained speech synthesis model.

[0122] Through the above steps S201 to S209, the ability of the original model to learn mixed information of various emotions can be improved relatively easily, thereby improving the speech synthesis capability of the original model. The model parameters that minimize the model loss value and optimize the emotional loss value are determined by parameter adjustment, thereby obtaining a speech synthesis model that meets the requirements. This makes the speech synthesis model no longer limited to synthesizing synthesized speech with a single emotion, but enables the speech synthesis model to learn mixed information of various emotions and synthesize synthesized speech data containing mixed emotional information, thus improving the speech quality of the synthesized speech data generated based on the speech synthesis model.

[0123] Please see Figure 3 In some embodiments, step S203 may include, but is not limited to, steps S301 to S304:

[0124] Step S301: Based on the sample audio and reference sentiment labels, calculate the sample sentiment embedding vector and sentiment embedding loss value;

[0125] Step S302: Based on the sample audio and reference sentiment labels, calculate the sample sentiment attribute vector and sentiment dissimilarity.

[0126] Step S303: Based on the sample sentiment embedding vector and the sample sentiment attribute vector, obtain the sample sentiment feature vector;

[0127] Step S304: Based on the emotional embedding loss value and the emotional difference, obtain the emotional loss value.

[0128] In step S301 of some embodiments, when calculating the sample sentiment embedding vector and sentiment embedding loss value based on the sample audio and the reference sentiment label, the sentiment embedder in the hybrid sentiment control network can be used to perform sentiment embedding on the sample audio to obtain the sample sentiment embedding vector. Further, preset cosine similarity algorithms, Euclidean distance, Manhattan distance, etc., are used to calculate the similarity between the sample sentiment embedding vector and the reference sentiment label, and the sentiment embedding loss value is obtained based on the calculation results.

[0129] In step S302 of some embodiments, when calculating the sample emotional attribute vector and emotional difference degree based on the sample audio and reference emotional label, the emotional difference module in the hybrid emotional control network can be used to perform emotional recognition on the sample audio to obtain a first emotional attribute vector. Then, the first emotional attribute vector is mapped to a predetermined vector space to obtain the sample emotional attribute vector. The feature dimension of the predetermined vector space is determined according to the feature dimension of the sample emotional embedding vector, that is, the feature dimension of the predetermined vector space is the same as the feature dimension of the sample emotional embedding vector. For example, in the hybrid emotional control network of the original model, there are five different emotions preset, namely Happy, Angry, Neural, Sad, and Surprise. If the reference emotional label of the current sample audio is Happy, then the emotional type of the sample audio is Happy. At this time, it is necessary to calculate the difference degree between this emotional type and the other four emotional types. This calculation process can be measured by different functions f1(x), f2(x), f3(x), and f4(x). The obtained first emotional attribute vector is a discrete 5-value emotional attribute vector.

[0130] Furthermore, this sentiment difference module aims to measure the difference between the sentiment type of the sample audio and other sentiment types, and finally normalize the difference to between 0 and 1. The smaller the normalized value, the smaller the difference and the higher the similarity; the larger the normalized value, the larger the difference and the lower the similarity. The sentiment type of the sample audio can be determined based on the reference sentiment label. The training process of the sentiment difference module is an optimization process similar to SVM, and the sentiment difference is mainly used to measure the training status of the sentiment difference module. The optimization process can be represented as shown in formula (1):

[0131] f(x n )=Wx n Formula (1)

[0132] Here, W is used to measure the differences between different emotion types. n f(x) represents the nth sample audio; n ) is a function used to measure the difference between the sentiment type of the nth sample audio and other sentiment types.

[0133] Suppose there exists an i-th sample audio x i With the j-th sample audio x j If A and B are different sets of emotions, then the optimization process should satisfy the following formulas (2) and (3).

[0134]

[0135]

[0136] Formula (2) represents the classification of sample audio based on the reference sentiment label of the sample audio, where if the i-th sample audio x i The j-th sample audio file belongs to the sentiment set A. j When it belongs to sentiment set B, the i-th sample audio x i and the j-th sample audio x j The degree of difference should be as large as possible, that is, the difference between sample audio belonging to different emotion sets should be as large as possible.

[0137] Formula (2) represents the classification of sample audio based on the reference sentiment label of the sample audio, where if the i-th sample audio x i and the j-th sample audio x j All belong to the sentiment set A, and the i-th sample audio is x. i and the j-th sample audio x j When all samples belong to sentiment set B, the i-th sample audio x i and the j-th sample audio x j The difference should be as small as possible; that is, the difference between sample audios belonging to the same emotional set should be as small as possible.

[0138] Based on this, the problem of solving W can be transformed into solving an optimization problem, and the result of solving the optimization problem can be used as the sentiment difference degree to be calculated in step S302. The optimization problem can be expressed as shown in formula (4), and the constraints of formula (4) include formula (5), formula (6) and formula (7).

[0139]

[0140]

[0141]

[0142] ξ ij ≥0; γ ij ≥0 Formula (7)

[0143] Where C is the penalty function. The larger the value of C, the greater the penalty for classification. ξ ij and γ ij For each sample audio, a corresponding slack variable is set, representing the degree to which the sample audio does not meet the constraints. st refers to subject to, which means that the constraints of formula (4) include formula (5), formula (6) and formula (7).

[0144] Wx i Pointer audio sample x iThe result after mapping by the difference function W, Wx j Pointer audio sample x j The result after mapping by the difference function W, therefore, W(x) i -x j This refers to the result after mapping two sample audio files using the difference function W. Since x... i and x j Since they all originate from the same sentiment set, the distance between them should be as small as possible, i.e., less than the slack variable γ. ij .

[0145] After solving each W based on the above formulas (1) to (7), the output results of the emotional difference module can be obtained relatively easily.

[0146] In step S303 of some embodiments, since the feature dimensions of the sample emotion embedding vector and the sample emotion attribute vector are the same, the sample emotion embedding vector and the sample emotion attribute vector can be directly added together to obtain the sample emotion feature vector.

[0147] In step S304 of some embodiments, the emotional loss value obtained based on the emotional embedding loss value and the emotional difference degree can be weighted and summed to obtain the emotional loss value. Specifically, the weight ratios of the emotional embedding loss value and the emotional difference degree are set, with the weight ratio of the emotional embedding loss value determined as 'a' and the weight ratio of the emotional difference value determined as 'b', where a + b = 1. Further, the emotional loss value = a * emotional loss value + b * emotional difference degree.

[0148] Through the above steps S301 to S304, the mixed emotion control module of the model can learn mixed information of various emotions, which can effectively improve the model's ability to perceive mixed emotions, thereby improving the speech synthesis ability of the speech synthesis model, enabling the speech synthesis model to synthesize synthesized speech data containing mixed emotional information, and improving the speech quality of the synthesized speech data.

[0149] In step S102 of some embodiments, the target text, preset reference audio, and target phoneme data can be input into a preset speech synthesis model using a computer program or preset script. The preset reference audio can be audio data collected from various databases or network platforms and stored in a fixed repository. The reference audio can be audio data from different application scenarios, languages, styles, and genres, without limitation. For example, in a financial transaction scenario, the reference audio is audio data containing commonly used dialogue in the financial field; in an insurance sales scenario, the reference audio is audio data containing descriptions of a particular insurance product, such as its type, cost, and target audience. The speech synthesis model includes a hybrid emotion control network, a phoneme coding network, a semantic coding network, a text coding network, and a speech synthesis network. The hybrid emotion control network is mainly used to control mixed emotions based on the input reference audio, learn and extract different types of emotional information, and thus generate mixed emotional information for speech synthesis. The phoneme coding network is mainly used to encode the input phoneme data and extract phoneme feature information. The text coding network is mainly used to encode the input text data and extract text content information. The semantic coding network is mainly used to fuse the extracted text content information, phoneme feature information, and mixed emotion information to form fused semantic information. The speech synthesis network is mainly used to synthesize speech based on the fused semantic information, mixed emotion information, and text content information to generate synthesized speech data corresponding to the input text.

[0150] Please see Figure 4 In some embodiments, the hybrid emotion control network includes an emotion embedder and an emotion difference module, and step S103 may include, but is not limited to, steps S401 to S404:

[0151] Step S401: The reference audio is embedded using an emotion embedder to obtain the target emotion embedding vector;

[0152] Step S402: Based on the emotion difference module, perform emotion recognition on the reference audio to obtain a preliminary emotion attribute vector;

[0153] Step S403: Map the initial emotional attribute vector to a preset vector space to obtain the target emotional attribute vector;

[0154] Step S404: Add the target emotion embedding vector and the target emotion attribute vector together to obtain the target emotion encoding vector.

[0155] In step S401 of some embodiments, the emotion embedder includes a convolutional layer, a GRU layer, and a linear layer. Audio features are extracted from the reference audio using the convolutional layer to obtain preliminary audio features; emotion embedding is performed on the preliminary audio features using the GRU layer to obtain a preliminary emotion embedding vector; and a linear transformation is performed on the preliminary emotion embedding vector using the linear layer to obtain the target emotion embedding vector.

[0156] In step S402 of some embodiments, when performing emotion recognition on the reference audio based on the emotion difference module, the proportion of each type of emotion in the emotion difference module can be set according to actual needs. The control of mixed emotions is achieved based on the percentage of different types of emotions, thereby obtaining a preliminary emotion attribute vector. For example, by manually specifying the percentage of each emotion, the difference between the emotion to be synthesized and Happy, Angry, Neural, Sad, and Surprise can be set to 0.7, 1, 1, 1, and 0, respectively. The resulting emotion attribute vector will represent the emotion distribution in the synthesized speech data as 30% Happy + 100% Surprise.

[0157] In step S403 of some embodiments, to match the dimension of the target emotion embedding vector, the hybrid emotion control network further includes a fully connected layer connected to the emotion difference module. The preliminary emotion attribute vector output by the emotion difference module is input to the fully connected layer. A preset vector space within the fully connected layer is used to perform feature mapping on the preliminary emotion attribute vector, that is, mapping the preliminary emotion attribute vector to this vector space to obtain the target emotion attribute vector. The feature dimension of the preset vector space is the same as the dimension of the target emotion embedding vector. For example, if the target emotion embedding vector is a 256-dimensional vector, then the feature dimension of the preset vector space is 256 dimensions. After feature mapping of the preliminary emotion attribute vector through the preset vector space, the obtained target emotion attribute vector is a 256-dimensional vector.

[0158] In step S404 of some embodiments, since the target emotion embedding vector and the target emotion attribute vector are vectors of the same dimension, the target emotion embedding vector and the target emotion attribute vector can be directly added together to obtain the target emotion encoding vector.

[0159] Through the above steps S401 to S404, the difference between the emotion of the synthesized speech data to be synthesized and each preset emotion can be determined relatively easily through the emotion difference module. This determines the emotion types contained in the synthesized speech data to be synthesized and the proportion of each emotion type, realizing the control of the mixed emotion information of the synthesized speech data. The emotion embedding information obtained based on the reference audio is fused with the mixed emotion information to obtain the overall emotion information of the synthesized speech data to be synthesized. This can achieve the effect of generating mixed emotion audio and improve the applicability and generalization of the speech synthesis method.

[0160] Please see Figure 5 In some embodiments, the emotion embedder includes convolutional layers, GRU layers, and linear layers, and step S401 may include, but is not limited to, steps S501 to S503:

[0161] Step S501: Extract audio features from the reference audio based on the convolutional layer to obtain preliminary audio features;

[0162] Step S502: Based on the GRU layer, perform emotion embedding on the preliminary audio features to obtain a preliminary emotion embedding vector;

[0163] Step S503: Perform a linear transformation on the initial emotion embedding vector based on the linear layer to obtain the target emotion embedding vector.

[0164] In step S501 of some embodiments, audio features are extracted from the reference audio based on the convolutional layer to capture the audio feature information in the reference audio and obtain preliminary audio features.

[0165] In step S502 of some embodiments, when performing emotion embedding on the preliminary audio features based on the GRU layer, the preliminary audio features at each time step and the hidden layer state features of the preliminary audio features at the previous time step are input into the GRU layer together. Based on the GRU layer, feature extraction is performed on the preliminary audio features at each time step and the hidden layer state features of the preliminary audio features at the previous time step, and the hidden layer state features of the preliminary audio features at that time step and the preliminary emotion embedding vector are output. The hidden layer state features at that time step are used to provide the preliminary audio features at the next time step for emotion embedding.

[0166] In step S503 of some embodiments, a linear transformation is performed on the preliminary emotion embedding vector based on a linear layer to change the feature dimension of the preliminary emotion embedding vector, converting the high-dimensional preliminary emotion embedding vector into a low-dimensional vector to obtain the target emotion embedding vector, so that the target emotion embedding vector can contain richer emotion feature information.

[0167] Through the above steps S501 to S503, the emotion embedding of the reference audio can be achieved more conveniently, and the emotion feature information of the reference audio can be extracted more accurately and comprehensively, thereby improving the feature comprehensiveness and feature quality of the target emotion embedding vector.

[0168] In step S104 of some embodiments, when phoneme encoding is performed on the target phoneme data based on the phoneme encoding network to obtain the target phoneme encoding vector, the phoneme sequence information of the target phoneme data is extracted by using the phoneme encoding network, which can obtain better feature representation and obtain a target phoneme encoding vector with higher feature quality.

[0169] In step S105 of some embodiments, when encoding the target text based on a text encoding network to obtain the target text encoding vector, the RoBERTa model can be used as the text encoding network. Using the RoBERTa model as the text encoding network to encode the target text can more easily extract the context information of the target text, so as to extract as much of the text content information of the target text as possible, thereby improving the feature quality of the generated target text encoding vector.

[0170] Please see Figure 6 In some embodiments, the semantic coding network includes a temporal prediction layer and an attention layer, and step S106 includes, but is not limited to, steps S601 to S604:

[0171] Step S601: Extract features from the target phoneme encoding vector to obtain the first phoneme latent vector;

[0172] Step S602: Pool the first phoneme latent vector based on the target text encoding vector to obtain the second phoneme latent vector;

[0173] Step S603: Perform frame expansion processing on the target text encoding vector based on the time prediction layer to obtain the target word embedding vector;

[0174] Step S604: Based on the attention layer, perform context encoding on the target word embedding vector, target sentiment encoding vector, first phoneme latent vector, and second phoneme latent vector to obtain the target semantic encoding vector.

[0175] In step S601 of some embodiments, feature extraction is performed on the target phoneme encoding vector to obtain the phoneme latent feature information in the target phoneme encoding vector, and a first phoneme latent vector is obtained.

[0176] In step S602 of some embodiments, when pooling the first phoneme latent vector based on the target text encoding vector, the phoneme information belonging to the same character in the first phoneme latent vector is pooled according to the character segmentation boundary information in the target text encoding vector to obtain the second phoneme latent vector, which is a character-level latent vector.

[0177] In step S603 of some embodiments, when performing frame expansion processing on the target text encoding vector based on the time prediction layer, the time of each word in the target text encoding vector is first predicted using the time prediction layer to obtain the predicted time parameter, and the target text encoding vector is expanded according to the predicted time parameter to obtain the target word embedding vector.

[0178] In step S604 of some embodiments, the target word embedding vector, target sentiment encoding vector, first phoneme latent vector, and second phoneme latent vector are input to the attention layer. An audio key vector and an audio value vector are determined based on the first audio latent vector, and an audio query vector is determined based on the target word embedding vector, target sentiment encoding vector, and second phoneme latent vector. Further, attention is calculated based on the audio key vector, audio value vector, and audio query vector to achieve contextual encoding of the target word embedding vector, target sentiment encoding vector, first phoneme latent vector, and second phoneme latent vector, resulting in a target semantic encoding vector.

[0179] Through the above steps S601 to S604, the text content information in the target text encoding vector, the mixed emotion information in the target emotion encoding vector, and the phoneme sequence information in the target phoneme encoding vector can be fused relatively easily, so that the obtained target semantic encoding vector contains the main feature information for speech synthesis, which can effectively improve the accuracy of speech synthesis and the speech quality of synthesized speech data.

[0180] Please see Figure 7 In some embodiments, the speech synthesis network includes a Mel spectrum predictor and a vocoder, and step S107 may include, but is not limited to, steps S701 to S703:

[0181] Step S701: Add the target emotion encoding vector, target semantic encoding vector, and target text encoding vector together to obtain the target audio representation vector;

[0182] Step S702: Based on the Mel spectrum predictor, the target audio representation vector is predicted to obtain the predicted spectrum feature vector.

[0183] Step S703: Based on the vocoder, the predicted spectral feature vector is used for speech synthesis to obtain synthesized speech data.

[0184] In step S701 of some embodiments, when adding the target emotion encoding vector, target semantic encoding vector, and target text encoding vector, the vectors can be added directly to obtain the target audio representation vector, which can improve computational efficiency. Alternatively, the target emotion encoding vector, target semantic encoding vector, and target text encoding vector can be first feature mapped to transform them into a vector space of the same dimension, so that the vector dimensions of the target emotion encoding vector, target semantic encoding vector, and target text encoding vector are the same. Then, the target emotion encoding vector, target semantic encoding vector, and target text encoding vector in the same dimension vector space are added to obtain the target audio representation vector. This method can improve the rationality of feature fusion and improve the feature quality of the target audio representation vector.

[0185] In step S702 of some embodiments, the Mel spectrum predictor can be a Tacotron2 model or a commonly used model in speech synthesis in related technologies, without limitation. Taking the Tacotron2 model as an example, the Mel spectrum predictor is a seq2seq network with an attention mechanism. The Mel spectrum predictor includes an encoder and a decoder. The encoder is a module consisting of a style embedding layer, three convolutional layers, and a bidirectional LSTM layer. First, the style embedding layer performs word embedding on the target audio representation vector to obtain an audio character sequence. Then, the word-embedded audio character sequence is input into the three convolutional layers for context information extraction to obtain an audio context vector. The audio context vector is then input into the bidirectional LSTM layer to obtain a spectral hidden state vector. The decoder includes two bidirectional LSTM layers and a projection layer. The bidirectional LSTM layers extract new context information from the spectral hidden state vector to obtain a spectral context vector. Then, the projection layer predicts the spectral context vector to obtain a predicted spectral feature vector.

[0186] Furthermore, in some other embodiments, in order to improve the feature quality of the spectral features, the output of the decoder can be fed into a series of convolutional layers for convolution processing, and the output of the last convolutional layer can be used as the predicted spectral feature vector.

[0187] In step S703 of some embodiments, the vocoder may be a HiFi-GAN vocoder, which includes an upsampling module and a residual module for multi-receptive field fusion. The upsampling module can perform upsampling processing on the predicted spectral feature vector to achieve transpose convolution of the predicted spectral feature vector to obtain initial speech features. The residual module is then used to reconstruct the initial speech features to obtain the reconstructed speech waveform, which is then used as the synthesized speech data.

[0188] In a specific example, the synthesized speech data consists of a description of insurance or financial products, containing the speaking style and emotional tone of an animated character. This synthesized speech data can attract potential viewers with the unique speaking style and emotional tone of the animated character, making them more interested in the insurance or financial products recommended by the synthesized speech data.

[0189] Through the above steps S701 to S703, the synthesized speech data can simultaneously contain mixed emotional information, phoneme feature information of the target text, and text content information, thereby effectively improving the accuracy of speech synthesis and the speech quality of the synthesized speech data.

[0190] The speech synthesis method of this application embodiment acquires target text and generates target phoneme data based on the target text. The target text, a preset reference audio, and the target phoneme data are input into a preset speech synthesis model, which includes a hybrid emotion control network, a phoneme coding network, a semantic coding network, a text coding network, and a speech synthesis network. Based on the hybrid emotion control network, emotion features are extracted from the reference audio to obtain a target emotion coding vector, enabling hybrid emotion control in the speech synthesis process, allowing the target emotion coding vector to contain mixed emotion information. Furthermore, the target phoneme data is phoneme-coded based on the phoneme coding network to obtain a target phoneme coding vector; the target text is text-coded based on the text coding network to obtain a target text coding vector, which facilitates the extraction of phoneme feature information and text content information from the target text. Furthermore, semantic encoding is performed on the target emotion encoding vector, target phoneme encoding vector, and target text encoding vector using a semantic encoding network to obtain the target semantic encoding vector. Finally, speech synthesis is performed on the target emotion encoding vector, target semantic encoding vector, and target text encoding vector using a speech synthesis network to obtain synthesized speech data. This allows the synthesized speech data to contain mixed emotional information from the reference audio, as well as phoneme feature information and text content information from the target text, thereby effectively improving the speech quality of the synthesized speech data. Consequently, in intelligent dialogues involving insurance products, financial products, etc., the synthesized speech expressed by the chatbot can better match the dialogue style preferences of the dialogue object. By adopting dialogue methods and styles that are more interesting to the dialogue object, the quality and effectiveness of the dialogue are improved, enabling intelligent voice dialogue services and improving customer service quality and customer satisfaction.

[0191] Please see Figure 8 This application also provides a speech synthesis apparatus that can implement the above-described speech synthesis method. The apparatus includes:

[0192] The data acquisition module 801 is used to acquire the target text and generate target phoneme data based on the target text;

[0193] The input module 802 is used to input the target text, the preset reference audio, and the target phoneme data into the preset speech synthesis model. The speech synthesis model includes a hybrid emotion control network, a phoneme coding network, a semantic coding network, a text coding network, and a speech synthesis network.

[0194] The emotion feature extraction module 803 is used to extract emotion features from the reference audio based on a hybrid emotion control network to obtain the target emotion encoding vector.

[0195] The phoneme encoding module 804 is used to encode the target phoneme data based on the phoneme encoding network to obtain the target phoneme encoding vector.

[0196] The text encoding module 805 is used to encode the target text based on a text encoding network to obtain the target text encoding vector.

[0197] Semantic coding module 806 is used to perform semantic coding on the target sentiment coding vector, target phoneme coding vector and target text coding vector based on the semantic coding network to obtain the target semantic coding vector;

[0198] The speech synthesis module 807 is used to synthesize speech data from the target emotion encoding vector, target semantic encoding vector, and target text encoding vector based on the speech synthesis network.

[0199] The specific implementation of this speech synthesis device is basically the same as the specific implementation of the speech synthesis method described above, and will not be repeated here.

[0200] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned speech synthesis method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0201] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0202] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0203] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the speech synthesis method of the embodiments of this application.

[0204] The input / output interface 903 is used to implement information input and output;

[0205] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0206] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0207] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0208] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the above-described speech synthesis method.

[0209] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0210] The speech synthesis method, speech synthesis device, electronic device, and computer-readable storage medium provided in this application acquire target text and generate target phoneme data based on the target text. The target text, preset reference audio, and target phoneme data are input into a preset speech synthesis model, which includes a hybrid emotion control network, a phoneme coding network, a semantic coding network, a text coding network, and a speech synthesis network. Based on the hybrid emotion control network, emotion features are extracted from the reference audio to obtain a target emotion coding vector, enabling hybrid emotion control in the speech synthesis process, allowing the target emotion coding vector to contain mixed emotion information. Furthermore, based on the phoneme coding network, the target phoneme data is phoneme-coded to obtain a target phoneme coding vector; based on the text coding network, the target text is text-coded to obtain a target text coding vector, which facilitates the extraction of phoneme feature information and text content information of the target text. Furthermore, semantic encoding is performed on the target emotion encoding vector, target phoneme encoding vector, and target text encoding vector using a semantic encoding network to obtain the target semantic encoding vector. Finally, speech synthesis is performed on the target emotion encoding vector, target semantic encoding vector, and target text encoding vector using a speech synthesis network to obtain synthesized speech data. This allows the synthesized speech data to contain mixed emotional information from the reference audio, as well as phoneme feature information and text content information from the target text, thereby effectively improving the speech quality of the synthesized speech data. Consequently, in intelligent dialogues involving insurance products, financial products, etc., the synthesized speech expressed by the chatbot can better match the dialogue style preferences of the dialogue object. By adopting dialogue methods and styles that are more interesting to the dialogue object, the quality and effectiveness of the dialogue are improved, enabling intelligent voice dialogue services and improving customer service quality and customer satisfaction.

[0211] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0212] It will be understood by those skilled in the art that Figure 1-7 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0213] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0214] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0215] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0216] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0217] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0218] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0219] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0220] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0221] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain the target text and generate target phoneme data based on the target text; The target text, the preset reference audio, and the target phoneme data are input into a preset speech synthesis model, which includes a hybrid emotion control network, a phoneme coding network, a semantic coding network, a text coding network, and a speech synthesis network. Based on the hybrid emotion control network, emotion features are extracted from the reference audio to obtain the target emotion encoding vector; The target phoneme data is phoneme encoded based on the phoneme encoding network to obtain the target phoneme encoding vector. The target text is encoded using the text encoding network to obtain the target text encoding vector. Based on the semantic coding network, the target emotion coding vector, the target phoneme coding vector, and the target text coding vector are semantically encoded to obtain the target semantic coding vector; Based on the speech synthesis network, speech is synthesized from the target emotion encoding vector, the target semantic encoding vector, and the target text encoding vector to obtain synthesized speech data; The semantic coding network includes a time prediction layer and an attention layer. The semantic coding network is used to perform semantic coding on the target sentiment coding vector, the target phoneme coding vector, and the target text coding vector to obtain a target semantic coding vector, including: Feature extraction is performed on the target phoneme encoding vector to obtain the first phoneme latent vector; Based on the word segmentation boundary information in the target text encoding vector, the phoneme information belonging to the same character in the first phoneme latent vector is pooled to obtain the second phoneme latent vector. The time prediction layer is used to predict the time of each character in the target text encoding vector to obtain the prediction time parameter. The target text encoding vector is then expanded based on the prediction time parameter to obtain the target word embedding vector. The target semantic encoding vector is obtained by performing context encoding on the target word embedding vector, the target sentiment encoding vector, the first phoneme latent vector, and the second phoneme latent vector based on the attention layer.

2. The speech synthesis method according to claim 1, characterized in that, The hybrid emotion control network includes an emotion embedder and an emotion difference module. The extraction of emotion features from the reference audio based on the hybrid emotion control network includes: The reference audio is embedded using the emotion embedder to obtain the target emotion embedding vector. Based on the emotional difference module, the reference audio is subjected to emotional recognition to obtain a preliminary emotional attribute vector; The initial emotion attribute vector is mapped to a preset vector space to obtain the target emotion attribute vector; The target emotion embedding vector and the target emotion attribute vector are added together to obtain the target emotion encoding vector.

3. The speech synthesis method according to claim 2, characterized in that, The emotion embedder includes convolutional layers, GRU layers, and linear layers. The embedding process performed on the reference audio based on the emotion embedder to obtain the target emotion embedding vector includes: Based on the convolutional layer, audio features are extracted from the reference audio to obtain preliminary audio features; Based on the GRU layer, emotion embedding is performed on the preliminary audio features to obtain a preliminary emotion embedding vector; The target emotion embedding vector is obtained by performing a linear transformation on the initial emotion embedding vector based on the linear layer.

4. The speech synthesis method according to claim 1, characterized in that, The speech synthesis network includes a Mel spectrum predictor and a vocoder. The speech synthesis is performed on the target emotion encoding vector, the target semantic encoding vector, and the target text encoding vector using the speech synthesis network to obtain synthesized speech data, including: The target emotion encoding vector, the target semantic encoding vector, and the target text encoding vector are vector-added to obtain the target audio representation vector; Based on the Mel spectrum predictor, the target audio representation vector is predicted to obtain a predicted spectrum feature vector. Based on the vocoder, speech synthesis is performed on the predicted spectral feature vector to obtain the synthesized speech data.

5. The speech synthesis method according to any one of claims 1 to 4, characterized in that, Before inputting the target text, the reference audio, and the target phoneme data into the preset speech synthesis model, the method further includes pre-training the speech synthesis model, specifically including: Obtain sample audio, reference sentiment tags for the sample audio, and sample text; wherein, the sample text includes the audio content of the sample audio; The sample audio, the reference sentiment tag, and the sample text are input into a preset original model; Based on the sample audio and the reference sentiment label, the sample sentiment feature vector and sentiment loss value are obtained; Phoneme encoding is performed on the sample phoneme data corresponding to the sample text to obtain the sample phoneme encoding vector; The sample text is text encoded to obtain the sample text encoding vector; Semantic encoding is performed on the sample sentiment feature vector, the sample phoneme encoding vector, and the sample text encoding vector to obtain the sample semantic encoding vector; The sample emotion feature vector, the sample semantic encoding vector, and the sample text encoding vector are used to perform speech synthesis to obtain the sample synthesized speech; Based on the synthesized speech and the sample audio, a model loss value is constructed; The parameters of the original model are updated based on the model loss value and the sentiment loss value.

6. The speech synthesis method according to claim 5, characterized in that, The process of obtaining the sample emotional feature vector and emotional loss value based on the sample audio and the reference emotional label includes: Based on the sample audio and the reference sentiment label, calculate the sample sentiment embedding vector and the sentiment embedding loss value; Based on the sample audio and the reference sentiment label, calculate the sample sentiment attribute vector and sentiment difference degree; Based on the sample sentiment embedding vector and the sample sentiment attribute vector, the sample sentiment feature vector is obtained; The emotional loss value is obtained based on the emotional embedding loss value and the emotional difference.

7. A speech synthesis device, characterized in that, The device includes: The data acquisition module is used to acquire target text and generate target phoneme data based on the target text; The input module is used to input the target text, the preset reference audio, and the target phoneme data into a preset speech synthesis model, wherein the speech synthesis model includes a hybrid emotion control network, a phoneme coding network, a semantic coding network, a text coding network, and a speech synthesis network. The emotion feature extraction module is used to extract emotion features from the reference audio based on the hybrid emotion control network to obtain a target emotion encoding vector. A phoneme encoding module is used to encode the target phoneme data based on the phoneme encoding network to obtain a target phoneme encoding vector. The text encoding module is used to encode the target text based on the text encoding network to obtain the target text encoding vector; The semantic encoding module is used to perform semantic encoding on the target emotion encoding vector, the target phoneme encoding vector, and the target text encoding vector based on the semantic encoding network to obtain the target semantic encoding vector; The speech synthesis module is used to perform speech synthesis on the target emotion encoding vector, the target semantic encoding vector, and the target text encoding vector based on the speech synthesis network to obtain synthesized speech data; The semantic coding network includes a time prediction layer and an attention layer. The semantic coding network is used to perform semantic coding on the target sentiment coding vector, the target phoneme coding vector, and the target text coding vector to obtain a target semantic coding vector, including: Feature extraction is performed on the target phoneme encoding vector to obtain the first phoneme latent vector; Based on the word segmentation boundary information in the target text encoding vector, the phoneme information belonging to the same character in the first phoneme latent vector is pooled to obtain the second phoneme latent vector. The time prediction layer is used to predict the time of each character in the target text encoding vector to obtain the prediction time parameter. The target text encoding vector is then expanded based on the prediction time parameter to obtain the target word embedding vector. The target semantic encoding vector is obtained by performing context encoding on the target word embedding vector, the target sentiment encoding vector, the first phoneme latent vector, and the second phoneme latent vector based on the attention layer.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method according to any one of claims 1 to 6.

Images