Speech synthesis method, speech synthesis device, electronic device, and storage medium

Abstract

The application provides a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, and belongs to the technical field of financial technology. The method comprises the following steps: obtaining a target text; performing phoneme extraction on the target text to obtain target phoneme data; performing emotion prediction on the reference mel-frequency cepstrum to obtain a target emotion feature vector; performing phoneme coding on the target phoneme data to obtain a target phoneme coding vector; performing vector fusion on the target emotion feature vector and the target phoneme coding vector to obtain a target audio representation vector; and performing speech synthesis on the target audio representation vector to obtain synthesized speech data. The application can be used in the voice call scene in the business field of network shopping, electronic transaction and the like, and the application can improve the voice quality of the synthesized speech.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a speech synthesis method, speech synthesis device, electronic device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent voice interaction is being widely used in finance, logistics, customer service and other fields, improving the service level of enterprise customer service through functions such as intelligent marketing, intelligent debt collection and content navigation.

[0003] Currently, chatbots are frequently used in financial service scenarios such as intelligent customer service and shopping guidance to provide corresponding service support to various users. The dialogue voice used by these chatbots is often generated through speech synthesis.

[0004] In the process of speech synthesis, related technologies often require control over the timbre and emotion of chatbots to improve the quality of synthesized speech. Most methods in these technologies employ separate modeling of reference speaker information and emotional information to control the generation of emotional and speaker features in the synthesized speech. This approach often weakens the emotional information in the synthesized speech, resulting in poor speech quality. Summary of the Invention

[0005] The main objective of this application is to provide a speech synthesis method, speech synthesis device, electronic device, and storage medium, which aim to improve the speech quality of synthesized speech.

[0006] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, the method comprising:

[0007] Obtain the target text;

[0008] Phoneme extraction is performed on the target text to obtain target phoneme data;

[0009] Sentiment prediction is performed on the reference Mel-Cepstral spectra to obtain the target sentiment feature vector;

[0010] The target phoneme data is phoneme encoded to obtain the target phoneme encoding vector;

[0011] The target emotion feature vector and the target phoneme encoding vector are fused to obtain the target audio representation vector;

[0012] The target audio representation vector is used for speech synthesis to obtain synthesized speech data.

[0013] In some embodiments, performing sentiment prediction on the reference Mel-Cepstral spectra to obtain a target sentiment feature vector includes:

[0014] Speaker coding processing is performed on the reference Mel cepstral to obtain speaker coding features;

[0015] The reference Mel cepstral spectrum is subjected to emotion coding processing to obtain audio emotion coding features;

[0016] Prosodic enhancement is performed on the reference Mel cepstral to obtain audio prosodic features;

[0017] The speaker coding features, the audio emotion coding features, and the audio prosody features are summed to obtain the target emotion feature vector.

[0018] In some embodiments, the speaker coding process performed on the reference Mel-Cepstral spectra to obtain speaker coding features includes:

[0019] Feature extraction is performed on the reference Mel cepstral to obtain the first spectral feature vector;

[0020] Speaker embedding is performed on the first spectral feature vector to obtain the speaker embedding vector;

[0021] The speaker's embedded vector is subjected to feature mapping to obtain the speaker's encoded features.

[0022] In some embodiments, the emotional coding processing of the reference Mel-Cepstral spectra to obtain audio emotional coding features includes:

[0023] Feature extraction is performed on the reference Mel cepstral to obtain a second spectral feature vector;

[0024] Emotion embedding is performed on the second spectral feature vector to obtain the emotion embedding vector;

[0025] The emotion embedding vector is subjected to feature mapping to obtain the audio emotion coding features.

[0026] In some embodiments, the prosodic enhancement of the reference Mel cepstral to obtain audio prosodic features includes:

[0027] Feature extraction is performed on the reference Mel cepstral to obtain the audio latent feature vector;

[0028] Prosodic enhancement is performed on the audio latent feature vector to obtain the audio prosodic features.

[0029] In some embodiments, performing prosodic enhancement on the audio latent feature vector to obtain the audio prosodic features includes:

[0030] Feature extraction is performed on the audio latent feature vector to obtain the third spectral feature vector;

[0031] The third spectral feature vector is fused with contextual features to obtain a fused audio feature vector;

[0032] Prosodic embedding is performed on the fused audio features to obtain a prosodic embedding vector;

[0033] The prosodic embedding vector is subjected to feature mapping to obtain the audio prosodic features.

[0034] In some embodiments, the step of performing speech synthesis on the target audio representation vector to obtain synthesized speech data includes:

[0035] Spectral prediction is performed on the target audio representation vector to obtain the predicted spectral feature vector;

[0036] The predicted spectral feature vector is used for speech synthesis to obtain the synthesized speech data.

[0037] To achieve the above objectives, a second aspect of this application provides a speech synthesis apparatus, the apparatus comprising:

[0038] The acquisition module is used to acquire the target text;

[0039] The feature extraction module is used to extract phonemes from the target text to obtain target phoneme data;

[0040] The sentiment prediction module is used to perform sentiment prediction on the reference Mel-Cephal spectrum to obtain the target sentiment feature vector;

[0041] A phoneme encoding module is used to encode the target phoneme data to obtain a target phoneme encoding vector;

[0042] The vector fusion module is used to perform vector fusion on the target emotion feature vector and the target phoneme encoding vector to obtain the target audio representation vector;

[0043] The speech synthesis module is used to perform speech synthesis on the target audio representation vector to obtain synthesized speech data.

[0044] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0045] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0046] The speech synthesis method, speech synthesis device, electronic device, and storage medium proposed in this application acquire target text; extract phonemes from the target text to obtain target phoneme data, which can conveniently obtain the phoneme sequence information corresponding to the target text. Furthermore, emotion prediction is performed on the reference Mel-Cepstral spectrogram to obtain a target emotion feature vector, which can conveniently achieve emotion control in the speech synthesis process and improve the comprehensiveness of the acquired emotion information; phoneme encoding is performed on the target phoneme data to obtain a target phoneme encoding vector; vector fusion is performed on the target emotion feature vector and the target phoneme encoding vector to obtain a target audio representation vector, which enables the target audio representation vector to simultaneously contain the phoneme feature information of the target text and the synthesized speech emotion information determined based on the reference Mel-Cepstral spectrogram. Finally, speech synthesis is performed on the target audio representation vector to obtain synthesized speech data. By using emotion control, the risk of loss of emotional information can be reduced, and the accuracy of emotional information in the synthesized speech data can be improved, thereby improving the speech quality of the synthesized speech data. As a result, in intelligent dialogues involving insurance products, financial products, etc., the synthesized speech expressed by the chatbot can better match the dialogue style preferences of the dialogue object. By adopting dialogue methods and styles that are more interesting to the dialogue object, the quality and effectiveness of dialogue can be improved, enabling intelligent voice dialogue services and improving customer service quality and customer satisfaction. Attached Figure Description

[0047] Figure 1 This is a flowchart of the speech synthesis method provided in the embodiments of this application;

[0048] Figure 2 yes Figure 1 The flowchart of step S103 in the process;

[0049] Figure 3 yes Figure 2 The flowchart of step S201 in the text;

[0050] Figure 4 yes Figure 2 The flowchart of step S202 in the text;

[0051] Figure 5 yes Figure 2 The flowchart of step S203 in the process;

[0052] Figure 6 yes Figure 5 The flowchart of step S502 in the document;

[0053] Figure 7 yes Figure 1 The flowchart of step S106 in the process;

[0054] Figure 8This is a schematic diagram of the speech synthesis device provided in the embodiments of this application;

[0055] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] First, let's analyze some of the terms used in this application:

[0060] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0061] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0062] Information Extraction (IE) is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0063] Mel-Frequency Cipstal Coefficients (MFCCs) are a set of key coefficients used to construct a Mel-Frequency Cipstal spectrum. From a segment of a music signal, a set of cepstrum values ​​can be obtained that is sufficient to represent the music signal. The Mel-Frequency Cipstal Coefficients are the cepstrum values ​​derived from this cepstrum (i.e., the spectrum of the spectrum). Unlike a regular cepstrum, the most distinctive feature of the Mel-Frequency Cipstrum is that its frequency bands are uniformly distributed across the Mel scale. In other words, compared to the linear cepstrum representations commonly seen, this frequency band is closer to the non-linear human auditory system. For example, Mel-Frequency Cipstals are frequently used in audio compression techniques.

[0064] A phoneme is the smallest unit of speech determined by the natural properties of speech. It is analyzed based on the articulation actions within a syllable, with each action constituting a phoneme.

[0065] GRU (Gate Recurrent Unit): GRU is a type of Recurrent Neural Network (RNN), similar to LSTM (Long-Short Term Memory). It was proposed to solve problems such as long-term memory and gradients in backpropagation.

[0066] Speech synthesis refers to the synthesis of intelligible and natural speech from text, also known as text-to-speech (TTS).

[0067] Multi-speaker emotion-controlled speech synthesis model refers to a TTS (Text-to-Speech) system that can control the speaker's timbre and emotion separately. The model can achieve speech synthesis in specific situations by using specified tags.

[0068] With the rapid development of artificial intelligence technology, intelligent voice interaction is being widely used in finance, logistics, customer service and other fields, improving the service level of enterprise customer service through functions such as intelligent marketing, intelligent debt collection and content navigation.

[0069] Currently, chatbots are frequently used in financial service scenarios such as intelligent customer service and shopping guidance to provide corresponding service support to various users. The dialogue voice used by these chatbots is often generated through speech synthesis.

[0070] Taking insurance service robots as an example, it is often necessary to merge the descriptive text of insurance products with the speaking style of a fixed subject to generate a voice description of the insurance product by that fixed subject. When the insurance service robot converses with some interested parties, it will automatically call upon this descriptive voice to introduce the insurance product to those parties.

[0071] In the process of speech synthesis, related technologies often require the control of the timbre and emotion of the chatbot to improve the quality of the synthesized speech. Most methods, in order to enable the speech synthesis model to decouple speaker timbre and emotional information, often adopt a method of separately modeling speaker information and emotional information to control the generation of emotional features and speaker features in the synthesized speech. That is, speaker information is removed when modeling emotional information, and emotional information is removed when modeling speaker timbre information. This approach often weakens the emotional information in the synthesized speech, resulting in poor speech quality.

[0072] Based on this, embodiments of this application provide a speech synthesis method, a speech synthesis apparatus, an electronic device, and a storage medium, aiming to improve the speech quality of synthesized speech.

[0073] The speech synthesis method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the speech synthesis method in this application is described.

[0074] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0075] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0076] The speech synthesis method provided in this application relates to the field of artificial intelligence technology. The speech synthesis method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech synthesis method, but is not limited to the above forms.

[0077] The speech synthesis method of this application can be used in voice call scenarios during electronic transactions, as well as in various scenarios such as online shopping and e-commerce, without limitation.

[0078] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0079] Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0080] Step S101: Obtain the target text;

[0081] Step S102: Extract phonemes from the target text to obtain target phoneme data;

[0082] Step S103: Perform sentiment prediction on the reference Mel cepstral spectrum to obtain the target sentiment feature vector;

[0083] Step S104: Phoneme encoding is performed on the target phoneme data to obtain the target phoneme encoding vector;

[0084] Step S105: Perform vector fusion on the target emotion feature vector and the target phoneme encoding vector to obtain the target audio representation vector;

[0085] Step S106: Perform speech synthesis on the target audio representation vector to obtain synthesized speech data.

[0086] Steps S101 to S106 of this embodiment involve acquiring target text; extracting phonemes from the target text to obtain target phoneme data, which facilitates the acquisition of phoneme sequence information corresponding to the target text; performing sentiment prediction on the reference Mel-Cepstral spectrogram to obtain a target sentiment feature vector, which facilitates sentiment control in the speech synthesis process and improves the comprehensiveness of the acquired sentiment information; encoding the target phoneme data to obtain a target phoneme encoding vector; and fusing the target sentiment feature vector and the target phoneme encoding vector to obtain a target audio representation vector, which enables the target audio representation vector to simultaneously contain the phoneme feature information of the target text and the sentiment information of the synthesized speech determined based on the reference Mel-Cepstral spectrogram. Finally, speech synthesis is performed on the target audio representation vector to obtain synthesized speech data. This approach reduces the risk of losing sentiment information through sentiment control, improves the accuracy of sentiment information in the synthesized speech data, and thus improves the speech quality of the synthesized speech data.

[0087] In step S101 of some embodiments, obtaining the target text includes, but is not limited to, the following methods:

[0088] (1) The target text is obtained by purposefully crawling the preset data source through web crawler. The preset data source includes a preset database or other network platforms that can provide text materials for speech synthesis.

[0089] (2) Obtain the raw text from a public dataset. The public dataset can be the LJSpeech dataset, which contains multiple English speech data recorded by the speakers and the corresponding text data.

[0090] The above methods make it relatively easy to obtain target text, extract the required information from multiple information channels, and improve data acquisition efficiency.

[0091] It should be noted that the target text may contain financial terminology, financial business templates, product descriptions of insurance products and wealth management products, as well as commonly used dialogue phrases in the financial field.

[0092] In step S102 of some embodiments, when extracting phonemes from the target text to obtain target phoneme data, the target text can be converted using a preset text conversion model and a reference dictionary to obtain the target phoneme data. Specifically, the preset text conversion model can be an open-source text-to-phoneme model, such as the g2p-seq2seq model, etc., and the reference dictionary can be the CMUSphinx data dictionary, which contains multiple words. A word list can be constructed using the text conversion model and the reference dictionary. The word list is formatted as one word or character per line. This word list can be used to convert the text content in the target text, transforming the corresponding words or characters into phoneme sequences, thereby forming the target phoneme data corresponding to the target text. This method can improve the accuracy and efficiency of the generated target phoneme data.

[0093] Before step S103 in some embodiments, it is also necessary to obtain a preset reference Mel cepstrum. The preset reference Mel cepstrum can be spectrum data extracted from reference audio collected from various databases or network platforms and stored in a fixed repository. The reference audio can be audio data from different application scenarios, languages, styles, and genres, without limitation.

[0094] For example, in financial transaction scenarios, reference audio is audio data containing commonly used dialogue in the financial field; in insurance sales scenarios, reference audio is audio data containing descriptions of a particular insurance product, such as its type, cost, and target audience.

[0095] In some embodiments, the speech synthesis method of this application can be implemented based on a preset speech synthesis model, which includes an emotion control network, a text encoding network, and a speech synthesis network. The emotion control network is mainly used to extract style, emotion, and prosodic enhancement control from multiple speakers based on the input reference Mel-Cepstral, obtaining fused feature information containing speaker style information, enhanced prosodic information, and emotional information. This compensates for the loss of prosodic information during the decoupling process of speaker timbre and emotion, improving the comprehensiveness of the fused feature information. The text encoding network is mainly used to encode the input phoneme data and extract phoneme feature information. The speech synthesis network is mainly used to perform speech synthesis using the fused feature information and phoneme feature information, generating synthesized speech data corresponding to the input text. This speech synthesis model can introduce multi-speaker emotion control with prosodic enhancement during the speech synthesis process, effectively compensating for the loss of prosodic information during the decoupling process of speaker timbre and emotion, achieving high-quality modeling and control of speakers and emotions, thereby improving the speech quality of the generated synthesized speech data.

[0096] Please see Figure 2In some embodiments, the emotion control network includes a speaker encoding module, an emotion encoding module, and a prosody enhancement module. Step S103 may include, but is not limited to, steps S201 to S204:

[0097] Step S201: Perform speaker coding on the reference Mel cepstral to obtain speaker coding features;

[0098] Step S202: Perform emotion coding processing on the reference Mel cepstral to obtain audio emotion coding features;

[0099] Step S203: Perform prosodic enhancement on the reference Mel cepstral to obtain audio prosodic features;

[0100] Step S204: Add the speaker coding features, audio emotion coding features, and audio prosody features to obtain the target emotion feature vector.

[0101] In step S201 of some embodiments, the speaker encoding module may include a first convolutional layer, a first GRU layer, and a first fully connected layer. Specifically, feature extraction is performed on the reference Mel-Cepstral spectra based on the first convolutional layer to obtain a first spectral feature vector; speaker embedding is performed on the first spectral feature vector based on the first GRU layer to obtain a speaker embedding vector; and feature mapping is performed on the speaker embedding vector based on the first fully connected layer to obtain speaker encoding features. This method can conveniently extract speaker style information from the reference Mel-Cepstral spectra to obtain speaker encoding features containing speaker style information. Speaker style information includes the speaker's timbre characteristics, speech rate characteristics, pitch characteristics, and so on.

[0102] In step S202 of some embodiments, the emotion coding module includes a second convolutional layer, a second GRU layer, and a second fully connected layer. Specifically, feature extraction is performed on the reference Mel-Cepstral spectra based on the second convolutional layer to obtain a second spectral feature vector; emotion embedding is performed on the second spectral feature vector based on the second GRU layer to obtain an emotion embedding vector; and feature mapping is performed on the emotion embedding vector based on the second fully connected layer to obtain audio emotion coding features. This method can conveniently extract emotion feature information from the reference Mel-Cepstral spectra to obtain audio emotion coding features containing emotion feature information. The emotion feature information includes the emotion type contained in the reference Mel-Cepstral spectra, which includes emotions such as happiness, sadness, and calmness, without limitation.

[0103] In step S203 of some embodiments, the prosody enhancement module may include a speech recognizer and a prosody encoder. Specifically, the speech recognizer extracts features from a reference Mel-Cepstral spectrogram to obtain an audio latent feature vector; the prosody encoder then enhances the audio latent feature vector to obtain audio prosodic features.

[0104] In step S204 of some embodiments, since the outputs of the speaker coding module, the emotion coding module, and the prosody enhancement module are set to the same dimension during model training, the speaker coding features, audio emotion coding features, and audio prosody features are feature vectors of the same dimension. The speaker coding features, audio emotion coding features, and audio prosody features can be directly added together to obtain the target emotion feature vector.

[0105] Through the above steps S201 to S204, multi-speaker emotion control with prosodic enhancement can be introduced into the speech synthesis process. This can effectively compensate for the loss of prosodic information in the process of decoupling speaker timbre and emotion, and achieve the goal of high-quality modeling and control of speaker and emotion. It helps to improve the comprehensive extraction of speaker style information and emotional information in the speech synthesis process, and is conducive to improving the speech quality of synthesized speech data.

[0106] Please see Figure 3 In some embodiments, the speaker encoding module includes a first convolutional layer, a first GRU layer, and a first fully connected layer, and step S201 may include, but is not limited to, steps S301 to S303:

[0107] Step S301: Extract features from the reference Mel cepstral to obtain the first spectral feature vector;

[0108] Step S302: Speaker embedding is performed on the first spectral feature vector to obtain the speaker embedding vector;

[0109] Step S303: Perform feature mapping on the speaker embedding vector to obtain the speaker coding features.

[0110] In step S301 of some embodiments, feature extraction can be performed on the reference Mel cepstral based on the first convolutional layer to capture speaker style information in the reference Mel cepstral and obtain a first spectral feature vector.

[0111] In step S302 of some embodiments, speaker embedding can be performed on the first spectral feature vector based on the first GRU layer to obtain a speaker embedding vector. The first spectral feature vector at each time step and the hidden layer state features of the first spectral feature vector at the previous time step are input into the first GRU layer. Based on the first GRU layer, feature extraction is performed on the first spectral feature vector at each time step and the hidden layer state features of the first spectral feature vector at the previous time step. The hidden layer state features of the first spectral feature vector at that time step and the speaker embedding vector are output. The hidden layer state features at that time step are used to provide the first spectral feature vector at the next time step for speaker style embedding.

[0112] In step S303 of some embodiments, a linear transformation is performed on the speaker embedding vector based on the first fully connected layer to change the feature dimension of the speaker embedding vector, converting the high-dimensional speaker embedding vector into a low-dimensional vector to obtain speaker coding features, so that the speaker coding features can contain richer speaker style feature information.

[0113] Through the above steps S301 to S303, the speaker style embedding of the reference Mel cepstral can be achieved relatively easily, and the speaker style feature information of the reference Mel cepstral can be extracted more accurately and comprehensively, thereby improving the feature quality of the speaker coding features and improving the speech quality of the generated synthetic speech data.

[0114] Please see Figure 4 In some embodiments, the emotion encoding module includes a second convolutional layer, a second GRU layer, and a second fully connected layer. Step S202 may include, but is not limited to, steps S401 to S403:

[0115] Step S401: Extract features from the reference Mel cepstral to obtain the second spectral feature vector;

[0116] Step S402: Perform sentiment embedding on the second spectral feature vector to obtain the sentiment embedding vector;

[0117] Step S403: Perform feature mapping on the emotion embedding vector to obtain audio emotion coding features.

[0118] In step S401 of some embodiments, feature extraction is performed on the reference Mel-Cepstral based on the second convolutional layer to capture the emotional feature information in the reference Mel-Cepstral and obtain the second spectral feature vector.

[0119] In step S402 of some embodiments, when performing sentiment embedding on the second spectral feature vector based on the second GRU layer to obtain the sentiment embedding vector, the second spectral feature vector at each time step and the hidden layer state features of the second spectral feature vector at the previous time step are input together into the second GRU layer. Based on the second GRU layer, feature extraction is performed on the second spectral feature vector at each time step and the hidden layer state features of the second spectral feature vector at the previous time step, and the hidden layer state features of the second spectral feature vector at that time step and the sentiment embedding vector are output. The hidden layer state features at that time step are used to provide the second spectral feature vector at the next time step for sentiment embedding.

[0120] In step S403 of some embodiments, a linear transformation is performed on the emotion embedding vector based on the second fully connected layer to change the feature dimension of the emotion embedding vector, converting the high-dimensional emotion embedding vector into a low-dimensional vector, thereby obtaining audio emotion coding features, so that the audio emotion coding features can contain more accurate emotion feature information.

[0121] Through the above steps S401 to S403, the emotion embedding of the reference Mel cepstral can be realized relatively easily, and the emotion feature information of the reference Mel cepstral can be extracted more accurately and comprehensively, thereby improving the feature quality of audio emotion coding features and improving the speech quality of the generated synthetic speech data.

[0122] Please see Figure 5 In some embodiments, the prosody enhancement module includes a speech recognizer and a prosody encoder, and step S203 includes, but is not limited to, steps S501 to S502:

[0123] Step S501: Extract features from the reference Mel cepstral to obtain the audio latent feature vector;

[0124] Step S502: Perform prosodic enhancement on the audio latent feature vector to obtain audio prosodic features.

[0125] In step S501 of some embodiments, to compensate for the loss of prosodic information caused by the decoupling of information between the speaker encoding module and the emotion encoding module, a speech recognizer based on Automatic Speech Recognition (ASR) technology is set in the prosodic enhancement module. Specifically, based on the speech recognizer, features are extracted from the reference Mel-Cepstral spectra to obtain audio latent state feature information in the reference Mel-Cepstral spectra, resulting in an audio latent feature vector. This audio latent feature vector contains prosodic information features unrelated to the speaker. Therefore, using the audio latent feature vector as input to the prosodic encoder can effectively compensate for the emotional information of the entire emotion control module, thereby compensating for the weakening of prosodic information and the loss of emotional information during speech synthesis.

[0126] In step S502 of some embodiments, the prosodic encoder may include a third convolutional layer, a global context module, a third GRU layer, and a third fully connected layer. Specifically, feature extraction is performed on the audio latent feature vector based on the third convolutional layer to obtain a third spectral feature vector; contextual feature fusion is performed on the third spectral feature vector based on the global context module to obtain a fused audio feature vector; prosodic embedding is performed on the fused audio feature vector based on the third GRU layer to obtain a prosodic embedding vector; and feature mapping is performed on the prosodic embedding vector based on the third fully connected layer to obtain audio prosodic features. Setting a global context module in the prosodic encoder can achieve the effect of context modeling, thereby reducing the risk that the emotional information conveyed by the emotional embedding will be weakened due to the entanglement of emotional information and speaker information, resulting in decoupling from speaker-related information. In this way, the prosodic encoder can be used to extract global contextual information from the audio latent feature vector, and the prosodic and emotional information in the speech synthesis process can be extracted more comprehensively.

[0127] Through the above steps S501 to S502, the audio latent feature vectors generated by the speech recognizer, which contain prosodic information features unrelated to the speaker, can be better utilized. By using the prosodic encoder to extract global contextual information from the audio latent feature vectors, the prosodic information and emotional information in the speech synthesis process can be extracted more comprehensively. This effectively compensates for the loss of prosodic information in the process of decoupling the speaker's timbre and emotion in speech synthesis, thereby improving the information comprehensiveness and speech quality of the synthesized speech data.

[0128] Please see Figure 6 In some embodiments, the prosody encoder includes a third convolutional layer, a global context module, a third GRU layer, and a third fully connected layer. Step S502 may include, but is not limited to, steps S601 to S604:

[0129] Step S601: Extract features from the audio latent feature vector to obtain the third spectral feature vector;

[0130] Step S602: Perform context feature fusion on the third spectral feature vector to obtain a fused audio feature vector;

[0131] Step S603: Perform prosodic embedding on the fused audio feature vector to obtain the prosodic embedding vector;

[0132] Step S604: Perform feature mapping on the prosody embedding vector to obtain audio prosody features.

[0133] In step S601 of some embodiments, feature extraction is performed on the audio latent feature vector based on the third convolutional layer to capture prosodic feature information in the reference Mel cepstral, and a third spectral feature vector is obtained, in which the prosodic features are independent of the speaker.

[0134] In step S602 of some embodiments, the global context module is mainly used to realize global context modeling, capture inter-channel dependencies and feature fusion. The specific process of obtaining the fused audio feature vector by performing global context feature fusion on the third spectral feature vector based on the global context module can be represented as shown in formula (1).

[0135]

[0136] Where, x i x represents the i-th third spectral feature vector of the input global context module; j x represents the j-th third spectral feature vector of the input global context module; m z represents the m-th third spectral feature vector of the input global context module; i This represents the i-th fused audio feature vector output by the global context module. Np represents the total number of third spectral feature vectors. This represents the j-th third spectral eigenvector x. i The result of exp calculation after 1×1 convolution; This represents the m-th third spectral eigenvector x. m The result of exp calculation after 1×1 convolution. This represents the weights of the global attention pooling.

[0137] From the above formula (1), it can be seen that the global context module has a certain relationship with the i-th third spectral feature vector x. i Global contextual feature fusion is performed to obtain the fused audio feature vector z. i The process includes: first, applying global attention pooling weights and the j-th third spectral feature vector x... j Perform a dot product, then sequentially process the dot product result through convolution, normalization, activation, and convolution again to obtain an intermediate feature vector. Finally, combine the intermediate feature vector with the i-th third spectral feature vector x. i By adding the vectors together, we obtain the fused audio feature vector z. i .

[0138] In step S603 of some embodiments, when performing prosodic embedding on the fused audio feature vector based on the third GRU layer to obtain the prosodic embedding vector, the fused audio feature vector at each time step and the hidden layer state features of the fused audio feature vector at the previous time step are input together to the third GRU layer. Based on the third GRU layer, feature extraction is performed on the fused audio feature vector at each time step and the hidden layer state features of the fused audio feature vector at the previous time step, and the hidden layer state features of the fused audio feature vector at that time step and the prosodic embedding vector are output. The hidden layer state features at that time step are used to provide the fused audio feature vector at the next time step for prosodic embedding.

[0139] In step S604 of some embodiments, a linear transformation is performed on the prosody embedding vector based on the third fully connected layer to change the feature dimension of the prosody embedding vector, converting the high-dimensional prosody embedding vector into a low-dimensional vector to obtain audio prosodic features, so that the audio prosodic features can contain more accurate prosodic feature information.

[0140] It should be noted that the number of the third convolutional layer, global context module, third GRU layer, and third fully connected layer in the prosody encoder can be set according to actual needs and is not limited.

[0141] For example, in one specific embodiment, the prosodic encoder includes six third convolutional layers, three global context modules, one third GRU layer, and one third fully connected layer. The reference Mel-Cepstral spectrogram is first input to the first third convolutional layer. The output of the first third convolutional layer serves as the input to the first global context module. The output of the first global context module serves as the input to the second third convolutional layer. The output of the second third convolutional layer serves as the input to the third third convolutional layer. The output of the third third convolutional layer serves as the input to the second global context module. The output of the second global context module serves as the input to the fourth third convolutional layer. The output of the fourth third convolutional layer serves as the input to the fifth third convolutional layer. The output of the fifth third convolutional layer serves as the input to the third global context module. The output of the third global context module serves as the input to the sixth third convolutional layer. The output of the sixth third convolutional layer serves as the input to the third GRU layer. The output of the third GRU layer serves as the input to the third fully connected layer. The output of the third fully connected layer is used as the final audio prosodic feature.

[0142] By implementing steps S601 to S604, setting a global context module in the prosodic encoder achieves contextual modeling, enhancing the emotional information learned by the speech synthesis model. Based on this prosodic encoder, the speech synthesis model can introduce multi-speaker emotion control with prosodic enhancement during speech synthesis, effectively compensating for the loss of prosodic information during the decoupling of speaker timbre and emotion. This achieves high-quality modeling and control of speaker and emotion, reducing the risk of weakened emotional information conveyed by the emotional embedding due to the entanglement of emotional and speaker information. This approach allows for the extraction of global contextual information from the audio latent feature vector using the prosodic encoder, enabling a more comprehensive extraction of prosodic and emotional information during speech synthesis, thereby improving the speech quality of the generated synthesized speech data.

[0143] In step S104 of some embodiments, when encoding the target phoneme data based on a text encoding network to obtain the target phoneme encoding vector, the RoBERTa model can be used as the text encoding network. Using the RoBERTa model as the text encoding network to encode the target phoneme data can more easily extract the context information of the target phoneme data, so that as much of the feature information of the target phoneme data as possible can be extracted, thereby improving the feature quality of the generated target phoneme encoding vector.

[0144] In step S105 of some embodiments, when performing vector fusion on the target emotion feature vector and the target phoneme encoding vector, the target emotion feature vector and the target phoneme encoding vector can be directly added or concatenated to obtain the target audio representation vector, which can improve computational efficiency. Alternatively, the target emotion feature vector and the target phoneme encoding vector can be first feature mapped to transform them into a vector space of the same dimension, so that the vector dimensions of the target emotion feature vector and the target phoneme encoding vector are the same. Then, the target emotion feature vector and the target phoneme encoding vector in the same dimension vector space are added or concatenated to obtain the target audio representation vector. This method can improve the rationality of feature fusion and improve the feature quality of the target audio representation vector.

[0145] Please see Figure 7 In some embodiments, the speech synthesis network includes a Mel spectrum predictor and a vocoder, and step S106 may include, but is not limited to, steps S701 to S702:

[0146] Step S701: Perform spectrum prediction on the target audio representation vector to obtain the predicted spectrum feature vector;

[0147] Step S702: Speech synthesis is performed on the predicted spectral feature vector to obtain synthesized speech data.

[0148] In step S701 of some embodiments, the Mel spectrum predictor can be a Tacotron2 model or a commonly used model in speech synthesis in related technologies, without limitation. Taking the Tacotron2 model as an example, the Mel spectrum predictor is a seq2seq network with an attention mechanism. The Mel spectrum predictor includes an encoder and a decoder. The encoder is a module consisting of a style embedding layer, three convolutional layers, and a bidirectional LSTM layer. First, the style embedding layer performs word embedding on the target audio representation vector to obtain an audio character sequence. Then, the word-embedded audio character sequence is input into the three convolutional layers for context information extraction to obtain an audio context vector. The audio context vector is then input into the bidirectional LSTM layer to obtain a spectral hidden state vector. The decoder includes two bidirectional LSTM layers and a projection layer. The bidirectional LSTM layers extract new context information from the spectral hidden state vector to obtain a spectral context vector. Then, the projection layer predicts the spectral context vector to obtain a predicted spectral feature vector.

[0149] Furthermore, in some other embodiments, in order to improve the feature quality of the spectral features, the output of the decoder can be fed into a series of convolutional layers for convolution processing, and the output of the last convolutional layer can be used as the predicted spectral feature vector.

[0150] In step S702 of some embodiments, the vocoder may be a HiFi-GAN vocoder, which includes an upsampling module and a residual module for multi-receptive field fusion. The upsampling module can perform upsampling processing on the predicted spectral feature vector to achieve transpose convolution of the predicted spectral feature vector to obtain initial speech features. The residual module is then used to reconstruct the initial speech features to obtain the reconstructed speech waveform, which is then used as the synthesized speech data.

[0151] In a specific example, the synthesized speech data consists of a description of insurance or financial products, containing the speaking style and emotional tone of an animated character. This synthesized speech data can attract potential viewers with the unique speaking style and emotional tone of the animated character, making them more interested in the insurance or financial products recommended by the synthesized speech data.

[0152] Through the above steps S701 to S702, the synthesized speech data can simultaneously include emotional information, speaker style information, phoneme feature information of the target text, and text content information from the reference Mel cepstral. This effectively improves the accuracy of speech synthesis and the speech quality of the synthesized speech data.

[0153] The speech synthesis method of this application embodiment obtains target text; extracts phonemes from the target text to obtain target phoneme data, which can conveniently obtain the phoneme sequence information corresponding to the target text. Furthermore, based on a preset speech synthesis model, an emotion control network performs emotion prediction on a reference Mel-Cepstral spectrogram to obtain a target emotion feature vector, which can conveniently achieve emotion control in the speech synthesis process and improve the comprehensiveness of the obtained emotion information. Specifically, the emotion control network includes a speaker encoding module, an emotion encoding module, and a prosodic enhancement module containing a global context module, which can effectively reduce the loss of prosodic information. Based on a text encoding network, the target phoneme data is phoneme encoded to obtain a target phoneme encoding vector; the target emotion feature vector and the target phoneme encoding vector are vector-added to obtain a target audio representation vector, which allows the target audio representation vector to simultaneously contain the phoneme feature information of the target text and the synthesized speech emotion information determined based on the reference Mel-Cepstral spectrogram. Finally, based on the speech synthesis network, speech is synthesized from the target audio representation vector to obtain synthesized speech data. This enables the introduction of multi-speaker emotion control with prosodic enhancement during the speech synthesis process. It can effectively compensate for the loss of prosodic information during the decoupling of speaker timbre and emotion, achieving the goal of high-quality modeling and control of speaker and emotion. This improves the accuracy of emotional information in the synthesized speech data, thereby improving the speech quality of the synthesized speech data. Consequently, in intelligent dialogues involving insurance products, financial products, etc., the synthesized speech expressed by the chatbot can better match the dialogue style preferences of the dialogue object. By adopting dialogue methods and styles that are more interesting to the dialogue object, the quality and effectiveness of dialogue are improved, enabling intelligent voice dialogue services and improving customer service quality and customer satisfaction.

[0154] Please see Figure 8 This application also provides a speech synthesis apparatus that can implement the above-described speech synthesis method. The apparatus includes:

[0155] Module 801 is used to acquire the target text;

[0156] The feature extraction module 802 is used to extract phonemes from the target text to obtain target phoneme data;

[0157] The sentiment prediction module 803 is used to perform sentiment prediction on the reference Mel cepstral spectrum to obtain the target sentiment feature vector.

[0158] The phoneme encoding module 804 is used to encode the target phoneme data to obtain the target phoneme encoding vector.

[0159] The vector fusion module 805 is used to perform vector fusion on the target emotion feature vector and the target phoneme encoding vector to obtain the target audio representation vector.

[0160] The speech synthesis module 806 is used to synthesize speech from the target audio representation vector to obtain synthesized speech data.

[0161] The specific implementation of this speech synthesis device is basically the same as the specific implementation of the speech synthesis method described above, and will not be repeated here.

[0162] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned speech synthesis method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0163] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0164] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0165] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the speech synthesis method of the embodiments of this application.

[0166] The input / output interface 903 is used to implement information input and output;

[0167] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0168] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0169] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0170] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the above-described speech synthesis method.

[0171] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0172] The speech synthesis method, speech synthesis device, electronic device, and computer-readable storage medium provided in this application embodiment acquire target text; extract phonemes from the target text to obtain target phoneme data, which can conveniently obtain the phoneme sequence information corresponding to the target text. Furthermore, an emotion control network based on a preset speech synthesis model performs emotion prediction on a reference Mel-Cepstral spectrogram to obtain a target emotion feature vector, which can conveniently achieve emotion control in the speech synthesis process and improve the comprehensiveness of the acquired emotion information. Specifically, setting a speaker encoding module, an emotion encoding module, and a prosodic enhancement module containing a global context module in the emotion control network can effectively reduce the loss of prosodic information. Phoneme encoding of the target phoneme data based on a text encoding network yields a target phoneme encoding vector; vector addition of the target emotion feature vector and the target phoneme encoding vector yields a target audio representation vector, which enables the target audio representation vector to simultaneously contain the phoneme feature information of the target text and the synthesized speech emotion information determined based on the reference Mel-Cepstral spectrogram. Finally, based on the speech synthesis network, speech is synthesized from the target audio representation vector to obtain synthesized speech data. This enables the introduction of multi-speaker emotion control with prosodic enhancement during the speech synthesis process. It can effectively compensate for the loss of prosodic information during the decoupling of speaker timbre and emotion, achieving the goal of high-quality modeling and control of speaker and emotion. This improves the accuracy of emotional information in the synthesized speech data, thereby improving the speech quality of the synthesized speech data. Consequently, in intelligent dialogues involving insurance products, financial products, etc., the synthesized speech expressed by the chatbot can better match the dialogue style preferences of the dialogue object. By adopting dialogue methods and styles that are more interesting to the dialogue object, the quality and effectiveness of dialogue are improved, enabling intelligent voice dialogue services and improving customer service quality and customer satisfaction.

[0173] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0174] It will be understood by those skilled in the art that Figure 1-7 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0175] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0177] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0178] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0180] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0182] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method characterized by, The method comprises: obtaining target text; performing phoneme extraction on the target text to obtain target phoneme data; performing emotion prediction on the reference mel-spectrogram to obtain a target emotion feature vector; performing phoneme coding on the target phoneme data to obtain a target phoneme coding vector; performing vector fusion on the target emotion feature vector and the target phoneme coding vector to obtain a target audio representation vector; performing speech synthesis on the target audio representation vector to obtain synthesized speech data; The emotion prediction on the reference mel-spectrogram to obtain a target emotion feature vector comprises: performing speaker coding processing on the reference mel-spectrogram to obtain a speaker coding feature; performing emotion coding processing on the reference mel-spectrogram to obtain an audio emotion coding feature; performing feature extraction on the reference mel-spectrogram to obtain an audio hidden feature vector; performing prosody enhancement on the audio hidden feature vector to obtain an audio prosody feature; performing feature addition on the speaker coding feature, the audio emotion coding feature, and the audio prosody feature to obtain the target emotion feature vector.

2. The speech synthesis method of claim 1, wherein, The speaker coding processing on the reference mel-spectrogram to obtain a speaker coding feature comprises: performing feature extraction on the reference mel-spectrogram to obtain a first frequency spectrum feature vector; performing speaker embedding on the first frequency spectrum feature vector to obtain a speaker embedding vector; performing feature mapping on the speaker embedding vector to obtain the speaker coding feature.

3. The speech synthesis method of claim 1, wherein, The emotion coding processing on the reference mel-spectrogram to obtain an audio emotion coding feature comprises: performing feature extraction on the reference mel-spectrogram to obtain a second frequency spectrum feature vector; performing emotion embedding on the second frequency spectrum feature vector to obtain an emotion embedding vector; performing feature mapping on the emotion embedding vector to obtain the audio emotion coding feature.

4. The speech synthesis method of claim 1, wherein, The prosody enhancement on the audio hidden feature vector to obtain the audio prosody feature comprises: performing feature extraction on the audio hidden feature vector to obtain a third frequency spectrum feature vector; performing context feature fusion on the third frequency spectrum feature vector to obtain a fused audio feature vector; performing prosody embedding on the fused audio feature to obtain a prosody embedding vector; performing feature mapping on the prosody embedding vector to obtain the audio prosody feature.

5. The speech synthesis method according to any one of claims 1 to 4, characterized by, The speech synthesis on the target audio representation vector to obtain synthesized speech data comprises: performing frequency spectrum prediction on the target audio representation vector to obtain a predicted frequency spectrum feature vector; performing speech synthesis on the predicted frequency spectrum feature vector to obtain the synthesized speech data.

6. A speech synthesis apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire target text; a feature extraction module configured to perform phoneme extraction on the target text to obtain target phoneme data; an emotion prediction module configured to perform emotion prediction on a reference mel-spectrogram to obtain a target emotion feature vector; a phoneme coding module configured to perform phoneme coding on the target phoneme data to obtain a target phoneme coding vector; a vector fusion module configured to perform vector fusion on the target emotion feature vector and the target phoneme coding vector to obtain a target audio representation vector; The voice synthesis module is configured to perform voice synthesis on the target audio feature vector to obtain synthesized voice data. The emotion prediction on the reference mel-frequency cepstrum comprises: performing speaker coding processing on the reference mel-frequency cepstrum to obtain speaker coding features; performing emotion coding processing on the reference mel-frequency cepstrum to obtain audio emotion coding features; performing feature extraction on the reference mel-frequency cepstrum to obtain an audio hidden feature vector; and performing prosody enhancement on the audio hidden feature vector to obtain audio prosody features; 7. An electronic device, comprising: performing feature addition on the speaker coding features, the audio emotion coding features, and the audio prosody features to obtain the target emotion feature vector.

8. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7. The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the voice synthesis method in any one of claims 1 to 5 when executing the computer program. The computer program is executed by the processor to implement the voice synthesis method in any one of claims 1 to 5.

Images