Speech synthesis method, speech synthesis device, electronic device, and storage medium

Abstract

Embodiments of the present application provide a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, and relate to the technical field of artificial intelligence and digital medical treatment. The speech synthesis method obtains an auxiliary emotion label of target text information by using a cross-domain emotion recognition sub-model; obtains a prosody embedding of a reference speech by using a prosody coding sub-model; obtains a pitch feature vector by using a pitch prediction sub-model; obtains a duration feature vector by using a duration prediction sub-model; and performs speech synthesis by using the pitch feature vector, the duration feature vector and a text coding vector to obtain speech content corresponding to the target text information. The auxiliary emotion label of the target text information is generated by using the cross-domain emotion recognition sub-model, and the correlation between the auxiliary emotion label and the prosody is combined in the speech synthesis process, so that the reference speech is selected to generate synthesized speech containing emotions, the naturalness of the synthesized speech is improved, and the application range of the text-to-speech synthesis technology is expanded.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and digital medical technology, and in particular to speech synthesis methods, speech synthesis devices, electronic devices and storage media. Background Technology

[0002] In recent years, speech synthesis technology, which converts text into speech, has been gradually applied to speech signal processing systems such as voice interaction, voice broadcasting, and personalized voice production. Effective use of speech synthesis technology can improve the user experience of voice interaction and has broad potential applications. Simultaneously, with the rise of speech synthesis technology, it can also support the needs of digital healthcare scenarios such as health management and electronic medical records.

[0003] In related technologies, for Text-to-Speech (TTS) processes, to improve the naturalness of the synthesized speech, it is necessary to incorporate emotional information into the speech synthesis model for speech content prediction. However, these technologies do not utilize the correlation between emotional information and prosody in speech content synthesis, resulting in low naturalness. Therefore, how to improve the utilization of the correlation between emotional information and prosody in speech synthesis to enhance its naturalness has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this application is to propose a speech synthesis method, a speech synthesis device, an electronic device, and a storage medium to improve the naturalness of speech synthesis results and expand the applicability of speech synthesis technology.

[0005] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, comprising:

[0006] The acquired target text information is input into the speech synthesis model, which includes: a cross-domain emotion recognition sub-model, a prosody coding sub-model, a pitch prediction sub-model, and a duration prediction sub-model.

[0007] The cross-domain sentiment recognition sub-model is used to perform sentiment recognition on the target text information to obtain the auxiliary sentiment label of the target text information;

[0008] The prosodic encoding sub-model is used to perform prosodic encoding on the reference speech based on the auxiliary emotion identifier to obtain the prosodic embedding of the reference speech;

[0009] The pitch prediction sub-model is used to predict the pitch of the prosodic embedding and the text encoding vector of the target text information to obtain a pitch feature vector.

[0010] The duration prediction sub-model is used to predict the duration of the prosodic embedding and the text encoding vector of the target text information to obtain a duration feature vector.

[0011] Speech synthesis is performed using the pitch feature vector, the duration feature vector, and the text encoding vector to obtain the speech content corresponding to the target text information.

[0012] In one embodiment, the cross-domain emotion recognition sub-model includes a first encoder, a second encoder, and a classifier; before performing emotion recognition on the target text information using the cross-domain emotion recognition sub-model to obtain the auxiliary emotion label of the target text information, the method further includes:

[0013] Obtain a speech emotion recognition dataset, which includes: a first dataset and a second dataset, wherein the first dataset includes a first speech sample and a speech emotion label, and the second dataset includes a second text sample;

[0014] The first encoder is used to generate a first data feature based on the first speech sample;

[0015] The second encoder is used to generate second data features based on the second text sample;

[0016] The parameters of the first encoder and the second encoder are adjusted according to the distribution loss values ​​of the first data feature and the second data feature until the distribution loss value reaches the first preset convergence condition.

[0017] The first data features are input into the classifier to obtain the predicted sentiment;

[0018] Based on the predicted emotion and the voice emotion label, the recognition loss value is obtained, and the parameters of the classifier are adjusted until the recognition loss value reaches the second preset convergence condition, thus obtaining the trained cross-domain emotion recognition sub-model.

[0019] In one embodiment, adjusting the parameters of the first encoder and the second encoder based on the distribution loss values ​​of the first data feature and the second data feature until the distribution loss value reaches a first preset convergence condition includes:

[0020] Obtain the first expected value of the first data feature after it has been mapped by a preset mapping function;

[0021] Obtain the second expected value of the second data feature after it has been mapped by a preset mapping function;

[0022] The distribution loss value is obtained based on the first expected value and the second expected value;

[0023] The parameters of the first encoder and the second encoder are adjusted until the distributed loss value reaches the first preset convergence condition, which is the minimum distributed loss value.

[0024] In one embodiment, before performing prosodic encoding on the reference speech based on the auxiliary emotion identifier using the prosodic encoding sub-model to obtain the prosodic embedding of the reference speech, the method further includes: selecting reference speech from a reference speech library based on the auxiliary emotion identifier, wherein the reference speech library includes reference speech of multiple speech emotion categories.

[0025] In one embodiment, the step of using the prosodic coding sub-model to perform prosodic coding on the reference speech based on the auxiliary emotion identifier to obtain the prosodic embedding of the reference speech includes:

[0026] The prosodic coding sub-model extracts the prosodic features of the reference speech to obtain the prosodic feature vector of the reference speech, wherein the prosodic features include speech rate features and pitch features;

[0027] The prosodic encoding sub-model generates the prosodic embedding based on the prosodic feature vector.

[0028] In one embodiment, the step of using the pitch feature vector, the duration feature vector, and the text encoding vector to perform speech synthesis to obtain the speech content corresponding to the target text information includes:

[0029] The text encoding vector is fused with duration information using the duration feature vector to obtain a text duration vector;

[0030] The pitch feature vector and the text duration vector are fused and used as the input to the decoder to obtain the predicted Mel spectrum;

[0031] Speech synthesis is performed based on the predicted Mel spectrum to generate the speech content.

[0032] In one embodiment, the prosody encoding sub-model is a Transformer model that includes an attention layer;

[0033] The step of using the prosodic coding sub-model to perform prosodic coding on the reference speech based on the auxiliary emotion identifier to obtain the prosodic embedding of the reference speech includes:

[0034] The prosodic feature vector of the reference speech with the auxiliary emotion label is extracted using the attention layer;

[0035] The prosodic embedding is generated based on the prosodic feature vector.

[0036] To achieve the above objectives, a second aspect of this application provides a speech synthesis apparatus, comprising:

[0037] The acquisition unit is used to input the acquired target text information into the speech synthesis model, which includes: a cross-domain emotion recognition sub-model, a prosody coding sub-model, a pitch prediction sub-model, and a duration prediction sub-model.

[0038] An auxiliary sentiment label generation unit is used to perform sentiment recognition on the target text information using the cross-domain sentiment recognition sub-model to obtain an auxiliary sentiment label for the target text information.

[0039] The prosodic embedding generation unit is used to perform prosodic encoding on the reference speech based on the auxiliary emotion identifier using the prosodic encoding sub-model to obtain the prosodic embedding of the reference speech;

[0040] The pitch prediction unit is used to predict the pitch of the prosody embedding and the text encoding vector of the target text information using the pitch prediction sub-model, so as to obtain a pitch feature vector.

[0041] The duration prediction unit is used to predict the duration of the prosodic embedding and the text encoding vector of the target text information using the duration prediction sub-model, so as to obtain a duration feature vector.

[0042] The speech synthesis prediction unit is used to perform speech synthesis using the pitch feature vector, the duration feature vector and the text encoding vector to obtain the speech content corresponding to the target text information.

[0043] The acquisition unit is used to input the acquired target text information into the speech synthesis model, which includes: a cross-domain emotion recognition sub-model, a prosody coding sub-model, a pitch prediction sub-model, and a duration prediction sub-model.

[0044] An auxiliary sentiment label generation unit is used to perform sentiment recognition on the target text information using the cross-domain sentiment recognition sub-model to obtain an auxiliary sentiment label for the target text information.

[0045] The prosodic embedding generation unit is used to perform prosodic encoding on the reference speech based on the auxiliary emotion identifier using the prosodic encoding sub-model to obtain the prosodic embedding of the reference speech;

[0046] The pitch prediction unit is used to predict the pitch of the prosody embedding and the text encoding vector of the target text information using the pitch prediction sub-model, so as to obtain a pitch feature vector.

[0047] The duration prediction unit is used to predict the duration of the prosodic embedding and the text encoding vector of the target text information using the duration prediction sub-model, so as to obtain a duration feature vector.

[0048] The speech synthesis prediction unit is used to perform speech synthesis using the pitch feature vector, the duration feature vector and the text encoding vector to obtain the speech content corresponding to the target text information.

[0049] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0050] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0051] The speech synthesis method, speech synthesis device, electronic device, and storage medium proposed in this application embodiment are as follows: The speech synthesis method inputs the acquired target text information into a speech synthesis model, uses a cross-domain emotion recognition sub-model to perform emotion recognition on the target text information to obtain an auxiliary emotion identifier for the target text information; uses a prosodic coding sub-model to perform prosodic coding on the reference speech based on the auxiliary emotion identifier to obtain a prosodic embedding of the reference speech; uses a pitch prediction sub-model to predict the pitch of the prosodic embedding and the text encoding vector of the target text information to obtain a pitch feature vector; uses a duration prediction sub-model to predict the duration of the prosodic embedding and the text encoding vector of the target text information to obtain a duration feature vector; and uses the pitch feature vector, duration feature vector, and text encoding vector to perform speech synthesis to obtain the speech content corresponding to the target text information. This embodiment uses a cross-domain emotion recognition sub-model to generate an auxiliary emotion identifier for the target text information, and combines the correlation between the auxiliary emotion identifier and prosodic recognition during the speech synthesis process to select the reference speech to generate synthesized speech containing emotion, thereby improving the naturalness of the synthesized speech and expanding the application scope of text-to-speech synthesis technology. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the speech synthesis model structure provided in an embodiment of the present invention.

[0053] Figure 2 This is a flowchart of the speech synthesis method provided in the embodiments of the present invention.

[0054] Figure 3 This is a cross-domain emotion recognition sub-model of the speech synthesis model provided in another embodiment of the present invention.

[0055] Figure 4This is a flowchart illustrating the training process of a cross-domain emotion recognition sub-model of a speech synthesis model provided in another embodiment of the present invention.

[0056] Figure 5 yes Figure 4 The flowchart for step S440.

[0057] Figure 6 yes Figure 2 The flowchart of step S130.

[0058] Figure 7 This is a speech synthesis flowchart of a speech synthesis model provided in another embodiment of the present invention.

[0059] Figure 8 This is a flowchart of a speech synthesis method provided in another embodiment of the present invention.

[0060] Figure 9 This is a structural block diagram of a speech synthesis device provided in another embodiment of the present invention.

[0061] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0063] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0065] First, let's clarify some of the terms used in this invention:

[0066] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0067] Attention units: Research on attention units first appeared in psychology, where it was observed that when a person sees a scene, although they are faced with the entire scene, their attention is focused on only a specific point. In 2014, Bahdanau et al. first introduced attention units into machine translation, combining them with neural networks to improve the accuracy of machine translation. Subsequent studies have attempted to combine attention units with neural networks for different tasks, all achieving significant advantages over traditional methods to varying degrees.

[0068] Convolutional Neural Networks (CNNs) are a class of feedforward neural networks that incorporate convolutional computations and have a deep structure. They are one of the representative algorithms in deep learning. CNNs possess representation learning capabilities, enabling translation-invariant classification of input information according to their hierarchical structure. CNNs are constructed by mimicking biological visual perception mechanisms and can perform both supervised and unsupervised learning. The shared parameters of convolutional kernels within their hidden layers and the sparsity of inter-layer connections allow CNNs to process gridded features with relatively low computational cost. A common CNN structure is: Input Layer - Convolutional Layer - Pooling Layer - Fully Connected Layer - Output Layer.

[0069] Mel spectrogram: This is the spectrum obtained by performing a Fourier transform on an acoustic signal, followed by a Mel-scale transformation. Spectrograms are often very large images; to obtain appropriately sized sound features, they can be transformed into Mel spectrograms after passing through a Mel-scale filter bank. In the Mel frequency domain, there is a linear relationship between the Mel frequencies of speech and human pitch perception. The Mel spectrum is obtained by combining the Mel frequency cepstrum and the spectrogram.

[0070] Embedding is a commonly used feature method in the field of deep learning. It maps high-dimensional raw data (such as images, sentences, speech, etc.) to low-dimensional manifolds, making the high-dimensional raw data separable after being mapped to the low-dimensional manifold. This mapping process is called embedding.

[0071] Long Short-Term Memory (LSTM) artificial neural networks are a type of Recurrent Neural Network (RNN). LSTMs are well-suited for modeling time-series data, such as text data. The computation process of LSTM can be summarized as follows: by forgetting information in the cell state and remembering new information, useful information for subsequent time steps is passed on, while useless information is discarded. At each time step, a hidden state is output, calculated from the previous hidden state and the current input. The hidden state is controlled by forget gates, remember gates, and output gates.

[0072] In recent years, speech synthesis technology, which converts text into speech, has been increasingly applied to speech signal processing systems such as voice interaction, voice broadcasting, and personalized voice production. Effective use of speech synthesis technology can improve the user experience of voice interaction and has broad potential applications. Speech synthesis systems are widely used in various scenarios in daily life, including voice dialogue systems and intelligent voice assistants. Applications of intelligent voice assistants include Siri, iFlytek Voice Assistant, telephone information query systems, in-vehicle navigation, and audiobooks. Applications of voice dialogue systems include language learning systems, real-time information broadcasting systems in airports and train stations, and information acquisition and communication systems for visually or verbally impaired individuals.

[0073] The applicant discovered that in related technologies, for the Text-to-Speech (TTS) process, to improve the naturalness of the synthesized speech, it is necessary to incorporate emotional information into the speech synthesis model for speech content prediction. However, these technologies do not utilize the correlation between emotional information and prosody in speech content synthesis, resulting in low naturalness. For example, many audiobook platforms currently offer audiobook readings with synthesized speech that sounds mechanical and lacks emotion, failing to achieve the nuanced intonation of a human voice, leading to a poor user experience. Therefore, how to improve the utilization of the correlation between emotional information and prosody in speech synthesis to enhance its naturalness has become an urgent technical problem to be solved.

[0074] Based on this, embodiments of the present invention provide a speech synthesis method, a speech synthesis device, an electronic device, and a storage medium. The speech synthesis method utilizes a cross-domain emotion recognition sub-model to generate auxiliary emotion tags for target text information. During the speech synthesis process, the correlation between the auxiliary emotion tags and prosody is combined to select reference speech to generate synthesized speech containing emotion, thereby improving the naturalness of the synthesized speech and expanding the application scope of text-to-speech synthesis technology.

[0075] The present invention provides a speech synthesis method, a speech synthesis device, an electronic device, and a storage medium, which are specifically described through the following embodiments. First, the speech synthesis method in the embodiments of the present invention is described.

[0076] This invention can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0077] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0078] The speech synthesis method provided in this invention relates to the field of artificial intelligence technology, and particularly to the field of data mining technology. The speech synthesis method provided in this invention can be applied to a terminal, a server, or a computer program running on either a terminal or a server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client supporting the speech synthesis method; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded into any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. The speech synthesis method can be executed by the terminal or the server, or by the terminal and the server working together.

[0079] In some embodiments, the terminal may be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. Additionally, the terminal may also be a smart in-vehicle device. This smart in-vehicle device applies the speech synthesis method or speech synthesis method of this embodiment to provide related services, enhancing the driving experience. The server may be an independent server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it may also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The server may have a speech synthesis method or a server-side implementation of the speech synthesis method installed on it. This server-side implementation can interact with the terminal; for example, corresponding software may be installed on the server. This software may be an application implementing the speech synthesis method or speech synthesis method, but is not limited to the above forms. The terminal and the server can be connected via Bluetooth, USB (Universal Serial Bus), or network communication methods, and this embodiment does not impose any restrictions.

[0080] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0081] To facilitate understanding of the embodiments of this application, the concept of text-to-speech synthesis in this application will be briefly introduced below with examples of specific application scenarios.

[0082] Text-to-speech synthesis is an important component of human-computer voice communication. By using text-to-speech synthesis technology, machines can speak like humans, converting information represented or stored in text form into speech, so that people can conveniently obtain this information through hearing.

[0083] Emotion recognition: Emotion is a phenomenon that integrates human behavior, thoughts, and feelings. Emotion recognition refers to extracting the corresponding emotional information expressed in text content. To obtain the emotional information of text content, features need to be extracted from the text data and classified to obtain the emotional information.

[0084] In one application scenario: User A sends a text message and can select a voice object identifier (mainly used to identify different timbres, such as a person's voice or other mechanical voices) as needed. User B receives the corresponding voice message and plays it; what they actually hear is the selected timbre voice, which also contains the emotional information from the text message. This process utilizes text-to-speech synthesis to convert User A's text message into corresponding voice information.

[0085] Input: Text message sent by user A;

[0086] Output: Target speech.

[0087] In another application scenario: User A needs to listen to the audio of the novel they are reading, i.e., to achieve audio reading. User A selects audio reading, the speech synthesis system obtains the corresponding novel text information, judges the emotional information based on the novel text content, and incorporates the corresponding emotional information during speech synthesis to generate a reading voice with intonation.

[0088] Input: The text content selected by user A;

[0089] Output: Target speech containing emotional information.

[0090] The speech synthesis method in the embodiments of the present invention will be described first below.

[0091] In one embodiment, reference is made to Figure 1 The speech synthesis model 100 includes: a cross-domain emotion recognition sub-model 200, a prosody coding sub-model 300, a pitch prediction sub-model 400, and a duration prediction sub-model 500. Figure 2 This is an optional flowchart of the speech synthesis method provided in the embodiments of the present invention, utilizing... Figure 1 The speech synthesis model 100 shown performs speech synthesis. Figure 2 The method may include, but is not limited to, steps S110 to S160. It is also understood that this embodiment... Figure 2 The order of steps S110 to S160 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0092] Step S110: Input the obtained target text information into the speech synthesis model.

[0093] In one embodiment, the speech synthesis model includes: a cross-domain emotion recognition sub-model, a prosody coding sub-model, a pitch prediction sub-model, and a duration prediction sub-model.

[0094] In one embodiment, the target text information can be obtained from user input, such as chat messages, or it can be obtained by the user selecting it on the page, such as selecting the current page to be read aloud, in which case the text content of the current page is the target text information. If speech synthesis of a paragraph is required, it can be achieved by synthesizing the speech of each text in the paragraph and then summarizing them. This embodiment does not impose specific limitations on this.

[0095] In one embodiment, a target speech identifier, i.e. the timbre of the synthesized speech, can also be selected simultaneously. Reference speech corresponding to different timbres is added to the above-mentioned reference speech, so that the prosody embedding obtained by the prosody prediction module includes the target speech identifier information.

[0096] Step S120: Use the cross-domain emotion recognition sub-model to perform emotion recognition on the target text information to obtain the auxiliary emotion label of the target text information.

[0097] In one embodiment, a cross-domain sentiment recognition sub-model is first trained using deep learning pre-training. Then, the trained sub-model is used to predict auxiliary sentiment labels based on the target text information. In one embodiment, a convolutional recurrent neural network structure based on an attention mechanism, such as the FCN-8S structure, is used to construct the cross-domain sentiment recognition sub-model.

[0098] It is understandable that the text content of the target text information contains the emotional information of the recipient. For example, when discussing a certain matter, one might express emotions related to happiness (happiness, indifference, sadness); when receiving an apology, one might express emotions related to forgiveness (forgiveness, noncommittal, unforgiveness); and when the emotional buildup in a novel reaches a climax, one might express emotions related to excitement. All of these belong to emotional information. In one embodiment, the cross-domain emotion recognition sub-model of this disclosure performs emotion classification on the input target text information to obtain auxiliary emotion labels. This involves dividing the emotional information in the target text information into different emotion classification results according to a preset classification standard. The preset classification standard can be happiness, sadness, grief, or anger, etc. This embodiment does not specifically limit the emotion classification standard; different classification standards can be set according to the actual use scenario.

[0099] In one embodiment, reference is made to Figure 3 The cross-domain emotion recognition sub-model 200 includes: a first encoder 210, a second encoder 220, and a classifier 230. (See reference...) Figure 4 The training process of the cross-domain emotion recognition sub-model includes steps S410 to S460.

[0100] Step S410: Obtain the speech emotion recognition dataset.

[0101] In one embodiment, reference is made to Figure 3 The speech emotion recognition dataset includes: a first dataset 241 and a second dataset 242. The first dataset includes a first speech sample and its corresponding speech emotion label, while the second dataset includes a second text sample. Since there are many speech emotion sample training datasets available in related technologies, this embodiment can utilize an existing speech emotion sample training dataset as the first dataset. For example, the Belfast emotion database includes five emotions: anger, sadness, happiness, fear, and neutrality. The second text sample can be any text sample. This embodiment does not specifically limit the acquisition method of the first and second datasets.

[0102] Step S420: Generate first data features based on the first speech sample using the first encoder.

[0103] In one embodiment, reference is made to Figure 3 The first encoder 210 extracts the first data feature of the first speech sample in the first dataset according to the model training requirements. The first speech sample is speech format data, and the first data feature can be one or more of the following: prosodic features, spectral features, sound quality features, or features based on the Teager energy operator (TEO).

[0104] Step S430: Generate second data features based on the second text sample using the second encoder.

[0105] In one embodiment, reference is made to Figure 3 The second encoder 220 extracts the second data features of the second text sample in the second dataset according to the model training requirements. The second text sample is text format data, and the second data features can be statistical features, such as maximum and minimum values, mean, standard deviation and other statistical information.

[0106] Step S440: Adjust the parameters of the first encoder and the second encoder according to the distribution loss values ​​of the first data feature and the second data feature until the distribution loss value reaches the first preset convergence condition.

[0107] In one embodiment, since the first dataset is a speech dataset and the second dataset is a text dataset, the data distributions of these two datasets are different. The cross-domain emotion recognition sub-model of this application needs to associate the tasks between these two datasets, transferring knowledge learned from the first dataset to the second dataset. This allows the generation of corresponding emotion labels for the text dataset through the emotion recognition process of the speech dataset. Therefore, this embodiment needs to utilize the distribution loss values ​​of the first and second data features to adjust the distributions of the first and second data features to make them more similar, thereby improving the accuracy of the emotion labels in the second dataset.

[0108] In one embodiment, reference is made to Figure 5 This is a flowchart illustrating a specific implementation of step S440 in one embodiment. In this embodiment, step S440, which involves adjusting the parameters of the first encoder and the second encoder based on the distribution loss values ​​of the first data feature and the second data feature until the distribution loss value reaches the first preset convergence condition, includes:

[0109] Step S441: Obtain the first expected value of the first data feature after it has been mapped by a preset mapping function.

[0110] Step S442: Obtain the second expected value of the second data feature after it has been mapped by a preset mapping function.

[0111] Step S443: Obtain the distribution loss value based on the first expected value and the second expected value.

[0112] Step S444: Adjust the parameters of the first encoder and the second encoder until the distributed loss value reaches the first preset convergence condition, which is the minimum distributed loss value.

[0113] In one embodiment, since the speech sample set and the text sample set have different features—that is, the two datasets belong to two different domains—in order for the cross-domain emotion recognition sub-model to transfer the ability to recognize emotions from speech samples to emotion recognition from text samples, and to label unlabeled text samples with emotion tags, the features encoded in the two domains need to be made to have the same distribution. It is understood that in this embodiment, the similarity in distribution does not have to be strictly identical; as long as the distributions are as close as possible, it is acceptable. In this embodiment, when training the cross-domain emotion recognition sub-model, in order for the cross-domain emotion recognition sub-model trained on speech samples to be able to perform emotion recognition across domains, distribution detection is performed on the first data features generated by the first encoder based on the first speech sample and the second data features generated by the second encoder based on the second text sample, so that the first data features and the second data features tend to have the same distribution.

[0114] In one embodiment, the maximum mean discrepancy (MMD) is used to calculate the similarity between the first and second data features. The MMD algorithm is used to measure whether two samples come from the same distribution.

[0115] In one embodiment, assuming the preset mapping function f belongs to the function domain F, step S441 maps the first data feature through the preset mapping function f to obtain a first expected value f1. Step S442 maps the second data feature through the preset mapping function f to obtain a second expected value f2. Then, step S443 obtains the distribution loss value based on the first expected value f1 and the second expected value f2. Here, the distribution loss value is the MMD distance between the two. In one embodiment, the MMD distance is the maximum value of the difference between the expected values ​​of the first data feature (containing distribution information) and the second data feature (containing distribution information) after being mapped through the preset mapping function f in the defined function domain F. The first expected value and the second expected value are the mean values ​​of the two mapped distributions.

[0116] In one embodiment, in the MMD distance, the function domain F is defined as an arbitrary vector within a unit sphere in the regenerated Hilbert space, i.e., the preset mapping function f satisfies the condition: ||f||<1. Then, the MMD distance is the difference between the mean distances of the first data feature and the second data feature in the Hilbert space.

[0117] In one embodiment, theoretically, a zero MMD distance represents that the two distributions are identically distributed. To improve the training efficiency of the cross-domain emotion recognition sub-model, in this embodiment, the parameters of the first encoder and the second encoder are adjusted until the distribution loss value reaches a first preset convergence condition. The first preset convergence condition is that the MMD distance is minimized. Here, "minimum" can be a small threshold. If the distribution loss value is less than this threshold, the first convergence condition is considered to have been met, the distribution loss value is minimized, and the first data features generated by the first encoder based on the first speech sample and the second data features generated by the second encoder based on the second speech sample tend to be identically distributed. This embodiment does not specifically limit the first preset convergence condition.

[0118] It is understood that in this embodiment, the MMD algorithm optimizes the parameters of the first encoder and the second encoder, rather than the selection of the first dataset and the second dataset. The purpose is that after optimization, the encoding features of the first encoder for speech information and the encoding features of the second encoder for text information are identically distributed.

[0119] Step S450: Input the first data features into the classifier to obtain the predicted sentiment.

[0120] In one embodiment, reference is made to Figure 3 After adjusting the parameters of the first encoder and the second encoder, the classifier 230 performs sentiment recognition on the first data features to obtain the predicted sentiment. It is understandable that, since the adjusted outputs of the first encoder and the second encoder are identically distributed, the classifier 230 can obtain an approximate predicted sentiment when performing sentiment recognition on the second data features.

[0121] Step S460: Obtain the recognition loss value based on the predicted emotion and voice emotion label, adjust the parameters of the classifier until the recognition loss value reaches the second preset convergence condition, and obtain the trained cross-domain emotion recognition sub-model.

[0122] In one embodiment, reference is made to Figure 3 After obtaining the predicted sentiment, the predicted sentiment and the sentiment label corresponding to the first speech sample are compared to calculate the recognition loss value, which can be the cross-entropy loss value. The parameters of the classifier 230 are adjusted based on the cross-entropy loss value. If the second preset convergence condition is met, the cross-domain sentiment recognition sub-model training is complete. It can be understood that the second preset convergence condition can be that the cross-entropy loss value is less than a threshold or that a preset number of convergence attempts has been reached. This embodiment does not specifically limit the second preset convergence condition.

[0123] As shown above, after training the cross-domain emotion recognition sub-model, the cross-domain emotion recognition sub-model trained on speech samples can be applied to emotion recognition of text samples. The pre-trained cross-domain emotion recognition sub-model can perform emotion recognition on any input target text information, obtaining auxiliary emotion labels for the target text information.

[0124] In one embodiment, the training samples used to train the speech synthesis model do not contain labels; instead, they contain text information. That is, manual annotation of the training samples is not required in this embodiment. The text information of the training samples can be collected using an input device (such as a touch panel or keyboard), retrieved from local storage or other devices, or downloaded from the internet. This embodiment does not specifically limit the method of obtaining the training samples. Utilizing a cross-domain emotion recognition sub-model to generate auxiliary emotion labels for the unlabeled training samples overcomes the problem of insufficient training sample quantity due to the low efficiency of manual annotation, thereby improving the training efficiency of the speech synthesis model during training.

[0125] Step S130: Use the prosodic coding sub-model to perform prosodic coding on the reference speech based on the auxiliary emotion label to obtain the prosodic embedding of the reference speech.

[0126] In one embodiment, before step S130, the method further includes: selecting a reference speech from a reference speech library based on an auxiliary emotion identifier, wherein the reference speech library includes reference speech of multiple speech emotion categories.

[0127] In one embodiment, after obtaining the auxiliary emotion marker, a corresponding reference speech is selected. For example, if the auxiliary emotion marker is anger, then a reference speech related to anger is selected.

[0128] In one embodiment, reference speech corresponding to different emotion tags is obtained in advance to construct a reference speech library. Different reference speech contains feature information of the corresponding emotion. To improve the practicality and accuracy of the reference speech, in one embodiment, a large number of original reference speech under the same emotion are first obtained, such as 100 original reference speech related to anger. Then, the large number of original reference speech are vector averaged to obtain a reference speech representing the average feature, thereby improving the accuracy of the reference speech.

[0129] In one embodiment, the prosodic encoding sub-model is a Transformer model that includes an attention layer. In one embodiment, the prosodic encoding sub-model consists of 3 Transformer layers, each containing a self-attention layer, wherein the kernel size is 5 and the filter size is 384.

[0130] In one embodiment, reference is made to Figure 6This is a flowchart illustrating a specific implementation of step S130 in one embodiment. In this embodiment, step S130, which generates the prosodic embedding of the reference speech corresponding to the auxiliary emotion identifier through the prosodic coding sub-model, includes:

[0131] Step S131: The prosodic coding sub-model extracts the prosodic feature vector of the reference speech.

[0132] In one embodiment, the prosodic coding sub-model includes a reference encoder, which contains an attention layer. The attention layer is used to extract prosodic features of the reference speech to obtain a prosodic feature vector of the reference speech. The prosodic features include speech rate features and pitch features.

[0133] Step S132: The prosodic encoding sub-model generates a prosodic embedding based on the prosodic feature vector.

[0134] In one embodiment, the prosodic coding sub-model includes an embedding encoder for generating a prosodic embedding based on a prosodic feature vector.

[0135] In one embodiment, the prosodic coding sub-model is first trained by deep learning pre-training. Then, the trained prosodic coding sub-model is used to extract the prosodic features of the reference speech based on the reference speech to obtain the prosodic feature vector of the reference speech. Finally, a prosodic embedding is generated based on the prosodic feature vector.

[0136] In one embodiment, prosodic features are obtained from reference speech by the reference encoder of the prosodic prediction model. For a reference speech segment, the information contained within it can be divided into two parts: the first part is the syllable pronunciation information corresponding to the syllables in the reference speech; the second part consists of other pronunciation features excluding syllable information, mainly including prosodic features, which include speech rate features and pitch features. It is understood that the speech rate and pitch differ under different emotional states; for example, compared to a calm state, an angry state has a faster speech rate and a higher pitch. Since prosodic features cannot be directly extracted from the reference speech, the prosodic prediction model in this embodiment is able to learn the ability to extract prosodic features from speech features during training.

[0137] Understandably, during training, the training samples for the prosodic coding sub-model are speech information and corresponding sentiment tags. Since the prosodic coding sub-model may not initially obtain prosodic embeddings of speech information that are completely related to the sentiment tags, it is necessary to train the prosodic coding sub-model using sentiment tags. This embodiment does not specifically limit the training process of the prosodic coding sub-model. In this embodiment, a pre-trained prosodic coding sub-model is used. First, a corresponding reference speech is selected based on the auxiliary sentiment tag, and then the prosodic coding sub-model is used to obtain the prosodic embedding of the reference speech.

[0138] Embedding is a feature processing method that maps high-dimensional raw data to a low-dimensional manifold, making the high-dimensional raw data separable after being mapped to the low-dimensional manifold. In one embodiment, step S133 uses a prosodic coding sub-model to perform an embedding operation on the prosodic feature vector to obtain the corresponding low-dimensional feature vector: prosodic embedding.

[0139] In one embodiment, the prosodic embedding and the text encoding vector of the target text information are first concatenated to obtain the speech synthesis vector.

[0140] In one embodiment, if the target text information is Chinese characters, Chinese characters are usually composed of initials and finals. According to the principles of linguistics, they can be further subdivided. The same final and different combinations of initials can represent different phonemes. Therefore, the initials and finals are defined as phoneme sets by using the principles of linguistics and a pre-built phoneme dictionary.

[0141] In one embodiment, initials and finals are defined as 66 types of phonemes forming a preset phoneme dictionary, which is represented as follows:

[0142] "a", "aa", "ai", "an", "ang", "ao", "b", "c", "ch", "d", "e", "ee", "ei", "en", "eng", "er", "f" ,"g","h","i","ia","ian","iang","iao","ie","ii","in","ing","iong","iu","ix","iy"," iz", "j", "k", "l", "m", "n", "o", "ong", "oo", "ou", "p", "q", "r", "s", "sh", "t", "u", "ua", "ua i", "uan", "uang", "ueng", "ui", "un", "uo", "uu", "v", "van", "ve", "vn", "vv", "x", "z", "zh".

[0143] As can be seen from the above, the speech synthesis model also includes a text encoder. The text encoder encodes the target text information to obtain a text encoding vector. This can be done by transcribing the text content of the input target text information into pinyin information, then selecting a preset phoneme dictionary, splitting the pinyin into individual phonemes according to the preset phoneme dictionary, and concatenating the individual phonemes into a phoneme sequence. The phoneme sequence is the text encoding vector.

[0144] As can be seen from the above, the target text information is first encoded to obtain a text encoding vector, so that it can be connected with the prosodic embedding obtained in the above steps. The connected vector is used as the speech synthesis vector for subsequent speech synthesis. That is, the speech synthesis vector contains text information and corresponding prosodic information of emotion.

[0145] Step S140: Use the pitch prediction sub-model to predict the pitch of the text encoding vector of the prosody embedding and the target text information to obtain the pitch feature vector.

[0146] In one embodiment, the pitch prediction sub-model is first trained by deep learning pre-training, and then the trained pitch prediction sub-model is used to obtain the pitch feature vector based on the speech synthesis vector (the prosodic embedding and the text encoding vector of the target text information mentioned above).

[0147] In one embodiment, the pitch prediction sub-model can be a convolutional neural network or a recurrent neural network. The input is a speech synthesis vector, and the output is the pitch features of each phoneme corresponding to the speech synthesis vector. The pitch feature of each phoneme is predicted by the pitch prediction sub-model in combination with the pitch features embedded in the prosody in the speech synthesis vector.

[0148] Step S150: Perform duration prediction on the prosodic embedding and the text encoding vector of the target text information through the duration prediction sub-model to obtain the duration feature vector.

[0149] In one embodiment, the duration prediction sub-model is first trained using deep learning pre-training. Then, the trained duration prediction sub-model is used to obtain the duration feature vector based on the speech synthesis vector (the prosodic embedding and the text encoding vector of the target text information mentioned above). The duration prediction sub-model can be trained using training samples to obtain optimal model parameters suitable for the scenario. This embodiment does not specifically limit the training process.

[0150] It is understandable that the duration of a speaker’s pronunciation of different words varies depending on their emotions. Therefore, by combining the speech rate features in the prosodic embedding, the duration prediction sub-model predicts the duration of each phoneme corresponding to the speech synthesis vector, thus forming a duration feature vector.

[0151] For example, in one embodiment:

[0152] The text encoding vector is: {j,in,t,i,an,t,i,an,q,i,h,en,h,ao};

[0153] The duration feature vector of the text encoding vector is represented as:

[0154] {j-0.06,in-0.09,t-0.03,i-0.03,an-0.06,t-0.03,i-0.03,an-0.06,q-0.03,i-0.09,h-0.03,en-0.06,h-0.03,ao-0.06};

[0155] Based on the prosodic embedding under the auxiliary emotion label, the above predicts the different pitches and durations corresponding to the target text information, with the duration used to represent the speech rate.

[0156] Step S160: Use pitch feature vector, duration feature vector and text encoding vector to perform speech synthesis to obtain the speech content corresponding to the target text information.

[0157] In one embodiment, speech synthesis is performed using pitch feature vectors, duration feature vectors, and text encoding vectors to obtain the speech content corresponding to the target text information.

[0158] In one embodiment, reference is made to Figure 7 The above-mentioned step S160, which uses pitch feature vectors, duration feature vectors, and text encoding vectors to perform speech synthesis and obtain the speech content corresponding to the target text information, includes:

[0159] Step S171: Use the duration feature vector to fuse the duration information of the text encoding vector to obtain the text duration vector.

[0160] In one embodiment, using duration feature vectors to fuse duration information in text encoding vectors refers to performing speech frame-level alignment of each phoneme in the text encoding vector according to the duration of each phoneme, to obtain a text duration vector.

[0161] For example, in one embodiment:

[0162] Phoneme sequence: {j,in,t,i,an,t,i,an,q,i,h,en,h,ao};

[0163] The duration feature vector of the text encoding vector is represented as:

[0164] {j-0.06,in-0.09,t-0.03,i-0.03,an-0.06,t-0.03,i-0.03,an-0.06,q-0.03,i-0.09,h-0.03,en-0.06,h-0.03,ao-0.06};

[0165] The corresponding speech frame duration is 0.03s. The duration of the text encoding vector is converted into the number of speech frames, expressed as:

[0166] {j-2 frames,in-3 frames,t-1 frames,i-1 frames,an-2 frames,t-1 frames,i-1 frames,an-2 frames,q-1 frames,i-3 frames,h-1 frames,en-2 frames,h-1 frames,ao-2 frames}.

[0167] Step S172: The pitch feature vector and the text duration vector are fused and used as the input to the decoder to obtain the predicted Mel spectrum.

[0168] In one embodiment, fusing the pitch feature vector and the text duration vector means setting a corresponding pitch for each frame of the text duration vector, and then using the fused result as the input to the decoder to obtain the predicted Mel spectrum.

[0169] Step S173: Perform speech synthesis based on the predicted Mel spectrum to generate speech content.

[0170] In one embodiment, a vocoder is used to convert the predicted Mel spectrum into corresponding speech content, which is a wave file that represents the speech signal in the form of a wave. This embodiment does not limit the form of the speech content.

[0171] In one embodiment, the process of adjusting the parameters of the speech synthesis model according to the speech content to obtain the trained speech synthesis model is described as follows: During the training of the speech synthesis model, the training samples not only include the target text information but also the training speech corresponding to the target text information. The speech content obtained in the above steps is compared with the training speech to obtain the speech synthesis loss value. Then, the parameters of the speech synthesis model are adjusted based on the speech synthesis loss value to train the speech synthesis model and obtain the trained speech synthesis model.

[0172] As can be seen from the above, referring to Figure 8 The speech synthesis method generates auxiliary emotion labels for the target text information through the cross-domain emotion recognition sub-model 200, and then generates the prosodic feature vector of the reference speech corresponding to the auxiliary emotion label through the reference encoder 310 of the prosodic coding sub-model 300. The embedding encoder 320 of the prosodic coding sub-model 300 generates the prosodic embedding of the prosodic feature vector. Then, the prosodic embedding and the text encoding vector of the target text information obtained by the text encoder 600 are concatenated to obtain the speech synthesis vector. The pitch prediction sub-model 400 and the duration prediction sub-model 500 are used to predict the pitch and duration of the speech synthesis vector, respectively, to obtain the pitch feature vector and the duration feature vector. Finally, the decoder 700 uses the pitch feature vector, the duration feature vector and the text encoding vector to perform speech synthesis to obtain the speech content corresponding to the target text information.

[0173] In one embodiment, in actual use, only text information needs to be input to use the trained speech synthesis model to synthesize speech and obtain synthesized speech. The obtained speech data can also be used for further guidance operations, etc. This embodiment does not make specific limitations in this regard.

[0174] The speech synthesis method of this application embodiment can be used in the medical field. The target text information is medical data collected with the patient's informed consent, such as personal health records, prescriptions, examination reports, etc. The application scenarios of the speech synthesis method of this application embodiment in the field of digital healthcare are described below.

[0175] In hospitals and other medical institutions, speech synthesis technology can enable robots or other intelligent terminals to converse with patients like humans, providing necessary guidance, advice, and services. The speech synthesis method described in this application achieves high naturalness in the synthesized speech of robots or other intelligent terminals. This not only alleviates the heavy workload of medical staff but also better serves patients and improves their user experience during communication. Alternatively, medical staff can use speech synthesis technology to convert medical records into speech, which can then be saved and backed up in a data center. This method not only speeds up the recording process but, more importantly, makes the documents more realistic and vivid, thus providing medical staff with more accurate information.

[0176] As described above, the speech synthesis method inputs the acquired target text information into the speech synthesis model. A cross-domain emotion recognition sub-model is used to perform emotion recognition on the target text information, obtaining an auxiliary emotion marker. A prosodic coding sub-model is used to perform prosodic encoding on the reference speech based on the auxiliary emotion marker, obtaining a prosodic embedding of the reference speech. A pitch prediction sub-model is used to predict the pitch of the prosodic embedding and the text encoding vector of the target text information, obtaining a pitch feature vector. A duration prediction sub-model is used to predict the duration of the prosodic embedding and the text encoding vector of the target text information, obtaining a duration feature vector. Finally, speech synthesis is performed using the pitch feature vector, duration feature vector, and text encoding vector to obtain the speech content corresponding to the target text information. This embodiment utilizes a cross-domain emotion recognition sub-model to generate an auxiliary emotion marker for the target text information. During the speech synthesis process, the correlation between the auxiliary emotion marker and prosodic recognition is combined to select reference speech to generate synthesized speech containing emotion, improving the naturalness of the synthesized speech and expanding the application scope of text-to-speech synthesis technology.

[0177] This invention also provides a speech synthesis apparatus that can implement the above-described speech synthesis method. The speech synthesis model includes: a cross-domain emotion recognition sub-model, a prosody coding sub-model, a pitch prediction sub-model, and a duration prediction sub-model, as described above. Figure 9 The device includes:

[0178] The acquisition unit 910 is used to input the acquired target text information into the speech synthesis model, which includes: a cross-domain emotion recognition sub-model, a prosody coding sub-model, a pitch prediction sub-model, and a duration prediction sub-model.

[0179] The auxiliary sentiment label generation unit 920 is used to perform sentiment recognition on the target text information using a cross-domain sentiment recognition sub-model to obtain the auxiliary sentiment label of the target text information.

[0180] The prosodic embedding generation unit 930 is used to perform prosodic encoding on the reference speech based on the auxiliary emotion label using the prosodic coding sub-model to obtain the prosodic embedding of the reference speech;

[0181] The pitch prediction unit 940 is used to predict the pitch of the text encoding vector of the prosody embedding and the target text information using the pitch prediction sub-model, so as to obtain the pitch feature vector.

[0182] The duration prediction unit 950 is used to predict the duration of the text encoding vector of the prosody embedding and the target text information using the duration prediction sub-model, so as to obtain the duration feature vector.

[0183] The speech synthesis prediction unit 960 is used to perform speech synthesis using pitch feature vectors, duration feature vectors and text encoding vectors to obtain the speech content corresponding to the target text information.

[0184] The specific implementation of the speech synthesis device in this embodiment is basically the same as the specific implementation of the speech synthesis method described above, and will not be repeated here.

[0185] This invention also provides an electronic device, comprising:

[0186] At least one memory;

[0187] At least one processor;

[0188] At least one program;

[0189] The program is stored in a memory, and the processor executes the at least one program to implement the speech synthesis method described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0190] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0191] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0192] The memory 1002 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to execute the speech synthesis method of the embodiments of this invention.

[0193] Input / output interface 1003 is used to implement information input and output;

[0194] Communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0195] The communication bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004).

[0196] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via a communication bus 1005.

[0197] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described speech synthesis method.

[0198] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0199] The speech synthesis method, speech synthesis device, electronic device, and storage medium proposed in this invention embodiment are as follows: The speech synthesis method inputs the acquired target text information into a speech synthesis model, uses a cross-domain emotion recognition sub-model to perform emotion recognition on the target text information to obtain an auxiliary emotion identifier for the target text information; uses a prosodic coding sub-model to perform prosodic coding on reference speech based on the auxiliary emotion identifier to obtain a prosodic embedding of the reference speech; uses a pitch prediction sub-model to predict the pitch of the prosodic embedding and the text encoding vector of the target text information to obtain a pitch feature vector; uses a duration prediction sub-model to predict the duration of the prosodic embedding and the text encoding vector of the target text information to obtain a duration feature vector; and uses the pitch feature vector, duration feature vector, and text encoding vector to perform speech synthesis to obtain the speech content corresponding to the target text information. This embodiment uses a cross-domain emotion recognition sub-model to generate an auxiliary emotion identifier for the target text information, and combines the correlation between the auxiliary emotion identifier and prosodic recognition during the speech synthesis process to select reference speech to generate synthesized speech containing emotion, thereby improving the naturalness of the synthesized speech and expanding the application scope of text-to-speech synthesis technology.

[0200] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0201] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0202] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0203] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0204] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0205] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0206] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0207] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0208] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0209] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0210] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, include: The acquired target text information is input into the speech synthesis model, which includes: a cross-domain emotion recognition sub-model, a prosody coding sub-model, a pitch prediction sub-model, and a duration prediction sub-model. The prosody coding sub-model is a Transformer model that includes an attention layer. The cross-domain sentiment recognition sub-model is used to perform sentiment recognition on the target text information to obtain the auxiliary sentiment label of the target text information; The prosodic encoding sub-model is used to perform prosodic encoding on the reference speech based on the auxiliary emotion identifier to obtain the prosodic embedding of the reference speech; The pitch prediction sub-model is used to predict the pitch of the prosodic embedding and the text encoding vector of the target text information to obtain a pitch feature vector. The duration prediction sub-model is used to predict the duration of the prosodic embedding and the text encoding vector of the target text information to obtain a duration feature vector. Speech synthesis is performed using the pitch feature vector, the duration feature vector, and the text encoding vector to obtain the speech content corresponding to the target text information; The step of using the prosodic coding sub-model to perform prosodic coding on the reference speech based on the auxiliary emotion label to obtain the prosodic embedding of the reference speech includes: extracting the prosodic feature vector of the reference speech based on the auxiliary emotion label using the attention layer; and generating the prosodic embedding based on the prosodic feature vector.

2. The speech synthesis method according to claim 1, characterized in that, The cross-domain emotion recognition sub-model includes a first encoder, a second encoder, and a classifier; before performing emotion recognition on the target text information using the cross-domain emotion recognition sub-model to obtain the auxiliary emotion label of the target text information, the method further includes: Obtain a speech emotion recognition dataset, which includes: a first dataset and a second dataset, wherein the first dataset includes a first speech sample and a speech emotion label, and the second dataset includes a second text sample; The first encoder is used to generate a first data feature based on the first speech sample; The second encoder is used to generate second data features based on the second text sample; The parameters of the first encoder and the second encoder are adjusted according to the distribution loss values ​​of the first data feature and the second data feature until the distribution loss value reaches the first preset convergence condition. The first data features are input into the classifier to obtain the predicted sentiment; Based on the predicted emotion and the voice emotion label, the recognition loss value is obtained, and the parameters of the classifier are adjusted until the recognition loss value reaches the second preset convergence condition, thus obtaining the trained cross-domain emotion recognition sub-model.

3. The speech synthesis method according to claim 2, characterized in that, The step of adjusting the parameters of the first encoder and the second encoder based on the distribution loss values ​​of the first data feature and the second data feature until the distribution loss value reaches a first preset convergence condition includes: Obtain the first expected value of the first data feature after it has been mapped by a preset mapping function; Obtain the second expected value of the second data feature after it has been mapped by a preset mapping function; The distribution loss value is obtained based on the first expected value and the second expected value; The parameters of the first encoder and the second encoder are adjusted until the distributed loss value reaches the first preset convergence condition, which is the minimum distributed loss value.

4. The speech synthesis method according to claim 1, characterized in that, Before performing prosodic encoding on the reference speech based on the auxiliary emotion identifier using the prosodic encoding sub-model to obtain the prosodic embedding of the reference speech, the method further includes: selecting reference speech from a reference speech library based on the auxiliary emotion identifier, wherein the reference speech library includes reference speech of multiple speech emotion categories.

5. The speech synthesis method according to claim 1, characterized in that, The step of using the prosodic coding sub-model to perform prosodic coding on the reference speech based on the auxiliary emotion identifier to obtain the prosodic embedding of the reference speech includes: The prosodic coding sub-model extracts the prosodic features of the reference speech to obtain the prosodic feature vector of the reference speech, wherein the prosodic features include speech rate features and pitch features; The prosodic encoding sub-model generates the prosodic embedding based on the prosodic feature vector.

6. The speech synthesis method according to claim 1, characterized in that, The step of using the pitch feature vector, the duration feature vector, and the text encoding vector to perform speech synthesis to obtain the speech content corresponding to the target text information includes: The text encoding vector is fused with duration information using the duration feature vector to obtain a text duration vector; The pitch feature vector and the text duration vector are fused and used as the input to the decoder to obtain the predicted Mel spectrum; Speech synthesis is performed based on the predicted Mel spectrum to generate the speech content.

7. A speech synthesis device, characterized in that, include: The acquisition unit is used to input the acquired target text information into the speech synthesis model. The speech synthesis model includes: a cross-domain emotion recognition sub-model, a prosody coding sub-model, a pitch prediction sub-model, and a duration prediction sub-model. The prosody coding sub-model is a Transformer model that includes an attention layer. An auxiliary sentiment label generation unit is used to perform sentiment recognition on the target text information using the cross-domain sentiment recognition sub-model to obtain an auxiliary sentiment label for the target text information. The prosodic embedding generation unit is used to perform prosodic encoding on the reference speech based on the auxiliary emotion identifier using the prosodic encoding sub-model to obtain the prosodic embedding of the reference speech; The pitch prediction unit is used to predict the pitch of the prosody embedding and the text encoding vector of the target text information using the pitch prediction sub-model, so as to obtain a pitch feature vector. The duration prediction unit is used to predict the duration of the prosodic embedding and the text encoding vector of the target text information using the duration prediction sub-model, so as to obtain a duration feature vector. The speech synthesis prediction unit is used to perform speech synthesis using the pitch feature vector, the duration feature vector and the text encoding vector to obtain the speech content corresponding to the target text information; The step of using the prosodic coding sub-model to perform prosodic coding on the reference speech based on the auxiliary emotion label to obtain the prosodic embedding of the reference speech includes: extracting the prosodic feature vector of the reference speech based on the auxiliary emotion label using the attention layer; and generating the prosodic embedding based on the prosodic feature vector.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method according to any one of claims 1 to 6.

Images