Speech synthesis method, speech synthesis device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a speech synthesis method, a speech synthesis device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology. The method comprises the steps of performing mask processing on an original Mel spectrum of a sample based on sample emotion information to obtain a masked Mel spectrum and a masked Mel spectrum of the sample; detecting the sample emotion information and the original Mel spectrum of the sample based on an emotion detection sub-model to obtain a sample target emotion feature; performing frequency spectrum generation on the sample masked Mel frequency spectrum, the sample target emotion features and the sample text based on a frequency spectrum generation sub-model to obtain a predicted Mel frequency spectrum; and adjusting the initial synthesis model based on the mask Mel spectrum, the predicted Mel spectrum and the sample target emotion feature to obtain a speech synthesis model, so as to perform speech synthesis on the target text, the target object speech with the target speech expression feature and the target emotion information with the target non-language emotion feature. According to the embodiment of the invention, synthetic speech with more accurate emotion expression can be generated.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a speech synthesis method, a speech synthesis device, an electronic device and a storage medium. Background Art

[0002] Speech synthesis refers to the process of creating natural language speech that meets the requirements based on given input data. At present, intelligent speech technology is usually used in task scenarios such as smart telephone customer service and smart sales in financial technology. However, human speech not only contains language information, but also some non-verbal information, such as crying, laughing, pausing, coughing, etc. These non-verbal information can be used to convey the speaker's different feelings and communication intentions. Therefore, adding some non-verbal information in the process of synthesizing speech can make the synthesized speech more natural and vivid, and closer to the speech in real life.

[0003] Based on this, the relevant technology usually combines voice information and non-language information in a certain way to synthesize a speech containing non-language information. However, this method does not deeply understand the relationship between language information and non-language information, which makes the connection between voice information and non-language information in the synthesized speech unnatural, thereby reducing the emotional expression of the synthesized speech. Therefore, how to generate a more accurate synthesized speech with more accurate emotional expression has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, which can generate synthesized speech with more accurate emotional expression.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a speech synthesis method, the method comprising:

[0006] Acquire sample data, wherein the sample data includes sample text, sample emotion information and sample original Mel spectrum, wherein the sample emotion information has sample non-language emotion features, and the sample original Mel spectrum has sample speech expression features of the sample object;

[0007] Inputting the sample text, the sample emotion information and the sample original Mel spectrum into an initial synthesis model, wherein the initial synthesis model includes an emotion detection sub-model and a spectrum generation sub-model;

[0008] Performing masking processing on the sample original Mel spectrum based on the sample emotion information to obtain a masked Mel spectrum and a sample masked Mel spectrum, wherein the masked Mel spectrum is used to characterize the Mel spectrum corresponding to the masked area in the sample original Mel spectrum, and the sample masked Mel spectrum is used to characterize the Mel spectrum obtained by masking the Mel spectrum value of the masked area in the sample original Mel spectrum;

[0009] Performing emotion feature detection on the sample emotion information and the sample original Mel spectrum based on the emotion detection sub-model to obtain the sample target emotion feature of the sample masked Mel spectrum;

[0010] Based on the spectrum generation sub-model, spectrum generation is performed on the masked Mel spectrum of the sample, the sample target sentiment feature and the sample text to obtain a predicted Mel spectrum of the sample object;

[0011] Adjusting parameters of the initial synthesis model based on the masked Mel spectrum, the predicted Mel spectrum and the sample target emotion feature to obtain a speech synthesis model;

[0012] Based on the speech synthesis model, speech synthesis processing is performed on the target text, the target object speech with the target speech expression characteristics and the target emotional information with the target non-language emotional characteristics to obtain the target synthesized speech with the target speech expression characteristics and the target emotional characteristics.

[0013] In some embodiments, the performing emotion detection on the sample emotion information and the sample original Mel spectrum based on the emotion detection sub-model to obtain the sample target emotion feature of the sample masked Mel spectrum includes:

[0014] Performing emotion detection on the sample emotion information to obtain the sample non-language emotion feature, wherein the sample non-language emotion feature is used to indicate the emotion feature of the masked Mel spectrum;

[0015] Performing emotion detection on the original Mel spectrum of the sample to obtain the sample language emotion feature;

[0016] The sample non-language emotion feature and the sample language emotion feature are feature concatenated to obtain the sample target emotion feature of the masked Mel spectrum of the sample.

[0017] In some embodiments, the initial synthesis model further includes a phoneme detection sub-model, and the spectrum generation sub-model performs spectrum generation on the sample masked Mel spectrum, the sample target emotional feature and the sample text to obtain a predicted Mel spectrum of the sample object, including:

[0018] Performing phoneme detection on the sample original Mel spectrum and the sample text based on the phoneme detection sub-model to obtain a sample phoneme sequence;

[0019] Based on the spectrum generation sub-model, spectrum generation is performed on the sample masked Mel spectrum, the sample target emotional feature and the sample phoneme sequence to obtain the predicted Mel spectrum.

[0020] In some embodiments, the performing spectrum generation on the sample masked Mel spectrum, the sample target emotion feature and the sample phoneme sequence based on the spectrum generation submodel to obtain a predicted Mel spectrum includes:

[0021] Performing time dimension alignment on the sample masked Mel spectrum based on the sample phoneme sequence to obtain the sample aligned Mel spectrum;

[0022] Based on the spectrum generation sub-model, spectrum generation is performed on the aligned Mel spectrum of the samples, the target emotion feature of the samples and the sample phoneme sequence to obtain the predicted Mel spectrum.

[0023] In some embodiments, the performing spectrum generation on the sample aligned Mel spectrum, the sample target emotion feature and the sample phoneme sequence based on the spectrum generation submodel to obtain the predicted Mel spectrum includes:

[0024] Aligning the sample target emotion features in time dimension based on the sample phoneme sequence to obtain the sample aligned emotion features;

[0025] Based on the spectrum generation sub-model, spectrum generation is performed on the sample aligned Mel spectrum, the sample aligned emotional features and the sample phoneme sequence to obtain the predicted Mel spectrum.

[0026] In some embodiments, the initial synthesis model further includes a vocoder, and the parameter adjustment of the initial synthesis model based on the masked Mel spectrum, the predicted Mel spectrum and the sample target emotion feature to obtain a speech synthesis model includes:

[0027] Calculating spectrum loss based on the masked Mel spectrum and the predicted Mel spectrum to obtain a spectrum loss value, wherein the masked Mel spectrum has the sample speech expression feature, the predicted Mel spectrum has the predicted speech expression feature, and the spectrum loss value is used to characterize the degree of difference between the sample speech expression feature and the predicted speech expression feature;

[0028] Performing spectrum conversion on the predicted Mel spectrum based on the vocoder to obtain predicted speech;

[0029] Extracting emotional features from the predicted speech to obtain predicted emotional features;

[0030] Calculate the emotion feature loss based on the predicted emotion feature and the sample target emotion feature to obtain an emotion feature loss value;

[0031] A model loss value is determined based on the spectrum loss value and the emotional feature loss value, and parameters of the initial synthesis model are adjusted based on the model loss value to obtain the speech synthesis model.

[0032] In some embodiments, the determining the model loss value based on the spectrum loss value and the emotional feature loss value comprises:

[0033] Extracting spectrum features from the masked Mel spectrum to obtain masked spectrum features;

[0034] Performing feature detection on the mask spectrum feature based on the original Mel spectrum of the sample to obtain a mask feature score, wherein the mask feature score is used to characterize the importance of the mask spectrum feature in the original Mel spectrum of the sample;

[0035] Determine a first loss weight of the spectrum loss value and a second loss weight of the emotion feature loss value based on the mask feature score;

[0036] The model loss value is obtained by performing weighted calculation based on the spectrum loss value, the first loss weight, the emotional feature loss value and the second loss weight.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech synthesis device, the device comprising:

[0038] An acquisition module is used to acquire sample data, wherein the sample data includes sample text, sample emotion information and sample original Mel spectrum, wherein the sample emotion information has sample non-language emotion characteristics, and the sample original Mel spectrum has sample speech expression characteristics of the sample object;

[0039] An input module, used for inputting the sample text, the sample emotion information and the sample original Mel spectrum into an initial synthesis model, wherein the initial synthesis model includes an emotion detection sub-model and a spectrum generation sub-model;

[0040] A masking module is used to perform masking processing on the original Mel spectrum of the sample based on the sample emotion information to obtain a masked Mel spectrum and a sample masked Mel spectrum, wherein the masked Mel spectrum is used to characterize the Mel spectrum corresponding to the masked area in the original Mel spectrum of the sample, and the sample masked Mel spectrum is used to characterize the Mel spectrum obtained by masking the Mel spectrum value of the masked area in the original Mel spectrum of the sample;

[0041] An emotion detection module is used to perform emotion feature detection on the sample emotion information and the sample original Mel spectrum based on the emotion detection sub-model to obtain the sample target emotion feature of the sample masked Mel spectrum;

[0042] A spectrum generation module, used for performing spectrum generation on the masked Mel spectrum of the sample, the sample target sentiment feature and the sample text based on the spectrum generation sub-model to obtain a predicted Mel spectrum of the sample object;

[0043] A parameter adjustment module, used for adjusting the parameters of the initial synthesis model based on the masked Mel spectrum, the predicted Mel spectrum and the sample target emotion feature to obtain a speech synthesis model;

[0044] The synthesis module is used to perform speech synthesis processing on the target text, the target object speech with the target speech expression characteristics and the target emotional information with the target non-verbal emotional characteristics based on the speech synthesis model to obtain the target synthesized speech with the target speech expression characteristics and the target emotional characteristics.

[0045] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements any method proposed in the first aspect of the embodiment of the present application.

[0046] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application further proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements any method proposed in the first aspect of the embodiments of the present application.

[0047] The speech synthesis method, speech synthesis device, electronic device and storage medium proposed in the embodiments of the present application first obtain sample data, the sample data includes sample text, sample emotion information and sample original Mel spectrum, the sample emotion information has sample non-language emotion characteristics, and the sample original Mel spectrum has sample speech expression characteristics of the sample object; further, the sample text, the sample emotion information and the sample original Mel spectrum are input into an initial synthesis model, the initial synthesis model includes an emotion detection sub-model and a spectrum generation sub-model; based on the speech conversion sub-model, the speech of the sample object is speech converted to obtain the sample original Mel spectrum; based on the sample emotion information, the sample original Mel spectrum is masked to obtain a masked Mel spectrum and a sample masked Mel spectrum, the masked Mel spectrum is used to characterize the Mel spectrum corresponding to the masked area in the sample original Mel spectrum, and the sample masked Mel spectrum is used to characterize the Mel spectrum corresponding to the masked area in the sample original Mel spectrum. The spectrum is used to characterize the Mel spectrum obtained by masking the Mel spectrum value of the masked area in the original Mel spectrum of the sample; based on the emotion detection sub-model, the emotion feature detection is performed on the sample emotion information and the original Mel spectrum of the sample to obtain the sample target emotion feature of the masked Mel spectrum of the sample; based on the spectrum generation sub-model, the spectrum generation is performed on the sample masked Mel spectrum, the sample target emotion feature and the sample text to obtain the predicted Mel spectrum of the sample object; further, based on the masked Mel spectrum, the predicted Mel spectrum and the sample target emotion feature, the parameters of the initial synthesis model are adjusted to obtain the speech synthesis model; further, based on the speech synthesis model, the target text, the target object speech with the target speech expression feature and the target emotion information with the target non-language emotion feature are subjected to speech synthesis processing to obtain the target synthesized speech with the target speech expression feature and the target emotion feature. The speech synthesis method provided in the present application will combine the input target emotion information with the target non-language emotion feature to generate a natural emotional speech with non-language information. Compared with the related art that only uses splicing to generate emotional speech, the speech synthesis model constructed in this application can fuse language information (i.e., information containing speech content) and non-speech information (i.e., information that does not contain speech content) during the speech generation process, and achieve more precise control of speech emotions based on the provided target object speech and target emotion information, thereby generating synthesized speech with more accurate emotional expression. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the speech synthesis method provided in an embodiment of the present application;

[0049] Figure 2 yes Figure 1 A flowchart of the specific method of step S140;

[0050] Figure 3 yes Figure 1 A flowchart of the specific method of step S150;

[0051] Figure 4 yes Figure 3 A flowchart of the specific method of step S320;

[0052] Figure 5 yes Figure 4 A flowchart of the specific method of step S420;

[0053] Figure 6 yes Figure 1 A flowchart of the specific method of step S160;

[0054] Figure 7 yes Figure 6 A flowchart of the specific method of step S650;

[0055] Figure 8 It is a module structure block diagram of the speech synthesis device provided in an embodiment of the present application;

[0056] Fig. 9 It is a hardware structure diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] First, some nouns involved in this application are analyzed:

[0061] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0062] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text comprehension. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. Natural language processing involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0063] Text-To-Speech (TTS): is a technology that converts text into speech. TTS generally includes two steps: the first step is text processing, which is mainly to convert text into a phoneme sequence and mark the start and end time, frequency change and other information of each phoneme; the second step is speech synthesis, which is mainly to generate speech based on the phoneme sequence (as well as the marked start and end time, frequency change and other information).

[0064] Speech rhythm: also known as prosody, refers to the changes in pitch, duration, intensity, tone, etc. of speech, which is the expression of the rhythm and rhythm of speech. Rhythm can include intonation, pauses and rhythm, which can convey the speaker's emotions, attitudes and information. The role of speech rhythm can express emotions (that is, different emotions can be expressed through changes in pitch, duration, intensity and tone), distinguish meanings (that is, in some languages, the same speech unit can express different meanings through changes in rhythm), enhance the beauty of language (that is, in art forms such as poetry and songs, the use of rhythm can make the language more beautiful and pleasant to the ear), etc.

[0065] Speech synthesis refers to the process of creating natural language speech that meets the requirements based on given input data. At present, intelligent speech technology is usually used in task scenarios such as smart telephone customer service and smart sales in financial technology. However, human speech not only contains language information, but also some non-verbal information, such as crying, laughing, pausing, coughing, etc. These non-verbal information can be used to convey the speaker's different feelings and communication intentions. Therefore, adding some non-verbal information in the process of synthesizing speech can make the synthesized speech more natural and vivid, and closer to the speech in real life.

[0066] Based on this, the relevant technology usually combines voice information and non-language information in a certain way to synthesize a speech containing non-language information. However, this method does not deeply understand the relationship between language information and non-language information, which makes the connection between voice information and non-language information in the synthesized speech unnatural, thereby reducing the emotional expression of the synthesized speech. Therefore, how to generate a more accurate synthesized speech with more accurate emotional expression has become a technical problem that needs to be solved urgently.

[0067] Based on this, the speech synthesis method, speech synthesis device, electronic device and storage medium provided in the embodiments of the present application can generate synthesized speech with more accurate emotional expression.

[0068] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0069] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0070] The speech synthesis method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The speech synthesis method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, or a smart watch, etc.; the server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms; the software can be an application that implements the speech synthesis method, etc., but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as object information, object voice data, object historical data, and object identity information, the permission or consent of the object will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain sensitive personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the object, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0073] Please refer to Figure 1 , Figure 1 It is an optional flow chart of the speech synthesis method provided in the embodiment of the present application. In some embodiments of the present application, the speech synthesis method of the embodiment of the present application includes but is not limited to steps S110 to S170:

[0074] Step S110, obtaining sample data, where the sample data includes sample text, sample emotion information, and sample original Mel spectrum;

[0075] Step S120, inputting the sample text, the sample emotion information and the sample original Mel spectrum into the initial synthesis model;

[0076] Step S130, masking the original Mel spectrum of the sample based on the sample emotion information to obtain a masked Mel spectrum and a masked Mel spectrum of the sample;

[0077] Step S140, performing emotion feature detection on the sample emotion information and the sample original Mel spectrum based on the emotion detection sub-model, and obtaining the sample target emotion feature of the sample masked Mel spectrum;

[0078] Step S150, performing spectrum generation on the masked Mel spectrum of the sample, the sample target sentiment feature and the sample text based on the spectrum generation sub-model to obtain a predicted Mel spectrum of the sample object;

[0079] Step S160, adjusting parameters of the initial synthesis model based on the masked Mel spectrum, the predicted Mel spectrum and the sample target emotion feature to obtain a speech synthesis model;

[0080] Step S170, based on the speech synthesis model, speech synthesis processing is performed on the target text, the target object speech with target speech expression characteristics, and the target emotional information with target non-verbal emotional characteristics to obtain a target synthesized speech with target speech expression characteristics and target emotional characteristics.

[0081] In step S110 of some embodiments, sample data refers to data pre-acquired for training models. The present application may first acquire a training sample set, which includes multiple sample data, and the training sample set may be acquired from an existing synthetic speech data set as needed, or may be acquired and annotated in real time according to a specific acquisition device. For each sample data, sample text, sample emotional information, and sample object voice may be included. Among them, the sample text has text content, so that the generated voice can express the text content. The sample emotional information has sample non-language emotional features, and the sample emotional information is equivalent to a prompt information that expresses emotional features in the form of non-language information (such as crying, laughing, pausing, coughing, etc., which can be used to convey the speaker's different feelings and communication intentions). The sample emotional information may be a voice containing non-language emotional features (such as a voice with laughter or crying), or a text prompt, such as "the generated voice has crying", without specific limitation. The sample object voice refers to a prompt voice used to learn the speech rhythm features of the speaker (i.e., the sample object), and the sample voice has the sample rhythm features of the sample object. The sample prosodic features can characterize the changes in pitch, duration, intensity, tone, etc. of the corresponding sample text.

[0082] It should be noted that, for example, in the intelligent overcoming scenario of financial customers, if a self-introduction voice of a customer service representative is to be synthesized, the corresponding sample text may be "I am the intelligent customer service representative BB of the AA application", and the sample emotional information with sample non-language emotional characteristics input at this time may be a voice with laughter and no language content, and the sample object voice at this time may be a pre-recorded voice of the sample object. After the speech synthesis model of this application is used, a voice with laughter and the speaking style of the sample object may be generated, and the content of the voice is the text content of the sample text.

[0083] It should be noted that, since the sample emotional information is used to learn the emotional features corresponding to the non-verbal information, the speech content contained in the sample emotional information is not important and will not be described in detail.

[0084] It should be noted that the sample object speech is used to represent a reference speech when performing speech synthesis on the text content in the sample text. It is for learning the prosodic features of the synthesized speaker. Therefore, the speech content in the sample object speech is not necessarily the same as the text content of the sample text. The sample text can be a text phoneme sequence obtained by a text-to-phoneme conversion model. The text factor conversion model can adopt the Deep Voice3 model, the grapheme to phoneme conversion model (G2P), etc. In order to improve the efficiency of the model, the speech content of the sample object speech can also be set to be the same as the sample text, so as to guide the training process of the model based on the sample speech as the target output speech when performing speech synthesis on the sample text.

[0085] It should be noted that the sample object voices expressed by different voice prosody may correspond to different voice emotions, such as happiness, anger, sadness, excitement, and coquettishness, etc. The voice emotion may be the emotion expressed in a voice segment, the emotion expressed in a sentence, or the emotion expressed in a word, etc., and the embodiments of the present application do not specifically limit this.

[0086] It should be noted that the storage format of voice in this application can be MP3 format, CDA format, WAV format, WMA format, RA format, MIDI format, OGG format, APE format or AAC format, etc., and this application is not limited.

[0087] It should be noted that the speech synthesis method of the present application can be used to assist in car broadcasting and announcements, car navigation, electronic dictionaries, smart phones, voice assistants, e-book reading, etc. For example, in the e-book reading of financial technology, the target object speech with the target speech expression characteristics and the target emotion information with the target non-language emotion characteristics can be synthesized to have the target speech expression characteristics of the target object, the speech content is the text content of the target text, and the target synthesized speech with the target non-language emotion characteristics.

[0088] It should be noted that this application will consider non-verbal cue information, that is, emotional information with non-verbal emotional characteristics, when training the speech synthesis model and actual speech synthesis, not just the emotional characteristics of the learning text, so that the generated speech is more natural and vivid, and closer to the speech in real life.

[0089] It should be noted that before the sample data is input into the initial synthesis model, the present application can first convert the sample object speech into a Mel spectrum representation based on the speech conversion model, that is, obtain the sample original Mel spectrum, which is a spectral feature characterizing the sound signal. Among them, the speech conversion sub-model can be a model built based on a Mel filter, and the design of the Mel filter can be based on the human ear's auditory perception of different frequencies, linearly convert the frequency into a Mel frequency scale, and redefine the frequency axis to make it more consistent with the perception of the human ear.

[0090] It should be noted that the present application can divide the frequency range into multiple Mel filters, the common number of which is between 20 and 40, so as to construct a speech conversion sub-model based on multiple Mel filters (i.e., Mel filter banks), and these Mel filters can be distributed in a triangular shape, covering a range from low frequency to high frequency. The present application does not elaborate on the specific process of speech conversion.

[0091] It should be noted that the present application can set the speech conversion model in the initial synthesis model, so that the input target object speech can be converted into Mel spectrum representation first, which will not be described in detail.

[0092] In step S120 of some embodiments, the initial synthesis model includes an emotion detection sub-model, a phoneme detection sub-model, and a spectrum generation sub-model. In addition, the initial synthesis model of the present application can be constructed based on a flow matching model, a large language model (LLM, which refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text), a machine learning model, etc., which is not limited here.

[0093] Among them, the flow matching model is a specific generative model. Its core idea is to transform the data distribution into a simple distribution, such as the standard normal distribution, by learning a reversible transformation, so that the given noise can be transformed into the desired data using this reversible transformation.

[0094] In step S130 of some embodiments, the present application may further perform masking on the original Mel spectrum of the sample, specifically, masking a part of the original Mel spectrum to generate a new speech Mel spectrogram, that is, obtaining a masked Mel spectrum and a sample masked Mel spectrum. The masked Mel spectrum is used to characterize the Mel spectrum corresponding to the masked area extracted from the original Mel spectrum of the sample, and the sample masked Mel spectrum is used to characterize the Mel spectrum obtained by masking the Mel spectrum value of the masked area in the original Mel spectrum of the sample.

[0095] It should be noted that the mask processing of the present application can adopt a random masking method, that is, the Mel spectrum value of the mask area selected in the original Mel spectrum of the sample can be replaced with a predicted value (such as 0, 1, etc.), but the dimension of the Mel spectrum graph has not changed, only some values ​​in the Mel spectrum graph have changed.

[0096] It should be noted that the initial synthesis model of the present application will use the remaining part of the original Mel spectrum of the sample (i.e. the Mel spectrum after masking the sample) as input to assist the model in generating speech. Finally, the error between the generated speech Mel spectrum and the masked part of the original Mel spectrum of the sample (i.e. the masked Mel spectrum) is calculated as the loss to update the parameters.

[0097] In step S140 of some embodiments, emotion detection can be further performed on the sample emotion information and the sample original Mel spectrum based on the emotion detection sub-model to obtain the sample target emotion feature of the sample masked Mel spectrum. The present application does not extract the phoneme sequence and emotion feature from the new sample masked Mel spectrum, but the phoneme sequence and emotion feature extracted from the sample original Mel spectrum include the features of the unmasked part, and the phoneme sequence and emotion feature extracted from the sample emotion information include the features of the masked part, so as to control the generation of speech that is closer to the original text content, speech rhythm features and emotion features.

[0098] It should be noted that the emotion detection sub-model, phoneme detection sub-model, and spectrum generation sub-model of the present application can specifically adopt model structures such as convolutional neural network (CNN), variational auto encoder (VAE), generative adversarial network (GAN), diffusion model, autoregressive model, etc., and no specific limitation is made here.

[0099] Please refer to Figure 2 , Figure 2 It is an optional flow chart of step S140 provided in an embodiment of the present application. In some embodiments of the present application, step S140 may specifically include but is not limited to steps S210 to S230:

[0100] Step S210, performing emotion detection on the sample emotion information to obtain the sample non-language emotion features;

[0101] Step S220, performing emotion detection on the original Mel spectrum of the sample to obtain the emotion feature of the sample language;

[0102] Step S230, performing feature concatenation on the sample non-language emotion feature and the sample language emotion feature to obtain the sample target emotion feature.

[0103] In steps S210 to S230 of some embodiments, the sample non-language emotion features corresponding to the sample emotion information are used to indicate the emotion features of the masked Mel spectrum. The sample non-language emotion features refer to the emotion features extracted from the sample emotion information (i.e., non-language prompt information). The sample language emotion features corresponding to the sample original Mel spectrum are used to indicate the emotion features corresponding to the unmasked Mel spectrum. Furthermore, the sample non-language emotion features and the sample language emotion features can be feature spliced ​​to obtain a complete sample target emotion feature.

[0104] It should be noted that the sample non-verbal emotional features and sample language emotional features detected in this application have different functions. The sample object voice is equivalent to the speaker's voice prompt, which is used to control what kind of speaker will speak the final generated voice, while the sample emotional information is equivalent to the non-verbal information prompt, which is used to control the emotion of the speech. For example, different people's laughter is different. There is a clear difference between object A speaking with a smile and object B speaking with a smile, so the different object rhythmic features expressed by the generated voice can be controlled according to the different input object voices. The sample emotional information can control whether the final generated voice is A speaking with a smile or A speaking with tears. Therefore, by performing feature splicing on the sample non-verbal emotional features and the sample language emotional features, a sample target emotional feature that can fully express the included language emotional features and non-verbal emotional features can be obtained.

[0105] In step S150 of some embodiments, the present application may further perform spectrum generation on the sample masked Mel spectrum, the sample target emotional features and the sample text based on the spectrum generation sub-model to obtain the predicted Mel spectrum of the sample object.

[0106] Please refer to Figure 3 , Figure 3 It is an optional flow chart of step S150 provided in an embodiment of the present application. In some embodiments of the present application, step S150 may specifically include but is not limited to steps S310 to S320:

[0107] Step S310, performing phoneme detection on the sample original Mel spectrum and the sample text based on the phoneme detection sub-model to obtain a sample phoneme sequence;

[0108] Step S320 , performing spectrum generation on the sample masked Mel spectrum, the sample target emotion feature and the sample phoneme sequence based on the spectrum generation sub-model to obtain a predicted Mel spectrum.

[0109] In step S310 of some embodiments, the present application may input the sample original Mel spectrum and sample text of the sample object speech into the phoneme detection sub-model for automatic speech recognition to detect a complete sample phoneme sequence.

[0110] It should be noted that the present application performs phoneme detection on the sample original Mel spectrum and sample text based on the phoneme detection sub-model to obtain the sample phoneme sequence specifically including: performing phoneme detection on the sample original Mel spectrum to obtain a first sample phoneme subsequence; performing phoneme detection on the sample text to obtain a second sample phoneme subsequence; performing feature splicing on the first sample phoneme subsequence and the second sample phoneme subsequence to obtain a sample phoneme sequence. Among them, the first sample phoneme subsequence corresponds to the phoneme sequence of the unmasked Mel spectrum in the sample original Mel spectrum. The second sample phoneme subsequence corresponds to the phoneme sequence of the masked Mel spectrum in the sample original Mel spectrum. Furthermore, the first sample phoneme subsequence and the second sample phoneme subsequence are feature spliced ​​to obtain a complete sample phoneme sequence.

[0111] It should be noted that the specific process of phoneme detection can be to input the sample object speech and sample text into a phoneme detection sub-model (such as a deep learning model) to process the audio signal. The phoneme detection sub-model can divide the sample object speech and sample text into different phoneme segments according to spectral features and acoustic characteristics. Furthermore, the phoneme detection sub-model can output phoneme annotations in each time period, which indicate the type of phonemes in the time period. Ultimately, all phoneme annotations will be organized into a sample phoneme subsequence, representing all phonemes of the sample phoneme subsequence corresponding to the sample object speech and sample text.

[0112] In step S320 of some embodiments, the spectrum generation submodel for spectrum generation in the present application can be constructed based on the stream matching model, and after obtaining the complete sample phoneme sequence, the duration can be predicted by detecting the second sample phoneme subsequence to obtain the target speech duration. Furthermore, the masked Mel spectrograms of the samples can be aligned in the time dimension based on the target speech duration.

[0113] Please refer to Figure 4 , Figure 4 It is an optional flow chart of step S320 provided in an embodiment of the present application. In some embodiments of the present application, step S320 may specifically include but is not limited to steps S410 to S420:

[0114] Step S410, aligning the sample masked Mel spectrum in time dimension based on the sample phoneme sequence to obtain the sample aligned Mel spectrum;

[0115] Step S420 , performing spectrum generation on the aligned Mel spectrum of the samples, the target emotion feature of the samples, and the sample phoneme sequence based on the spectrum generation sub-model to obtain a predicted Mel spectrum.

[0116] In step S410 and step S420 of some embodiments, since the sample masked Mel spectrum is equivalent to the Mel spectrum of the sample object, and the complete sample phoneme sequence may be longer than the sample masked Mel spectrum in the time dimension. Therefore, the sample masked Mel spectrum can be filled with a corresponding number of preset elements (such as 0 or 1, etc.) to align with the complete sample phoneme sequence in the time dimension. Afterwards, the spectrum is generated for the sample aligned Mel spectrum, the sample target emotional features and the sample phoneme sequence based on the spectrum generation submodel to obtain a predicted Mel spectrum.

[0117] In the above embodiments, the present application ensures the accurate association between the phoneme information and the model output by aligning and generating the phoneme sequence to the Mel spectrum. These generated Mel spectrums can not only reflect the speech content, but also tend to the target emotion, making the final generated speech more natural and expressive. Compared with the related art that only uses splicing to generate emotional speech, the speech synthesis model constructed in the present application can extract features, splice and fuse, align, etc. of language information (i.e., information containing speech content) and non-speech information (i.e., information that does not contain speech content) during the speech generation process, so as to achieve more refined control of speech emotion, thereby generating synthesized speech with more accurate emotional expression.

[0118] Please refer to Figure 5 , Figure 5 It is an optional flow chart of step S420 provided in an embodiment of the present application. In some embodiments of the present application, step S420 may specifically include but is not limited to steps S510 to S520:

[0119] Step S510, aligning the sample target emotion features in the time dimension based on the sample phoneme sequence to obtain the sample aligned emotion features;

[0120] Step S520 , performing spectrum generation on the sample aligned Mel spectrum, the sample aligned emotional features and the sample phoneme sequence based on the spectrum generation sub-model to obtain a predicted Mel spectrum.

[0121] In step S510 and step S520 of some embodiments, in order to ensure that the output predicted Mel spectrum not only represents the correct phoneme information, but also reflects the specified non-language emotional features, the present application can also align the sample target emotional features in the time dimension based on the sample phoneme sequence when performing spectrum generation based on the spectrum generation sub-model to obtain the sample aligned emotional features. Furthermore, the aligned sample aligned emotional features, the sample aligned Mel spectrum, and the sample phoneme sequence are fed to the model as input. In this way, the generated predicted Mel spectrum can take into account both the text content and the emotional features, so that the final generated speech expression matches the expected target emotion. Among them, the predicted Mel spectrum is a Mel spectrum with sample non-language emotional features, predicted emotional features, and corresponding to the text content of the sample text.

[0122] It should be noted that, in order to improve the quality of the generated Mel spectrum, the present application may also perform some post-processing on the output predicted Mel spectrum, such as denoising, smoothing, etc. In addition, the initial synthesis model of the present application also includes a vocoder. After obtaining the predicted Mel spectrum, speech generation can be performed on the predicted Mel spectrum based on the vocoder to obtain audible predicted speech.

[0123] In step S160 of some embodiments, after obtaining the predicted Mel spectrum, the present application can adjust the parameters of the initial synthesis model based on the masked Mel spectrum, the predicted Mel spectrum and the sample target emotion characteristics to train the initial synthesis model until the preset training end condition is reached to obtain a speech synthesis model.

[0124] Please refer to Figure 6 , Figure 6 It is an optional flow chart of step S160 provided in an embodiment of the present application. In some embodiments of the present application, step S160 may specifically include but is not limited to steps S610 to S650:

[0125] Step S610, calculating spectrum loss based on the masked Mel spectrum and the predicted Mel spectrum to obtain a spectrum loss value;

[0126] Step S620, performing spectrum conversion on the predicted Mel spectrum based on the vocoder to obtain predicted speech;

[0127] Step S630, extracting emotional features from the predicted speech to obtain predicted emotional features;

[0128] Step S640, calculating the emotion feature loss based on the predicted emotion feature and the sample target emotion feature to obtain the emotion feature loss value;

[0129] Step S650, determining a model loss value based on the spectrum loss value and the emotional feature loss value, and adjusting parameters of the initial synthesis model based on the model loss value to obtain a speech synthesis model.

[0130] In step S610 of some embodiments, the masked Mel spectrum (i.e., the real Mel spectrum) has the sample speech expression characteristics of the sample object, the predicted Mel spectrum (synthesized by the model according to the phoneme sequence and the emotional characteristics) has the predicted speech expression characteristics, and the spectrum loss value is used to characterize the degree of difference between the sample speech expression characteristics and the predicted speech expression characteristics. The loss function used in the spectrum loss calculation can be the mean square error (MSE) or the mean absolute error (MAE), etc., to measure the gap between the spectrum generated by the model and the target spectrum.

[0131] In step S620 and step S630 of some embodiments, the vocoder is a model that converts a mel spectrum into an audio waveform. Common vocoders include WaveGlow, Parallel WaveGAN, or HiFi-GAN. The predicted mel spectrum is input into the vocoder, and the corresponding audio signal, i.e., the predicted speech, is generated through its internal algorithm. Furthermore, the predicted speech is subjected to emotional feature extraction to obtain the predicted emotional feature, so that the emotional consistency between the predicted emotional feature extracted from the generated predicted speech and the sample target emotional feature can be evaluated.

[0132] It should be noted that the emotional feature extraction of the predicted speech can use a pre-trained emotion recognition model or an audio processing method, such as a feature extractor based on sentiment analysis, to extract features such as pitch, tone, intensity, etc. into a feature vector representing the emotional state.

[0133] In step S640 of some embodiments, an emotion feature loss calculation is performed based on the predicted emotion feature and the sample target emotion feature to compare the speech emotion generated by the model and obtain an emotion feature loss value.

[0134] In step S650 of some embodiments, further, a model loss value is determined based on the spectrum loss value and the emotional feature loss value, and parameters of the initial synthesis model are adjusted based on the model loss value to obtain a speech synthesis model, that is, the model parameters are optimized by comprehensively considering the spectrum loss and emotional feature loss to improve the quality of speech synthesis.

[0135] In the above embodiment, the present application constitutes a complete generation and optimization process from spectrum generation to speech synthesis, and then to emotional feature extraction and loss calculation, ensuring that the generated speech can achieve the expected effect in terms of content and emotional expression. This comprehensive method provides a good framework for emotional speech synthesis, making the generated speech more natural and expressive.

[0136] It should be noted that the preset training end condition can be when the synthesis accuracy of the initial synthesis model is greater than or equal to the preset accuracy threshold, or the current number of iterations of the initial synthesis model reaches the preset number of iterations, that is, the currently adjusted model can generate complete speech that meets user feedback. In addition, the synthesis accuracy is calculated based on the emotional similarity between the predicted emotional features and the sample target emotional features, and the spectral similarity between the masked Mel spectrum and the predicted Mel spectrum, and the function for similarity calculation can be selected according to actual needs, such as cosine similarity calculation, mean square error loss function, cross entropy loss function, contrast loss function, etc., which are not specifically limited here.

[0137] Please refer to Figure 7 , Figure 7 It is an optional flow chart of step S650 provided in an embodiment of the present application. In some embodiments of the present application, step S650 may specifically include but is not limited to steps S710 to S740:

[0138] Step S710, extracting spectrum features from the masked Mel spectrum to obtain masked spectrum features;

[0139] Step S720, performing feature detection on the mask spectrum feature based on the original Mel spectrum of the sample to obtain a mask feature score;

[0140] Step S730, determining a first loss weight of the spectrum loss value and a second loss weight of the emotion feature loss value based on the mask feature score;

[0141] Step S740, performing weighted calculation based on the spectrum loss value, the first loss weight, the emotional feature loss value and the second loss weight to obtain the model loss value.

[0142] In step S710 of some embodiments, in order to obtain a more accurate model loss value and improve the speech synthesis ability of the finally trained speech synthesis model, the present application may first select a feature extraction algorithm: a spectral feature extraction method (such as Mel Frequency Cepstrum Coefficient (MFCC)) and a deep learning model (such as CNN or RNN) may be used for feature extraction to obtain a rich spectral feature description. These features may include spectral energy, frequency distribution, and phase information.

[0143] In step S720 of some embodiments, further, feature matching, similarity calculation (such as cosine similarity) or convolutional neural network methods can be used to calculate the similarity between the mask spectrum feature and the sample original Mel spectrum. Through calculation, a score value is obtained to indicate the importance of the mask spectrum feature in the sample original Mel spectrum. A high score indicates that the mask spectrum feature is more critical for reconstructing or generating the target spectrum.

[0144] In step S730 of some embodiments, further, a range or ratio may be set, and the mask feature scores may be normalized to obtain a first loss weight (for spectral loss) and a second loss weight (for emotional feature loss), and ensure that the two loss weights can reflect the importance of different features. These weights may be derived through a simple linear relationship or adjusted through a nonlinear function (such as a sigmoid function).

[0145] In step S740 of some embodiments, further, a weighted calculation is performed based on the spectrum loss value, the first loss weight, the emotional feature loss value, and the second loss weight to obtain a model loss value, that is, the spectrum loss and the emotional feature loss are comprehensively considered to calculate the overall loss value of the model, providing a basis for model optimization. In this way, the specific calculation of the model loss value can be: model loss value = first loss weight × spectrum loss value + second loss weight × emotional feature loss value. This weighted calculation ensures that the model can take into account both spectrum quality and emotional expression during the optimization process. The model loss value finally calculated will be used for reverse optimization in the training process to guide the model to adjust parameters and improve the synthesis effect.

[0146] In the above embodiments, when calculating the model loss value, the present application can combine the spectrum loss value with the emotional feature loss value, and assign corresponding weights to them respectively to meet different concerns. Furthermore, the back propagation algorithm and optimizer (such as Adam or SGD) can be used to update the parameters of the initial synthesis model according to the model loss value, in order to improve the performance of the generated speech, including speech quality and emotional consistency. In this way, the final model after adjustment is an optimized speech synthesis model, which can more accurately generate speech of the target emotional state.

[0147] In step S170 of some embodiments, in practical applications, such as smart phones, a text-to-speech speech synthesis device can be installed on the smart phone, and the speech synthesis device deploys the speech synthesis model trained by the present application. When the operation of converting the target text into a synthesized speech with the target prosodic features of the target speech is detected, the smart phone can generate a speech synthesis service request and send the speech synthesis service request to the speech synthesis device. By responding to the speech synthesis service request, the smart phone uses the speech synthesis device to extract the target text, the target object speech with the target speech expression features, and the target emotional information with the target non-language emotional features from the speech synthesis service request. Further, the speech synthesis model can generate a text information with a speaking content of the target text, a target speech expression feature of the speaking object and the target object speech similar to each other, and a target speech Mel spectrum with the target non-language emotional features in the non-language information from the features extracted from these three information, and the target speech Mel spectrum is passed through the vocoder to generate the target synthesized speech. That is to say, after the target text, target object speech and target emotion information are input into the trained speech synthesis model, the speech synthesis model can first perform Mel spectrum conversion on the target object speech, and after obtaining the predicted Mel spectrum, generate the target synthesized speech through the vocoder.

[0148] It should be noted that the non-Company software tools or components appearing in the embodiments of the present application are merely examples and do not represent actual use.

[0149] A speech synthesis method provided in an embodiment of the present application can generate more vivid and natural emotional speech with non-language information, such as speech clips with laughter or crying. Compared with the related art that only uses splicing to generate emotional speech, the speech synthesis model in the present application can integrate language information and non-language information in the speech generation process, and achieve more refined control of speech emotion by providing a variety of non-language information prompts, guide the emotional expression of generated speech, and communicate with users more cordially in the use scenario, not only responding to speech information, but also giving corresponding feedback to emotional information. In addition, the speech synthesis model of the present application adopts a stream matching model for modeling, which can effectively improve the generation speed. Therefore, the speech synthesis model constructed by the present application can combine the masking and speech conversion methods to integrate language information (i.e., information containing speech content) and non-speech information (i.e., information not containing speech content) in the speech generation process, and achieve more refined control of speech emotion according to the target object speech and target emotional information provided, so as to generate more accurate synthetic speech with emotional expression.

[0150] Please refer to Figure 8 , Figure 8: is a schematic diagram of the module structure of the speech synthesis device provided in the embodiment of the present application. In some embodiments of the present application, the speech synthesis device may specifically include:

[0151] An acquisition module 810 is used to acquire sample data, where the sample data includes sample text, sample emotion information, and a sample original Mel spectrum. The sample emotion information has sample non-language emotion features, and the sample original Mel spectrum has sample speech expression features of the sample object.

[0152] An input module 820 is used to input the sample text, the sample emotion information and the sample original Mel spectrum into an initial synthesis model, where the initial synthesis model includes an emotion detection sub-model and a spectrum generation sub-model;

[0153] The masking module 830 is used to perform masking processing on the original Mel spectrum of the sample based on the sample emotion information to obtain a masked Mel spectrum and a sample masked Mel spectrum, wherein the masked Mel spectrum is used to characterize the Mel spectrum corresponding to the masked area in the original Mel spectrum of the sample, and the sample masked Mel spectrum is used to characterize the Mel spectrum obtained by masking the Mel spectrum value of the masked area in the original Mel spectrum of the sample;

[0154] The emotion detection module 840 is used to perform emotion feature detection on the sample emotion information and the sample original Mel spectrum based on the emotion detection sub-model, and obtain the sample target emotion feature of the sample masked Mel spectrum;

[0155] The spectrum generation module 850 is used to generate a spectrum for the sample masked Mel spectrum, the sample target sentiment feature and the sample text based on the spectrum generation sub-model to obtain a predicted Mel spectrum of the sample object;

[0156] A parameter adjustment module 860 is used to adjust the parameters of the initial synthesis model based on the masked Mel spectrum, the predicted Mel spectrum and the sample target emotion feature to obtain a speech synthesis model;

[0157] The synthesis module 870 is used to perform speech synthesis processing on the target text, the target object speech with target speech expression characteristics and the target emotional information with target non-verbal emotional characteristics based on the speech synthesis model to obtain the target synthesized speech with target speech expression characteristics and target emotional characteristics.

[0158] It should be noted that the speech synthesis device of the embodiment of the present application is used to execute the above-mentioned speech synthesis method, and the speech synthesis device of the embodiment of the present application corresponds to the aforementioned speech synthesis method. For the specific training process, please refer to the aforementioned speech synthesis method, which will not be described in detail here.

[0159] An embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the speech synthesis method of the embodiment of the present application is implemented.

[0160] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0161] Combine the following Fig. 9 The electronic device according to the embodiment of the present application is introduced in detail.

[0162] Please refer to Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, including:

[0163] The processor 910 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0164] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the speech synthesis method of the embodiment of this application;

[0165] Input / output interface 930, used to implement information input and output;

[0166] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0167] bus 950 , which transmits information between the various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0168] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0169] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the speech synthesis method of the embodiment of the present application is implemented.

[0170] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0171] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0172] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0173] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0175] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0176] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0181] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Acquire sample data, wherein the sample data includes sample text, sample emotion information and sample original Mel spectrum, wherein the sample emotion information has sample non-language emotion features, and the sample original Mel spectrum has sample speech expression features of the sample object; Inputting the sample text, the sample emotion information and the sample original Mel spectrum into an initial synthesis model, wherein the initial synthesis model includes an emotion detection sub-model and a spectrum generation sub-model; Performing masking processing on the sample original Mel spectrum based on the sample emotion information to obtain a masked Mel spectrum and a sample masked Mel spectrum, wherein the masked Mel spectrum is used to characterize the Mel spectrum corresponding to the masked area in the sample original Mel spectrum, and the sample masked Mel spectrum is used to characterize the Mel spectrum obtained by masking the Mel spectrum value of the masked area in the sample original Mel spectrum; Performing emotion feature detection on the sample emotion information and the sample original Mel spectrum based on the emotion detection sub-model to obtain the sample target emotion feature of the sample masked Mel spectrum; Based on the spectrum generation sub-model, spectrum generation is performed on the masked Mel spectrum of the sample, the sample target sentiment feature and the sample text to obtain a predicted Mel spectrum of the sample object; Adjusting parameters of the initial synthesis model based on the masked Mel spectrum, the predicted Mel spectrum and the sample target emotion feature to obtain a speech synthesis model; Based on the speech synthesis model, speech synthesis processing is performed on the target text, the target object speech with the target speech expression characteristics and the target emotional information with the target non-language emotional characteristics to obtain the target synthesized speech with the target speech expression characteristics and the target emotional characteristics.

2. The method according to claim 1, characterized in that The performing emotion detection on the sample emotion information and the sample original Mel spectrum based on the emotion detection sub-model to obtain the sample target emotion feature of the sample masked Mel spectrum includes: Performing emotion detection on the sample emotion information to obtain the sample non-language emotion feature, wherein the sample non-language emotion feature is used to indicate the emotion feature of the masked Mel spectrum; Performing emotion detection on the original Mel spectrum of the sample to obtain the sample language emotion feature; The sample non-language emotion feature and the sample language emotion feature are feature concatenated to obtain the sample target emotion feature of the masked Mel spectrum of the sample.

3. The method according to claim 1, characterized in that The initial synthesis model further includes a phoneme detection sub-model, and the spectrum generation sub-model performs spectrum generation on the masked Mel spectrum of the sample, the sample target emotional feature and the sample text to obtain a predicted Mel spectrum of the sample object, including: Performing phoneme detection on the sample original Mel spectrum and the sample text based on the phoneme detection sub-model to obtain a sample phoneme sequence; Based on the spectrum generation sub-model, spectrum generation is performed on the sample masked Mel spectrum, the sample target emotional feature and the sample phoneme sequence to obtain the predicted Mel spectrum.

4. The method according to claim 3, characterized in that The performing spectrum generation on the sample masked Mel spectrum, the sample target emotion feature and the sample phoneme sequence based on the spectrum generation sub-model to obtain a predicted Mel spectrum includes: Performing time dimension alignment on the sample masked Mel spectrum based on the sample phoneme sequence to obtain the sample aligned Mel spectrum; Based on the spectrum generation sub-model, spectrum generation is performed on the aligned Mel spectrum of the samples, the target emotion feature of the samples and the sample phoneme sequence to obtain the predicted Mel spectrum.

5. The method according to claim 4, characterized in that The performing spectrum generation on the sample aligned Mel spectrum, the sample target emotion feature and the sample phoneme sequence based on the spectrum generation sub-model to obtain the predicted Mel spectrum includes: Performing time dimension alignment on the sample target emotion features based on the sample phoneme sequence to obtain the sample aligned emotion features; Based on the spectrum generation sub-model, spectrum generation is performed on the sample aligned Mel spectrum, the sample aligned emotional features and the sample phoneme sequence to obtain the predicted Mel spectrum.

6. The method according to any one of claims 1 to 5, characterized in that: The initial synthesis model further includes a vocoder, and the parameter adjustment of the initial synthesis model based on the masked Mel spectrum, the predicted Mel spectrum and the sample target emotion feature to obtain a speech synthesis model includes: Calculating spectrum loss based on the masked Mel spectrum and the predicted Mel spectrum to obtain a spectrum loss value, wherein the masked Mel spectrum has the sample speech expression feature, the predicted Mel spectrum has the predicted speech expression feature, and the spectrum loss value is used to characterize the degree of difference between the sample speech expression feature and the predicted speech expression feature; Performing spectrum conversion on the predicted Mel spectrum based on the vocoder to obtain predicted speech; Extracting emotional features from the predicted speech to obtain predicted emotional features; Calculate the emotion feature loss based on the predicted emotion feature and the sample target emotion feature to obtain an emotion feature loss value; A model loss value is determined based on the spectrum loss value and the emotional feature loss value, and parameters of the initial synthesis model are adjusted based on the model loss value to obtain the speech synthesis model.

7. The method according to claim 6, characterized in that The determining of the model loss value based on the spectrum loss value and the emotional feature loss value comprises: Extracting spectrum features from the masked Mel spectrum to obtain masked spectrum features; Performing feature detection on the mask spectrum feature based on the original Mel spectrum of the sample to obtain a mask feature score, wherein the mask feature score is used to characterize the importance of the mask spectrum feature in the original Mel spectrum of the sample; Determine a first loss weight of the spectrum loss value and a second loss weight of the emotion feature loss value based on the mask feature score; The model loss value is obtained by performing weighted calculation based on the spectrum loss value, the first loss weight, the emotional feature loss value and the second loss weight.

8. A speech synthesis device, characterized in that: The device comprises: An acquisition module is used to acquire sample data, wherein the sample data includes sample text, sample emotion information and sample original Mel spectrum, wherein the sample emotion information has sample non-language emotion characteristics, and the sample original Mel spectrum has sample speech expression characteristics of the sample object; An input module, used for inputting the sample text, the sample emotion information and the sample original Mel spectrum into an initial synthesis model, wherein the initial synthesis model includes an emotion detection sub-model and a spectrum generation sub-model; A masking module is used to perform masking processing on the original Mel spectrum of the sample based on the sample emotion information to obtain a masked Mel spectrum and a sample masked Mel spectrum, wherein the masked Mel spectrum is used to characterize the Mel spectrum corresponding to the masked area in the original Mel spectrum of the sample, and the sample masked Mel spectrum is used to characterize the Mel spectrum obtained by masking the Mel spectrum value of the masked area in the original Mel spectrum of the sample; An emotion detection module is used to perform emotion feature detection on the sample emotion information and the sample original Mel spectrum based on the emotion detection sub-model to obtain the sample target emotion feature of the sample masked Mel spectrum; A spectrum generation module, used for performing spectrum generation on the masked Mel spectrum of the sample, the sample target sentiment feature and the sample text based on the spectrum generation sub-model to obtain a predicted Mel spectrum of the sample object; A parameter adjustment module, used for adjusting the parameters of the initial synthesis model based on the masked Mel spectrum, the predicted Mel spectrum and the sample target emotion feature to obtain a speech synthesis model; The synthesis module is used to perform speech synthesis processing on the target text, the target object speech with the target speech expression characteristics and the target emotional information with the target non-verbal emotional characteristics based on the speech synthesis model to obtain the target synthesized speech with the target speech expression characteristics and the target emotional characteristics.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images