Speech synthesis method, speech synthesis device, electronic device, and storage medium

Abstract

The speech synthesis method, the speech synthesis device, the electronic equipment and the storage medium provided by the embodiment of the present application relate to the fields of artificial intelligence and financial technology. The method comprises the following steps: content encoding is performed on a mask acoustic feature representation sequence by a content encoding teacher model and an initial content encoding student model respectively, so as to obtain a first content vector and a second content vector; target loss data is calculated according to the first content vector and the second content vector; the initial content encoding student model is adjusted in parameters according to the target loss data, so as to obtain a target content encoding student model; content encoding is performed on target speech data by the target content encoding student model, so as to obtain a target content vector; and decoding is performed according to the target content vector, a target speaker vector and a target prosody vector, so as to obtain target synthesized speech data. The embodiment of the present application can guarantee the content accuracy and emotion of speech synthesis, and also improves the speech synthesis efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and the technical field of financial technology, and in particular relates to a speech synthesis method, a speech synthesis device, an electronic device and a storage medium. BACKGROUND

[0002] At present, speech conversion (also known as speaker conversion) in speech synthesis is widely used in financial customer service scenarios. Speech conversion aims to convert the voice of one person into another person without changing the content of the speech. The prior art proposes a method of decoupling speech into two parts of timbre and content based on an autoencoder, wherein the content encoder uses a bottleneck layer to constrain the content features from the channel level. Due to the limited size of the data set used by the content encoder during training, and the limited prediction performance of the autoencoder with a simple network structure, the content encoding accuracy is low, resulting in poor content accuracy of speech synthesis. In addition, although the prediction performance of the autoencoder with a complex network structure is good, the prediction efficiency is low.

[0003] Therefore, how to provide a speech synthesis method capable of ensuring the content accuracy of speech synthesis and improving the efficiency of speech synthesis has become a technical problem to be solved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, which can ensure the content accuracy and emotion of speech synthesis, and also improve the efficiency of speech synthesis.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application provides a speech synthesis method, which comprises:

[0006] obtaining an acoustic feature representation sequence of speech sample data to obtain an initial acoustic feature representation sequence;

[0007] mask processing the initial acoustic feature representation sequence to obtain a masked acoustic feature representation sequence;

[0008] content encoding the masked acoustic feature representation sequence by a preset content encoding teacher model to obtain a first content vector;

[0009] content encoding the masked acoustic feature representation sequence by a preset initial content encoding student model to obtain a second content vector; wherein the parameter quantity of the content encoding teacher model is greater than the parameter quantity of the initial content encoding student model;

[0010] content distillation loss calculation according to the first content vector and the second content vector to obtain target loss data;

[0011] Parameter adjustment is performed on the initial content coding student model according to the target loss data, to obtain a target content coding student model;

[0012] Target speech data is obtained;

[0013] The target content vector is obtained by performing content coding on the target speech data by the target content coding student model;

[0014] The target synthesized speech data is obtained by decoding according to the target content vector, a preset target speaker vector, and a preset target prosody vector.

[0015] In some embodiments, the content coding teacher model comprises a speech recognition pre-training sub-model and a normalization layer;

[0016] The first content vector is obtained by performing content coding on the masked acoustic feature representation sequence by the preset content coding teacher model, comprising:

[0017] The first prediction coding sequence is obtained by performing prediction coding on the masked acoustic feature representation sequence by the speech recognition pre-training sub-model;

[0018] The first content vector is obtained by performing instance normalization on the first prediction coding sequence by the normalization layer.

[0019] In some embodiments, the initial content coding student model comprises an attention sub-model and a content representation learning sub-model;

[0020] The second content vector is obtained by performing content coding on the masked acoustic feature representation sequence by the preset initial content coding student model, comprising:

[0021] The second prediction coding sequence is obtained by performing prediction coding on the masked acoustic feature representation sequence by the attention sub-model;

[0022] The second content vector is obtained by performing content representation on the second prediction coding sequence by the content representation learning sub-model.

[0023] In some embodiments, before the parameter adjustment is performed on the initial content coding student model according to the target loss data to obtain a target content coding student model, the method further comprises updating the target loss data, specifically comprising:

[0024] A sample prosody vector is obtained by performing prosody coding on the speech sample data by a preset prosody coding model;

[0025] A sample speaker vector is obtained by performing speaker recognition on the speech sample data;

[0026] decoding according to the second content vector, the sample speaker vector and the sample prosody vector, to obtain voice reconstruction data;

[0027] loss reconstruction according to the voice sample data and the voice reconstruction data, to obtain reconstruction loss data;

[0028] updating the target loss data according to the reconstruction loss data.

[0029] In some embodiments, the prosody encoding model comprises an acoustic feature extraction sub-model and a prosody feature extraction sub-model;

[0030] The prosody of the voice sample data is encoded by a preset prosody encoding model to obtain a sample prosody vector;

[0031] The acoustic feature extraction sub-model is used to extract acoustic features from the voice sample data to obtain an initial acoustic vector;

[0032] The prosody feature extraction sub-model is used to extract prosody features from the initial acoustic vector to obtain an acoustic prosody vector;

[0033] The initial acoustic vector and the acoustic prosody vector are fused to obtain the sample prosody vector.

[0034] In some embodiments, after the prosody of the voice sample data is encoded by a preset prosody encoding model to obtain a sample prosody vector, the method further comprises:

[0035] The sample prosody vector is subjected to emotion recognition to obtain a sample emotion category;

[0036] Emotion recognition loss calculation is performed according to the sample emotion category and a preset sample emotion label to obtain prosody loss data;

[0037] The prosody encoding model is adjusted according to the prosody loss data to obtain a target prosody encoding model; wherein the target prosody encoding model is used to encode the target voice data to obtain the target prosody vector.

[0038] In some embodiments, the content distillation loss calculation according to the first content vector and the second content vector to obtain the target loss data comprises:

[0039] Difference calculation is performed according to the first content vector and the second content vector to obtain a content encoding error;

[0040] Cosine calculation is performed on the content encoding error to obtain cosine data;

[0041] The hyperbolic cosine data is logarithmically calculated to obtain the target loss data.

[0042] To achieve the above object, a second aspect of the embodiment of the present application provides a speech synthesis device, which comprises:

[0043] a speech acquisition module, configured to acquire an acoustic feature representation sequence of speech sample data to obtain an initial acoustic feature representation sequence;

[0044] a mask processing module, configured to perform mask processing on the initial acoustic feature representation sequence to obtain a masked acoustic feature representation sequence;

[0045] a content encoding module, configured to perform content encoding on the masked acoustic feature representation sequence by a preset content encoding teacher model to obtain a first content vector;

[0046] The content encoding module is further configured to perform content encoding on the masked acoustic feature representation sequence by a preset initial content encoding student model to obtain a second content vector; wherein a parameter quantity of the content encoding teacher model is greater than a parameter quantity of the initial content encoding student model.

[0047] a loss calculation module, configured to perform content distillation loss calculation according to the first content vector and the second content vector to obtain target loss data;

[0048] a parameter adjustment module, configured to perform parameter adjustment on the initial content encoding student model according to the target loss data to obtain a target content encoding student model;

[0049] a speech synthesis module, configured to:

[0050] acquire target speech data;

[0051] perform content encoding on the target speech data by the target content encoding student model to obtain a target content vector;

[0052] decode according to the target content vector, a preset target speaker vector and a preset target prosody vector to obtain target synthesized speech data.

[0053] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech synthesis method of the first aspect when executing the computer program.

[0054] To achieve the above object, a fourth aspect of the embodiment of the present application provides a computer readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the speech synthesis method in the first aspect.

[0055] The speech synthesis method, the speech synthesis device, the electronic device and the storage medium provided by the present application first acquire an acoustic feature representation sequence of speech sample data to obtain an initial acoustic feature representation sequence. Secondly, the initial acoustic feature representation sequence is subjected to mask processing to obtain a masked acoustic feature representation sequence. Then, the masked acoustic feature representation sequence is subjected to content coding by a content coding teacher model and an initial content coding student model to obtain a first content vector and a second content vector. Then, the target loss data is calculated according to the first content vector and the second content vector, and the initial content coding student model is subjected to parameter adjustment according to the target loss data to obtain a target content coding student model. The second content vector output by the target content coding student model is nearly the same as the first content vector output by the content coding teacher model, that is, the target content coding student model has similar coding accuracy to the content coding teacher model, and the parameter amount of the target content coding student model is smaller, so the content coding efficiency is higher. Then, the target content coding student model is used to code the target speech data, and a target content vector with high accuracy can be obtained, so that the quality of the target synthesized speech data obtained based on the target content vector, a target speaker vector and a target prosody vector is better, and the efficiency is higher. In summary, the embodiment can guarantee the content accuracy of speech synthesis and improve the efficiency of speech synthesis.

[0056] Additional aspects and advantages of the present application will be made apparent by the following description. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a flowchart of the speech synthesis method provided by the embodiment of the present application;

[0058] Figure 2 is Figure 1 is a flowchart of step 103 in

[0059] Figure 3 is Figure 1 is a flowchart of step 104 in

[0060] Figure 4 is a flowchart of the speech synthesis method provided by another embodiment of the present application;

[0061] Figure 5 is Figure 4 is a flowchart of step 401 in

[0062] Figure 6 is a flowchart of a speech synthesis method provided by another embodiment of the present application;

[0063] Figure 7 is a whole schematic diagram of a speech synthesis method provided by an embodiment of the present application;

[0064] Figure 8 is a module structure block diagram of a speech synthesis device provided by an embodiment of the present application;

[0065] Figure 9 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0067] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0069] First, the terms involved in the present application are analyzed:

[0070] Artificial intelligence (artificial intelligence, AI): is a new technical science to study, develop and produce a new intelligent machine that can react in a similar way to human intelligence. The field of research includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0071] Natural language processing (NLP): NLP uses computers to process, understand and use human languages (such as Chinese, English, etc.), and NLP is a branch of artificial intelligence, an interdisciplinary subject of computer science and linguistics, and is also commonly referred to as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and linguistic research related to language computing.

[0072] Voice conversion technology: refers to a technology for converting original voice data into target voice data while keeping the semantic content unchanged, wherein the original voice data is the voice of an original speaker, and the target voice data is the voice of a target speaker, that is, the original voice of the original speaker is converted into the target voice of the target speaker with the same semantic content through voice conversion technology.

[0073] At present, financial customer service has become an indispensable part of financial institutions, and financial customer service is also increasingly automated and intelligent. Among them, voice conversion technology is particularly widely used in financial customer service. Voice conversion aims to convert the voice of one person into another person without changing the content of the speech, only changing the speaker's identity.

[0074] However, the content accuracy of the synthesized voice of the voice customer service system of the related technology is low, which affects the user experience. For example, when guiding the user, the speaking content in the synthesized voice is not clearly expressed, and the user cannot be effectively guided. In addition, the synthesized voice of the voice customer system of the related technology often lacks emotion, which also affects the user experience. For example, when soothing the user's emotions, the intelligent voice customer service is difficult to show natural emotions, and cannot effectively soothe the user. Although the related technology has a technology for extracting prosodic information in the voice to improve the emotion of the synthesized voice, it ignores the loss of content information in the synthesized voice, resulting in a synthesized voice that has emotion but still has low content accuracy. For example, the related technology also has a voice synthesis model of a benefit complex network structure, which ensures the content accuracy but greatly affects the voice synthesis efficiency.

[0075] Based on this, the embodiments of the present application propose a voice synthesis method, which can ensure the content accuracy of voice synthesis and improve the efficiency of voice synthesis, and also enable the synthesized voice to have emotion.

[0076] The speech synthesis method provided by the embodiments of the present application can be applied to a terminal and a server end, and can also be software running in the server end. The server end can be configured as a separate physical server, or can be configured as a server cluster or a distributed system formed by multiple physical servers, or can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform. The software can be an application that implements the speech synthesis method, but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0078] The embodiments of the present application provide a speech synthesis method and a speech synthesis device, an electronic device, and a storage medium. The embodiments are specifically described as follows. First, the speech synthesis method in the embodiments of the present application is described.

[0079] It should be noted that in each specific embodiment of the present application, when it is necessary to perform relevant processing according to the user's speech data and other data related to the user's identity or characteristics, the user's permission or consent is obtained first, and the collection, use, and processing of the data comply with relevant laws, regulations, and standards.

[0080] Reference Figure 1 , Figure 1 is an optional flowchart of the speech synthesis method provided by the embodiments of the present application, which can include but is not limited to steps 101 to 109.

[0081] Step 101, obtaining an acoustic feature representation sequence of speech sample data to obtain an initial acoustic feature representation sequence;

[0082] Step 102, performing mask processing on the initial acoustic feature representation sequence to obtain a masked acoustic feature representation sequence;

[0083] Step 103, content encoding is performed on the masked acoustic feature representation sequence by a preset content encoding teacher model to obtain a first content vector;

[0084] Step 104, content encoding is performed on the masked acoustic feature representation sequence by a preset initial content encoding student model to obtain a second content vector; the parameter quantity of the content encoding teacher model is greater than the parameter quantity of the initial content encoding student model;

[0085] Step 105, content distillation loss calculation is performed according to the first content vector and the second content vector to obtain target loss data;

[0086] Step 106, the initial content encoding student model is adjusted in parameters according to the target loss data to obtain a target content encoding student model;

[0087] Step 107, target speech data is obtained;

[0088] Step 108, content encoding is performed on the target speech data by the target content encoding student model to obtain a target content vector;

[0089] Step 109, decoding is performed according to the target content vector, a preset target speaker vector and a preset target prosody vector to obtain target synthesized speech data.

[0090] The steps 101 to 109 shown in the embodiments of the present application calculate target loss data according to the first content vector and the second content vector, and adjust the initial content encoding student model in parameters according to the target loss data to obtain a target content encoding student model. The second content vector output by the target content encoding student model is nearly the same as the first content vector output by the content encoding teacher model, that is, the target content encoding student model has similar encoding accuracy to the content encoding teacher model, and the parameter quantity of the target content encoding student model is smaller, so the content encoding efficiency is higher. Then, content encoding is performed on the target speech data by the target content encoding student model, and a target content vector with higher accuracy can be obtained, so that the quality of the target synthesized speech data obtained based on the target content vector, the target speaker vector and the target prosody vector is better and the efficiency is higher. In summary, the embodiments can guarantee the content accuracy of speech synthesis and improve the efficiency of speech synthesis.

[0091] When the embodiments of the present application are applied to telephone customer service, online customer service and other scenarios in the field of financial technology, highly natural and fluent speech output is obtained, user experience is improved, and communication barriers between users and intelligent customer service are reduced.

[0092] In step 101 of some embodiments, an acoustic feature representation sequence of the speech sample data is obtained, to obtain an initial acoustic feature representation sequence. The speech sample data is used to represent the speech uttered by the original speaker according to the original speech content. The initial acoustic feature representation sequence refers to a sequence of feature parameters used to describe the speech sample data. Common acoustic features include: (1) Short-T ime Energy: describes the intensity of the speech signal within a short time. (2) Short-T ime Zero Cross i ng Rate: describes the rate of change of the speech signal waveform. (3) Mel-Frequency Cepstra l Coeff i c i ents (MFCCs): a feature representation form commonly used in speech recognition, designed based on the perceptual characteristics of the human ear to sound. (4) Li near Pred i ct i ve Cod i ng Coeff i c i ents (LPCs): used to model the spectral envelope of the speech signal. (5) Me l-Spectrogram: describes the spectrogram representation of the speech signal in the Mel frequency.

[0093] Specifically, when the speech synthesis method is applied in a terminal, the speech sample data can be obtained by recording, Bluetooth transmission, wired transmission, or downloading. When the speech sample data is obtained by recording, the terminal is correspondingly configured with a microphone, and audio acquisition is performed through the microphone to realize recording of the speech sample data. When the speech synthesis method is applied in a server, the speech sample data can be uploaded to the server by the terminal, or downloaded from other servers or databases by the server.

[0094] When distinguishing the speaker, the timbre can be used for distinction. This is because the vocal cords and the vibration mode of the vocal cords of different people are different, resulting in different parameters such as vibration frequency and vibration amplitude of the sound, thereby causing different timbres of different speakers. The original speech content is used to represent the text information of the speech sample data. The speech sample data can be Chinese or English speech, etc.

[0095] In an embodiment, step 101 includes: obtaining speech sample data; and performing acoustic feature representation extraction on the speech sample data using a clustering algorithm to obtain an initial acoustic feature representation sequence. For example, for speech sample data x, clustering processing is performed using a clustering algorithm to obtain an initial acoustic feature representation sequence [x1, x2, …, xt]. t represents the length of the sequence.

[0096] It should be noted that in addition to using the clustering algorithm, other methods can also be used to extract acoustic feature representations from the speech sample data, such as neural network models.

[0097] In step 102 of some embodiments, the initial acoustic feature representation sequence is masked to obtain a masked acoustic feature representation sequence. Masking refers to a technique in the field of signal processing that uses a mask to process a signal. The mask usually refers to a binary sequence that indicates which part of the signal needs to be processed or retained. For example, for the initial acoustic feature representation sequence [x1, x2, …, xt], after masking, the masked acoustic feature representation sequence [x1, x2, [mask], x4, [mask], …, X1] can be obtained. The masked acoustic feature representation sequence is a representation sequence with a mask.

[0098] In step 103 of some embodiments, the masked acoustic feature representation sequence is content encoded by a preset content encoding teacher model to obtain a first content vector. The content encoding teacher model is a neural network model, and the content encoding effect is good but the parameter quantity is large and the running speed is slow. In this embodiment, the parameters of the content encoding teacher model are fixed, and the output first content vector can be used as knowledge to supervise the training of the initial content encoding student model.

[0099] In some embodiments, the content encoding teacher model includes a speech recognition pre-training sub-model and a normalization layer. Referring to Figure 2 , step 103 includes:

[0100] Step 201, predicting and encoding the masked acoustic feature representation sequence by the speech recognition pre-training sub-model to obtain a first prediction encoding sequence;

[0101] Step 202, instance normalization of the first prediction encoding sequence by the normalization layer to obtain the first content vector.

[0102] Specifically, the speech recognition pre-training sub-model is a neural network model combined with a mask prediction mechanism. The speech recognition pre-training sub-model can predict the content masked in the mask acoustic feature representation sequence, and can learn the long-time structure of the unmasked acoustic feature representation and the speech sample data. The mask acoustic feature representation sequence is input into the speech recognition pre-training sub-model for prediction coding, and a first prediction coding sequence can be obtained. The first prediction coding sequence includes content information and also contains speaker information. Therefore, the first prediction coding sequence needs to be input into a normalization layer for instance normalization, and a first content vector can be obtained. The first content vector can represent the speaking content in the speech sample data and does not contain the speaker information. The speech recognition pre-training sub-model can be a HuBERT (Hidden-Unit BERT) model. The normalization layer is specifically an IN (Instance Normalization) layer.

[0103] The above embodiment has the benefit that the prediction coding is performed before the instance normalization, thereby improving the accuracy of obtaining the first content vector.

[0104] In step 104 of some embodiments, the mask acoustic feature representation sequence is content coded by a preset initial content coding student model to obtain a second content vector. The initial content coding student model is a neural network model with poor content coding effect but small parameter quantity and fast running speed. The parameter quantity of the content coding teacher model is greater than that of the initial content coding student model. The mask acoustic feature representation sequence is input into the initial content coding student model for content coding, and a second content vector can be obtained.

[0105] In some embodiments, the initial content coding student model includes an attention sub-model and a content representation learning sub-model. Referring to Figure 3 , step 104 includes:

[0106] Step 301, the mask acoustic feature representation sequence is prediction coded by the attention sub-model to obtain a second prediction coding sequence;

[0107] Step 302, the second prediction coding sequence is content represented by the content representation learning sub-model to obtain a second content vector.

[0108] Specifically, the attention sub-model is a neural network model combined with a mask prediction mechanism. The attention sub-model can predict the content masked in the sequence of masked acoustic feature representations, and can learn the long-time structure of the unmasked acoustic feature representations and the speech sample data. By inputting the sequence of masked acoustic feature representations into the attention sub-model for prediction coding, a second prediction coding sequence can be obtained. The second prediction coding sequence includes content information and also contains speaker information. Therefore, the second prediction coding sequence needs to be input into a content representation learning sub-model for content representation, and a second content vector can be obtained. The second content vector can represent the content of speech sample data and does not contain speaker information. The attention sub-model can have a similar model structure as the HuBERT (Hidden-Unit BERT) model, but the parameter amount of the attention sub-model is less than that of the HuBERT model. The content representation learning sub-model can be a recurrent neural network (RNN).

[0109] The above-mentioned embodiments have the benefit that the content representation is performed after the prediction coding, which improves the content coding vector and the accuracy of the second content vector.

[0110] In step 105 of some embodiments, content distillation loss calculation is performed according to the first content vector and the second content vector to obtain target loss data. The target loss data is used to represent the content coding error between the initial content coding student model and the content coding teacher model. The smaller the target loss data is, the smaller the content coding error is, and the closer the content coding accuracy of the initial content coding student model is to the content coding teacher model.

[0111] In an embodiment, step 105 includes:

[0112] performing difference calculation on the first content vector and the second content vector to obtain a content coding error;

[0113] performing hyperbolic cosine calculation on the content coding error to obtain hyperbolic cosine data;

[0114] performing logarithmic calculation on the hyperbolic cosine data to obtain the target loss data.

[0115] The calculation process of the target loss data is shown in the following formula:

[0116] L = L distill = Log(cosh(R1-R2)) formula (1).

[0117] In formula (1), L represents the target loss function, L distill represents the content distillation loss data, R1 represents the first content vector, and R2 represents the second content vector.

[0118] The advantage of the above embodiments is that, instead of directly using the content encoding error as the target loss data, hyperbolic cosine function and logarithmic function are introduced to calculate the target loss data for the content encoding error, which can improve the model convergence speed and improve the model training efficiency.

[0119] It should be noted that the target loss data can also be calculated using functions such as the mean square error function and the mean absolute error function, but this embodiment does not specifically limit this.

[0120] After obtaining L distill After that, you can directly execute step 106, which is to directly use L. distill The parameters of the initial content-encoded student model are adjusted to obtain the target content-encoded student model.

[0121] In one embodiment, reference is made to Figure 4 Before step 106, the speech synthesis method of this application embodiment further includes: updating the target loss data, specifically including:

[0122] Step 401: Perform prosodic encoding on the speech sample data using a preset prosodic encoding model to obtain the sample prosodic vector;

[0123] Step 402: Speaker identification is performed on the speech sample data to obtain the sample speaker vector;

[0124] Step 403: Decode the speech data based on the second content vector, the sample speaker vector, and the sample prosody vector to obtain the speech reconstruction data;

[0125] Step 404: Perform loss reconstruction based on speech sample data and speech reconstruction data to obtain reconstruction loss data;

[0126] Step 405: Update the target loss data based on the reconstructed loss data.

[0127] In step 401, the speech sample data can be input into the prosodic coding model for prosodic encoding to obtain sample prosodic vectors. The prosodic coding model is a neural network model used for prosodic encoding of speech. In one embodiment, the prosodic coding model can be designed based on the structure of ECAPA-TDNN.

[0128] In one embodiment, the prosodic coding model includes an acoustic feature extraction sub-model and a prosodic feature extraction sub-model. (Refer to...) Figure 5 Step 401 includes:

[0129] Step 501: Extract acoustic features from the speech sample data using the acoustic feature extraction sub-model to obtain the initial acoustic vector;

[0130] At step 502, the prosody feature extraction sub-model is used to extract prosody features from the initial acoustic vector to obtain an acoustic prosody vector.

[0131] At step 503, the initial acoustic vector and the acoustic prosody vector are fused to obtain a sample prosody vector.

[0132] Specifically, the network structure of the acoustic feature extraction sub-model is basically the same as that of the ECAPA-TDNN network, and is mainly used for acoustic feature extraction to obtain an initial acoustic vector. Considering that the emotional prosody in the initial acoustic vector is difficult to fully reflect, the embodiment introduces a prosody feature extraction sub-model. The initial acoustic vector is input into the prosody feature extraction sub-model for prosody feature extraction to obtain an acoustic prosody vector. The acoustic prosody vector is used to represent the emotional prosody information in the speech sample data. The emotional prosody information includes fundamental frequency, energy, glottal period, etc. Then, the initial acoustic vector and the acoustic prosody vector are fused, for example, the vector dimensions of the initial acoustic vector and the acoustic prosody vector are consistent, and the fusion can be performed in a concatenation manner to obtain a sample prosody vector. The sample prosody vector is a feature representation vector with emotional prosody information.

[0133] The above embodiment has the benefit that acoustic feature extraction is performed first, then prosody feature extraction is performed, and the obtained initial acoustic vector and acoustic prosody vector are fused to obtain a sample prosody vector, thereby improving the accuracy of obtaining the sample prosody vector.

[0134] At step 402, the speaker recognition model can be used to perform speaker recognition on the speech sample data to obtain a sample speaker vector. Voiceprint recognition is a biometric recognition technology that verifies identity by analyzing the speech characteristics of an individual. Voiceprint recognition models usually use deep learning techniques such as convolutional neural networks (CNN) or recurrent neural networks (RNN) to extract and recognize sound features.

[0135] At step 403, the decoder can be used to decode the second content vector, the sample speaker vector, and the sample prosody vector to obtain speech reconstruction data. Decoder (Decoder) is a neural network structure in deep learning, which is usually used in combination with encoder (Encoder) to convert the high-dimensional representation learned by the encoder into the target output. In this embodiment, the decoder can be used to convert the second content vector, the sample speaker vector, and the prosody vector into speech reconstruction data.

[0136] In one example, step 403 includes: splicing the second content vector, the sample speaker vector, and the sample prosody vector to obtain a target spliced vector; and inputting the target spliced vector into the decoder for decoding to obtain the speech reconstruction data.

[0137] In another example, the step 403 comprises: performing vector summation on the second content vector, the sample speaker vector and the sample prosody vector to obtain a target summation vector; inputting the target summation vector into the decoder to obtain the speech reconstruction data.

[0138] In the step 404, the speech sample data and the speech reconstruction data can be loss reconstructed by a preset loss function to obtain reconstruction loss data. The loss function can be selected from an absolute error function, or a mean square error loss function, etc.

[0139] In the step 405, the target loss data is updated according to the reconstruction loss data.

[0140] In an embodiment, the step 405 can comprise: adding the reconstruction loss data and the content distillation loss data to obtain the target loss data. The target loss data can be shown in the following formula:

[0141] L = L distill + L recon Formula (2).

[0142] In the formula (2), L represents the target loss data, L distill represents the content distillation loss data, and L recon represents the reconstruction loss data.

[0143] The embodiment of the steps 401 to 405 has the benefit that, by updating the target loss data, the influence of the speech reconstruction data on the target loss data is embodied, and the accuracy of the target loss data is further improved.

[0144] In an embodiment, updating the target loss data further comprises:

[0145] multiplying the preset content distillation weight and the target loss data to obtain first loss data;

[0146] adding the first loss data and the reconstruction loss data to obtain target fusion loss data, and updating the target loss data to the target fusion loss data.

[0147] It should be noted that, before the target loss data is updated, the target loss data is the same as the content distillation loss data, that is, the first loss data is actually equal to the multiplication of the content distillation weight and the content distillation loss data.

[0148] The calculation process of the target loss data is shown in the following formula:

[0149] L = L recon + μ * L distill Formula (3).

[0150] In formula (3), L represents target loss data, L recon represents reconstruction loss data, L distill represents content distillation loss data. μ represents a content distillation weight, and μ is greater than 0 and less than 1.

[0151] The above embodiment has the benefit that, when updating the target loss data, the importance of the speech reconstruction loss data is fully reflected, and the accuracy of speech synthesis can be ensured.

[0152] In an embodiment, after step 401, referring to Figure 6 , the speech synthesis method further comprises:

[0153] Step 601: performing emotion recognition on the sample prosody vector to obtain a sample emotion category;

[0154] Step 602: performing emotion recognition loss calculation on the sample emotion category and a preset sample emotion label to obtain prosody loss data;

[0155] Step 603: performing parameter adjustment on the prosody coding model according to the prosody loss data to obtain a target prosody coding model.

[0156] In step 601, the sample emotion category can be obtained by performing emotion recognition on the sample prosody vector through an emotion recognition model. The emotion recognition model is a deep learning model designed to recognize emotions or emotional information contained in speech data. The emotion recognition model can be a speech emotion recognition model (SER).

[0157] In step 602, the prosody loss data can be obtained by performing emotion recognition loss calculation on the sample emotion category and the sample emotion label through a preset loss function. The loss function can be an absolute error function or a mean square error loss function.

[0158] In step 603, the prosody coding model is adjusted according to the prosody loss data, and during the parameter adjustment process, the prosody loss data is compared with a preset first loss threshold. If the prosody loss data is greater than the first loss threshold, the parameter adjustment of the prosody coding model is continued until the prosody loss data is less than or equal to the first loss threshold, and then the parameter adjustment is stopped to obtain the target prosody coding model. The target prosody coding model can be used to perform prosody coding on the target speech data appearing below to obtain a target prosody vector.

[0159] The above embodiment has the benefit that the sample emotion category can be identified based on the sample prosody information, and the prosody loss data can be calculated based on the sample emotion category, so that the target prosody encoder can be obtained by adjusting the parameters of the prosody encoder based on the prosody loss data, and the encoding performance of the target prosody encoder can be improved.

[0160] In step 106 of some embodiments, the initial content encoding student model can be adjusted in parameters according to the target loss data to obtain a target content encoding student model.

[0161] Referring to the above, the initial content encoding student model can be adjusted in parameters to obtain a target content encoding student model using the target loss data shown in any one of formula (1), formula (2), and formula (3). The target content encoding student model is the same as the initial content encoding student model in model structure, but different in model parameters. In the parameter adjustment process, the target loss data is compared with a preset second loss threshold. If the target loss data is greater than the second loss threshold, the parameter adjustment of the initial content encoding student model is continued until the target loss data is less than or equal to the second loss threshold, and then the parameter adjustment is stopped to obtain the target content encoding student model.

[0162] In step 107 of some embodiments, target speech data is obtained. The target speech data is obtained in a manner similar to the process of obtaining the speech sample data described above, which will not be repeated here. The target speech data is used to represent the speech uttered by the target speaker according to the target speech content. The target speech content is used to represent the text information of the target speech data. It should be noted that the target speech data can be Chinese speech, English speech, etc.

[0163] In step 108 of some embodiments, the target content vector can be obtained by content encoding the target speech data by the target content encoding student model. Since the target content encoding student model is obtained by adjusting the parameters of the initial content encoding student model in steps 101 to 106, it has high content encoding capability, so that the target content vector can well represent the content features of the target speech content.

[0164] In step 109 of some embodiments, the target content vector, the preset target speaker vector and the preset target prosody vector can be decoded by a decoder to obtain target synthesized speech data. The decoder has been explained above and will not be repeated here. The target speaker vector can be obtained by performing voiceprint coding on the speaker speech data by a voiceprint recognition model. The target prosody vector can be obtained by performing prosody coding on the emotional speech data by a target prosody coding model. The speaker speech data is speech data used to provide speaker information. The emotional speech data is speech data used to provide emotional prosody information. The speaker speech data and the emotional speech data can be the same speech data or different speech data.

[0165] Please refer to Figure 7 In an embodiment, the speech synthesis method comprises the following implementation steps:

[0166] (1) obtaining an acoustic feature representation sequence of the speech sample data to obtain an initial acoustic feature representation sequence;

[0167] (2) performing mask processing on the initial acoustic feature representation sequence to obtain a masked acoustic feature representation sequence;

[0168] (3) performing predictive coding on the masked acoustic feature representation sequence by a speech recognition pre-training sub-model to obtain a first predictive coding sequence, and performing instance normalization on the first predictive coding sequence by a normalization layer to obtain a first content vector;

[0169] (4) performing predictive coding on the masked acoustic feature representation sequence by an attention sub-model to obtain a second predictive coding sequence, and performing content representation on the second predictive coding sequence by a content representation learning sub-model to obtain a second content vector;

[0170] (5) performing content distillation loss calculation according to the first content vector and the second content vector to obtain content distillation loss data L distill ;

[0171] (6) performing prosody coding on the speech sample data by a prosody coding model to obtain a sample prosody vector;

[0172] (7) performing speaker recognition on the speech sample data to obtain a sample speaker vector;

[0173] (8) decoding the second content vector, the sample speaker vector and the sample prosody vector by a decoder to obtain speech reconstruction data;

[0174] (9) performing loss reconstruction according to the speech sample data and the speech reconstruction data to obtain reconstruction loss data L recon ;

[0175] (10) summing the reconstruction loss data and the content distillation loss data to obtain target loss data;

[0176] (11) adjusting parameters of the initial content encoding student model according to the target loss to obtain a target content encoding student model;

[0177] (12) performing emotion recognition on the sample prosody vector by an emotion recognition model to obtain a sample emotion category;

[0178] (13) performing emotion recognition loss calculation according to the sample emotion category and a preset sample emotion label to obtain prosody loss data L p ;

[0179] (14) adjusting parameters of the prosody encoding model according to the prosody loss data L p to obtain a target prosody encoding model.

[0180] After obtaining the target content encoding student model and the target prosody encoding model through the above steps (11) to (14), the target content vector can be obtained by performing content encoding on the target speech data by the target content encoding student model, and the target prosody vector can be obtained by performing prosody encoding on the emotional speech data by the target prosody encoding model, and the speaker vector can also be obtained by performing voiceprint recognition on the speaker speech data by the voiceprint recognition model. Then, the target synthesized speech data is obtained by decoding the target content vector, the target speaker vector and the target prosody vector by the decoder.

[0181] The embodiments of the present application have at least the following advantages: 1. More natural speech expression: through emotion prosody modeling and conversion, the speech output of the intelligent customer service is more expressive and natural, which can simulate the speech communication characteristics of human customer service and improve the communication experience of users and intelligent customer service. 2. Improve user experience: introduce emotion prosody features, so that the bank intelligent customer service system can recognize and adapt to the emotional needs of users, better meet the communication needs of users in different scenarios, and improve customer satisfaction.

[0182] Please refer to Figure 8 , the embodiments of the present application also provide a speech synthesis device, which can implement the above speech synthesis method, Figure 8A module structure block diagram of a speech synthesis device provided by an embodiment of the present application is provided, and the device comprises: a speech acquisition module 801, a mask processing module 802, a content encoding module 803, a loss calculation module 804, a parameter adjustment module 805, and a speech synthesis module 806. The speech acquisition module 801 is configured to acquire an acoustic feature representation sequence of speech sample data, and obtain an initial acoustic feature representation sequence. The mask processing module 802 is configured to perform mask processing on the initial acoustic feature representation sequence, and obtain a mask acoustic feature representation sequence. The content encoding module 803 is configured to perform content encoding on the mask acoustic feature representation sequence by using a preset content encoding teacher model, and obtain a first content vector. The content encoding module 803 is further configured to perform content encoding on the mask acoustic feature representation sequence by using a preset initial content encoding student model, and obtain a second content vector. The parameter quantity of the content encoding teacher model is greater than the parameter quantity of the initial content encoding student model. The loss calculation module 804 is configured to perform content distillation loss calculation according to the first content vector and the second content vector, and obtain target loss data. The parameter adjustment module 805 is configured to perform parameter adjustment on the initial content encoding student model according to the target loss data, and obtain a target content encoding student model. The speech synthesis module 806 is configured to: acquire target speech data; perform content encoding on the target speech data by using the target content encoding student model, and obtain a target content vector; and perform decoding according to the target content vector, a preset target speaker vector, and a preset target prosody vector, and obtain target synthesized speech data.

[0183] In an embodiment, the speech synthesis device further comprises a loss updating module configured to update the target loss data.

[0184] In another embodiment, the speech synthesis device further comprises a prosody encoding model adjustment module configured to: perform emotion recognition on the sample prosody vector, and obtain a sample emotion category; perform emotion recognition loss calculation according to the sample emotion category and a preset sample emotion label, and obtain prosody loss data; and perform parameter adjustment on a prosody encoding model according to the prosody loss data, and obtain a target prosody encoding model.

[0185] It should be noted that the specific implementation of the speech synthesis device is basically the same as the specific embodiments of the speech synthesis method described above, and will not be described here again.

[0186] An embodiment of the present application further provides an electronic device, which comprises a memory, a processor, a program stored in the memory and capable of running on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the speech synthesis method described above. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0187] Please refer to Figure 9 ,Figure 9 An electronic device of another embodiment is illustrated, and the electronic device includes:

[0188] The processor 901 can be implemented with a central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and the like, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0189] The memory 902 can be implemented with a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), and the like. The memory 902 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the voice synthesis method of the embodiments of the present application.

[0190] The input / output interface 903 is configured to implement information input and output.

[0191] The communication interface 904 is configured to implement the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable, and the like) or a wireless manner (for example, a mobile network, a WI-FI, a Bluetooth, and the like).

[0192] The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0193] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between the device.

[0194] The embodiments of the present application further provide a storage medium, the storage medium is a computer readable storage medium, and is configured to store computer readable information. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the voice synthesis method.

[0195] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely from the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0196] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0197] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0198] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0199] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0200] The terms "first", "second", "third", "fourth" and the like used in the specification of the present application and the above-described drawings, if any, are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0201] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0202] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0203] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0204] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0205] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for making an electronic device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0206] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A speech synthesis method characterized by, The method comprises: obtaining an acoustic feature representation sequence of speech sample data, to obtain an initial acoustic feature representation sequence; mask processing the initial acoustic feature representation sequence to obtain a masked acoustic feature representation sequence; content encoding the masked acoustic feature representation sequence through a preset content encoding teacher model to obtain a first content vector; content encoding the masked acoustic feature representation sequence through a preset initial content encoding student model to obtain a second content vector; wherein the parameter quantity of the content encoding teacher model is greater than the parameter quantity of the initial content encoding student model; content distillation loss calculation according to the first content vector and the second content vector to obtain target loss data; updating the target loss data, specifically including: prosodic coding the speech sample data through a preset prosodic coding model to obtain a sample prosodic vector; speaker recognition of the speech sample data to obtain a sample speaker vector; decoding according to the second content vector, the sample speaker vector and the sample prosodic vector to obtain speech reconstruction data; loss reconstruction according to the speech sample data and the speech reconstruction data to obtain reconstruction loss data; updating the target loss data according to the reconstruction loss data; parameter adjustment of the initial content encoding student model according to the target loss data to obtain a target content encoding student model; obtaining target speech data; content encoding the target speech data through the target content encoding student model to obtain a target content vector; decoding according to the target content vector, a preset target speaker vector and a preset target prosodic vector to obtain target synthesized speech data.

2. The method of claim 1, wherein, The content encoding teacher model comprises a speech recognition pre-training sub-model and a normalization layer; The content encoding the masked acoustic feature representation sequence through the preset content encoding teacher model to obtain the first content vector comprises: predictive encoding the masked acoustic feature representation sequence through the speech recognition pre-training sub-model to obtain a first predictive encoding sequence; instance normalization of the first predictive encoding sequence through the normalization layer to obtain the first content vector.

3. The method of claim 1, wherein, The initial content encoding student model comprises an attention sub-model and a content representation learning sub-model; The content encoding the masked acoustic feature representation sequence through the preset initial content encoding student model to obtain the second content vector comprises: predictive encoding the masked acoustic feature representation sequence through the attention sub-model to obtain a second predictive encoding sequence; content representation of the second predictive encoding sequence through the content representation learning sub-model to obtain the second content vector.

4. The method according to any one of claims 1 to 3, characterized in that, The prosodic coding model comprises an acoustic feature extraction sub-model and a prosodic feature extraction sub-model; The prosodic coding the speech sample data through the preset prosodic coding model to obtain the sample prosodic vector; acoustic feature extraction of the speech sample data through the acoustic feature extraction sub-model to obtain an initial acoustic vector; extracting prosodic features from the initial acoustic feature sequence to obtain an acoustic prosodic feature sequence; fusing the initial acoustic feature sequence and the acoustic prosodic feature sequence to obtain the sample prosodic feature sequence.

5. The method according to any one of claims 1 to 3, characterized in that, After the prosodic coding of the speech sample data is performed by the preset prosodic coding model to obtain the sample prosodic feature sequence, the method further comprises: performing emotion recognition on the sample prosodic feature sequence to obtain a sample emotion category; performing emotion recognition loss calculation according to the sample emotion category and a preset sample emotion label to obtain prosodic loss data; adjusting parameters of the prosodic coding model according to the prosodic loss data to obtain a target prosodic coding model; wherein the target prosodic coding model is used to perform prosodic coding on the target speech data to obtain the target prosodic feature sequence.

6. The method according to any one of claims 1 to 3, characterized in that, The content distillation loss calculation according to the first content vector and the second content vector to obtain target loss data comprises: performing difference calculation according to the first content vector and the second content vector to obtain a content coding error; performing cosine calculation on the content coding error to obtain cosine data; performing logarithmic calculation on the cosine data to obtain the target loss data.

7. A speech synthesis apparatus characterized by comprising: The device comprises: a speech acquisition module configured to acquire an acoustic feature representation sequence of speech sample data to obtain an initial acoustic feature representation sequence; a mask processing module configured to perform mask processing on the initial acoustic feature representation sequence to obtain a masked acoustic feature representation sequence; a content coding module configured to perform content coding on the masked acoustic feature representation sequence by a preset content coding teacher model to obtain a first content vector; the content coding module is further configured to perform content coding on the masked acoustic feature representation sequence by a preset initial content coding student model to obtain a second content vector; wherein a parameter quantity of the content coding teacher model is greater than a parameter quantity of the initial content coding student model; a loss calculation module configured to perform content distillation loss calculation according to the first content vector and the second content vector to obtain target loss data; a loss update module configured to update the target loss data, specifically comprising: performing prosodic coding on the speech sample data by a preset prosodic coding model to obtain a sample prosodic feature sequence; performing speaker recognition on the speech sample data to obtain a sample speaker vector; performing decoding according to the second content vector, the sample speaker vector and the sample prosodic feature sequence to obtain speech reconstruction data; performing loss reconstruction according to the speech sample data and the speech reconstruction data to obtain reconstruction loss data; updating the target loss data according to the reconstruction loss data; a parameter adjustment module configured to adjust parameters of the initial content coding student model according to the target loss data to obtain a target content coding student model; a speech synthesis module configured to: acquire target speech data; perform content coding on the target speech data by the target content coding student model to obtain a target content vector. According to the target content vector, a preset target speaker vector and a preset target prosody vector, target synthesized speech data is obtained.

8. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the speech synthesis method of any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the speech synthesis method of any one of claims 1 to 6.

Images