Speech synthesis method, speech synthesis device, electronic equipment and storage medium

Through the acoustic feature representation sequence processing of speech sample data and parameter adjustment of content encoding student model, combined with pronunciation coding and emotion recognition, the problem of low accuracy and efficiency of speech synthesis content in the prior art is solved, and high-quality speech synthesis is achieved , improves the user experience.

CN118942442AActive Publication Date: 2024-11-12PING AN TECH (SHENZHEN) CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411118568.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-12
Estimated Expiration
2044-08-14

AI Technical Summary

Technical Problem

The existing speech synthesis technology has problems of low content accuracy and low efficiency in financial customer service scenarios. Especially, although the self-encoder with complex network structure has good prediction performance, the prediction efficiency is low, resulting in the content accuracy of synthesized speech. and emotional performance.

Method used

A pronunciation method is proposed, by obtaining the acoustic feature representation sequence of speech sample data, mask processing and content encoding, and using the content encoding teacher model and the initial content encoding student model to calculate content distillation loss, adjust parameters to improve content encoding students The accuracy and efficiency of the model, combined with prosthetic coding and emotion recognition, generate high-quality target synthetic speech data.

Benefits of technology

It achieves the improvement of content accuracy and emotional performance of speech synthesis, and at the same time improves the efficiency of speech synthesis, and can generate more natural and smooth voice output, reducing communication barriers between users and intelligent customer service.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942442A_ABST
    Figure CN118942442A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a speech synthesis method, a speech synthesis device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence and the field of financial science and technology. The method comprises the following steps: respectively carrying out content coding on a mask acoustic feature representation sequence through a content coding teacher model and an initial content coding student model to obtain a first content vector and a second content vector; calculating target loss data according to the first content vector and the second content vector; performing parameter adjustment on the initial content coding student model according to the target loss data to obtain a target content coding student model; performing content coding on the target voice data through the target content coding student model to obtain a target content vector; and decoding according to the target content vector, the target speaker vector and the target rhythm vector to obtain target synthetic voice data. According to the embodiment of the invention, the content accuracy and emotion of speech synthesis can be ensured, and the speech synthesis efficiency is also improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence technology and financial technology, and in particular to a speech synthesis method, a speech synthesis device, an electronic device and a storage medium. Background Art

[0002] Currently, voice conversion (also known as speaker conversion) in speech synthesis is widely used in financial customer service scenarios. Voice conversion aims to convert the voice of one person into that of another person without changing the content of the speech. The prior art proposes a method based on an autoencoder to decouple speech into two parts: timbre and content, in which the content encoder uses a bottleneck layer to constrain content features from the channel level. Since the size of the data set used in training the content encoder is limited, and the prediction performance of the autoencoder with a simple network structure is limited, the content encoding accuracy is low, resulting in poor content accuracy in speech synthesis. In addition, although the prediction performance of the autoencoder with a complex network structure is good, the prediction efficiency is low.

[0003] Therefore, how to provide a speech synthesis method that can ensure the accuracy of speech synthesis content and improve speech synthesis efficiency has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, which can ensure the accuracy and emotion of the content of speech synthesis and improve the efficiency of speech synthesis.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a speech synthesis method, the method comprising:

[0006] Acquiring an acoustic feature representation sequence of the speech sample data to obtain an initial acoustic feature representation sequence;

[0007] Performing masking processing on the initial acoustic feature representation sequence to obtain a masked acoustic feature representation sequence;

[0008] Content encoding the masked acoustic feature representation sequence is performed using a preset content encoding teacher model to obtain a first content vector;

[0009] Content encoding the masked acoustic feature representation sequence is performed using a preset initial content encoding student model to obtain a second content vector; wherein the parameter amount of the content encoding teacher model is greater than the parameter amount of the initial content encoding student model;

[0010] Calculate content distillation loss according to the first content vector and the second content vector to obtain target loss data;

[0011] Adjusting parameters of the initial content encoding student model according to the target loss data to obtain a target content encoding student model;

[0012] Obtain target voice data;

[0013] Content encoding is performed on the target speech data by using the target content encoding student model to obtain a target content vector;

[0014] Decoding is performed according to the target content vector, the preset target speaker vector and the preset target prosody vector to obtain target synthesized speech data.

[0015] In some embodiments, the content encoding teacher model includes a speech recognition pre-trained sub-model and a normalization layer;

[0016] The step of performing content encoding on the mask acoustic feature representation sequence by using a preset content encoding teacher model to obtain a first content vector includes:

[0017] Performing predictive coding on the masked acoustic feature representation sequence through the speech recognition pre-trained sub-model to obtain a first predictive coding sequence;

[0018] The first predicted coding sequence is instance-normalized by the normalization layer to obtain the first content vector.

[0019] In some embodiments, the initial content encoding student model includes an attention sub-model and a content representation learning sub-model;

[0020] The step of performing content encoding on the mask acoustic feature representation sequence by using a preset initial content encoding student model to obtain a second content vector includes:

[0021] Performing predictive coding on the masked acoustic feature representation sequence through the attention sub-model to obtain a second predictive coding sequence;

[0022] The second predicted coding sequence is content represented by the content representation learning sub-model to obtain the second content vector.

[0023] In some embodiments, before adjusting the parameters of the initial content encoding student model according to the target loss data to obtain the target content encoding student model, the method further includes: updating the target loss data, specifically including:

[0024] Prosody encoding the speech sample data is performed using a preset prosody encoding model to obtain a sample prosody vector;

[0025] Performing speaker recognition on the speech sample data to obtain a sample speaker vector;

[0026] Decoding is performed according to the second content vector, the sample speaker vector and the sample prosody vector to obtain speech reconstruction data;

[0027] Perform loss reconstruction according to the speech sample data and the speech reconstruction data to obtain reconstructed loss data;

[0028] The target loss data is updated according to the reconstructed loss data.

[0029] In some embodiments, the prosodic coding model includes an acoustic feature extraction sub-model and a prosodic feature extraction sub-model;

[0030] The prosody encoding of the speech sample data is performed using a preset prosody encoding model to obtain a sample prosody vector;

[0031] Extracting acoustic features from the speech sample data using the acoustic feature extraction submodel to obtain an initial acoustic vector;

[0032] Extracting rhythmic features from the initial acoustic vector using the rhythmic feature extraction sub-model to obtain an acoustic rhythmic vector;

[0033] The initial acoustic vector and the acoustic prosody vector are fused to obtain the sample prosody vector.

[0034] In some embodiments, after prosody encoding the speech sample data using a preset prosody encoding model to obtain a sample prosody vector, the method further includes:

[0035] Performing emotion recognition on the sample rhythm vector to obtain a sample emotion category;

[0036] Emotion recognition loss is calculated based on the sample emotion category and the preset sample emotion label to obtain rhythm loss data;

[0037] The parameters of the prosody coding model are adjusted according to the prosody loss data to obtain a target prosody coding model; wherein the target prosody coding model is used to perform prosody coding on the target speech data to obtain the target prosody vector.

[0038] In some embodiments, performing content distillation loss calculation according to the first content vector and the second content vector to obtain target loss data includes:

[0039] Performing a difference calculation based on the first content vector and the second content vector to obtain a content coding error;

[0040] Performing hyperbolic cosine calculation on the content coding error to obtain hyperbolic cosine data;

[0041] Perform logarithmic calculation on the hyperbolic cosine data to obtain the target loss data.

[0042] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech synthesis device, the device comprising:

[0043] A speech acquisition module, used to acquire an acoustic feature representation sequence of speech sample data to obtain an initial acoustic feature representation sequence;

[0044] A mask processing module, used for performing mask processing on the initial acoustic feature representation sequence to obtain a masked acoustic feature representation sequence;

[0045] A content encoding module, configured to perform content encoding on the masked acoustic feature representation sequence through a preset content encoding teacher model to obtain a first content vector;

[0046] The content encoding module is further used to perform content encoding on the mask acoustic feature representation sequence through a preset initial content encoding student model to obtain a second content vector; wherein the parameter amount of the content encoding teacher model is greater than the parameter amount of the initial content encoding student model;

[0047] a loss calculation module, configured to perform content distillation loss calculation according to the first content vector and the second content vector to obtain target loss data;

[0048] A parameter adjustment module, used for adjusting the parameters of the initial content encoding student model according to the target loss data to obtain a target content encoding student model;

[0049] Speech synthesis module for:

[0050] Obtain target voice data;

[0051] Content encoding is performed on the target speech data by using the target content encoding student model to obtain a target content vector;

[0052] Decoding is performed according to the target content vector, the preset target speaker vector and the preset target prosody vector to obtain target synthesized speech data.

[0053] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the speech synthesis method described in the first aspect when executing the computer program.

[0054] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the speech synthesis method described in the first aspect.

[0055] The speech synthesis method, speech synthesis device, electronic device and storage medium proposed in the present application first obtain the acoustic feature representation sequence of speech sample data to obtain the initial acoustic feature representation sequence. Secondly, the initial acoustic feature representation sequence is masked to obtain the masked acoustic feature representation sequence. Then, the masked acoustic feature representation sequence is content encoded by the content encoding teacher model and the initial content encoding student model to obtain the first content vector and the second content vector. Then, the target loss data is calculated according to the first content vector and the second content vector, and the parameters of the initial content encoding student model are adjusted according to the target loss data to obtain the target content encoding student model. The second content vector output by the target content encoding student model is almost the same as the first content vector output by the content encoding teacher model, that is, the target content encoding student model has a coding accuracy similar to that of the content encoding teacher model, and the number of parameters of the target content encoding student model is small, so the content encoding efficiency is high. Then, the target content encoding student model is used to content encode the target speech data, and a target content vector with higher accuracy can be obtained, so that the quality of the target synthesized speech data obtained based on the target content vector, the target speaker vector and the target prosody vector is better and more efficient. In summary, this embodiment can ensure the accuracy of speech synthesis content and improve speech synthesis efficiency.

[0056] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of a speech synthesis method provided in an embodiment of the present application;

[0058] Figure 2 yes Figure 1 Flowchart of step 103 in

[0059] Figure 3 yes Figure 1 Flowchart of step 104 in;

[0060] Figure 4 is a flow chart of a speech synthesis method provided by another embodiment of the present application;

[0061] Figure 5 yes Figure 4 Flow chart of step 401 in;

[0062] Figure 6 is a flow chart of a speech synthesis method provided by another embodiment of the present application;

[0063] Figure 7 is an overall schematic diagram of the speech synthesis method provided in an embodiment of the present application;

[0064] Figure 8 is a block diagram of the module structure of the speech synthesis device provided in an embodiment of the present application;

[0065] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0067] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0069] First, some nouns involved in this application are analyzed:

[0070] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0071] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information image processing, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0072] Speech conversion technology: refers to the technology of converting original speech data into target speech data while keeping the semantic content unchanged, wherein the original speech data is the speech emitted by the original speaker, and the target speech data is the speech emitted by the target speaker. That is, the original speech emitted by the original speaker is converted into the target speech emitted by the target speaker with the same semantics through speech conversion technology.

[0073] At present, financial customer service has become an indispensable part of financial institutions, and financial customer service is becoming more and more automated and intelligent. Among them, voice conversion technology is particularly widely used in financial customer service. Voice conversion aims to convert the voice of one person into another person without changing the content of the speech, only changing the identity of the speaker.

[0074] However, the content accuracy of the synthesized speech of the voice customer service system of the related art is low, which affects the user experience. For example, when guiding users, the speech content in the synthesized speech is not clearly expressed, and the user cannot be effectively guided. In addition, the synthesized speech of the voice customer system of the related art often lacks emotion, which also affects the user experience. For example, when soothing the user's emotions, it is difficult for the intelligent voice customer service to show natural emotions and cannot effectively soothe the user. Although the related art has a technology that extracts rhythmic information from the speech to improve the emotion of the synthesized speech, it ignores the loss of content information in the synthesized speech, resulting in the final synthesized speech. Although there are emotions, the content accuracy is still low. For example, there are also some speech synthesis models with complex network structures in the related art. Although the content accuracy is guaranteed, it greatly affects the speech synthesis efficiency.

[0075] Based on this, an embodiment of the present application proposes a speech synthesis method, which can ensure the accuracy of the content of speech synthesis and improve the efficiency of speech synthesis, and can also make the synthesized speech have emotion.

[0076] The speech synthesis method provided in the embodiment of the present application can be applied to the terminal and the server side, and can also be software running on the server side. The server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech synthesis method, etc., but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0078] The embodiments of the present application provide a speech synthesis method and a speech synthesis device, an electronic device, and a storage medium, which are specifically illustrated by the following embodiments. First, the speech synthesis method in the embodiments of the present application is described.

[0079] It should be noted that in each specific implementation method of the present application, when it comes to performing relevant processing based on the user's voice data or other data related to the user's identity or characteristics, the user's permission or consent must be obtained first, and the collection, use and processing of such data will comply with relevant laws, regulations and standards.

[0080] Reference Figure 1 , Figure 1 It is an optional flowchart of the speech synthesis method provided in an embodiment of the present application, which may include but is not limited to steps 101 to 109.

[0081] Step 101, obtaining an acoustic feature representation sequence of speech sample data to obtain an initial acoustic feature representation sequence;

[0082] Step 102, performing mask processing on the initial acoustic feature representation sequence to obtain a masked acoustic feature representation sequence;

[0083] Step 103, content encoding the masked acoustic feature representation sequence through a preset content encoding teacher model to obtain a first content vector;

[0084] Step 104, content encoding the masked acoustic feature representation sequence through a preset initial content encoding student model to obtain a second content vector; wherein the parameter amount of the content encoding teacher model is greater than the parameter amount of the initial content encoding student model;

[0085] Step 105, performing content distillation loss calculation based on the first content vector and the second content vector to obtain target loss data;

[0086] Step 106, adjusting parameters of the initial content encoding student model according to the target loss data to obtain a target content encoding student model;

[0087] Step 107, obtaining target voice data;

[0088] Step 108, content encoding the target speech data through the target content encoding student model to obtain a target content vector;

[0089] Step 109 , decoding is performed according to the target content vector, the preset target speaker vector and the preset target prosody vector to obtain target synthesized speech data.

[0090] In steps 101 to 109 shown in the embodiment of the present application, target loss data is calculated based on the first content vector and the second content vector, and the parameters of the initial content coding student model are adjusted based on the target loss data to obtain a target content coding student model. The second content vector output by the target content coding student model is almost the same as the first content vector output by the content coding teacher model, that is, the target content coding student model has a coding accuracy similar to that of the content coding teacher model, and the target content coding student model has a smaller number of parameters, so the content coding efficiency is higher. Then, the target content coding student model is used to perform content encoding on the target speech data, and a target content vector with higher accuracy can be obtained, so that the quality of the target synthesized speech data obtained based on the target content vector, the target speaker vector and the target rhythm vector is better and more efficient. In summary, this embodiment can ensure the content accuracy of speech synthesis and improve the efficiency of speech synthesis.

[0091] When the embodiments of the present application are applied to scenarios such as telephone customer service and online customer service in the field of financial technology, it has highly natural and smooth voice output, improves user experience, and reduces communication barriers between users and intelligent customer service.

[0092] In step 101 of some embodiments, an acoustic feature representation sequence of speech sample data is obtained to obtain an initial acoustic feature representation sequence. The speech sample data is used to represent the speech uttered by the original speaker according to the original speech content. The initial acoustic feature representation sequence refers to a feature parameter sequence used to describe the speech sample data. Common acoustic features include: (1) Short-time Energy: describes the intensity of the speech signal in a short time. (2) Short-time Zero Crossing Rate: describes the rate of change of the speech signal waveform. (3) Mel-Frequency Cepstrum Coefficients (MFCCs): is a feature representation form commonly used in speech recognition, designed based on the perception characteristics of the human ear to sound. (4) Linear Prediction Coding Coefficients (LPCs): used to model the spectral envelope of the speech signal. (5) Mel-Spectrogram: describes the spectral representation of speech signals at Mel frequencies.

[0093] Specifically, when the speech synthesis method is applied to a terminal, the speech sample data can be obtained by recording, Bluetooth transmission, wired transmission, or downloading. When the speech sample data is obtained by recording, the terminal is correspondingly configured with a microphone, and the microphone is used to collect audio to achieve the recording of the speech sample data. When the speech synthesis method is applied to a server, the speech sample data can be uploaded to the server by the terminal, or downloaded by the server from other servers or databases.

[0094] When distinguishing speakers, you can distinguish them by timbre. This is because different people have different vocal cords and different ways of vibrating their vocal cords, which leads to different parameters such as the vibration frequency and vibration amplitude of the sound, resulting in different timbre for different speakers. The original voice content is used to represent the text information of the voice sample data. The voice sample data can be voice composed of Chinese or English.

[0095] In one embodiment, step 101 includes: obtaining speech sample data; extracting acoustic feature representations from the speech sample data using a clustering algorithm to obtain an initial acoustic feature representation sequence. For example, after clustering the speech sample data x using a clustering algorithm, an initial acoustic feature representation sequence [x1, x2, ..., xt] can be obtained. t represents the length of the sequence.

[0096] It should be noted that, in addition to using clustering algorithms, other methods can also be used to extract acoustic feature representations from speech sample data, such as using a neural network model.

[0097] In step 102 of some embodiments, the initial acoustic feature representation sequence is masked to obtain a masked acoustic feature representation sequence. Masking refers to a technique in the field of signal processing that uses a mask to process a signal. A mask generally refers to a binary sequence that indicates which parts of a signal need to be processed or retained. For example, for the initial acoustic feature representation sequence [x1, x2,…, xt], after masking, a masked acoustic feature representation sequence [x1, x2, [mask], x4, [mask],…, Xt] can be obtained. The masked acoustic feature representation sequence is a representation sequence with a mask.

[0098] In step 103 of some embodiments, the masked acoustic feature representation sequence is content encoded by a preset content encoding teacher model to obtain a first content vector. The content encoding teacher model is a neural network model with good content encoding effect but a large number of parameters and slow running speed. In this embodiment, the parameters of the content encoding teacher model are fixed, and the output first content vector can be used as knowledge to supervise the training of the initial content encoding student model.

[0099] In some embodiments, the content encoding teacher model includes a speech recognition pre-trained sub-model and a normalization layer. Figure 2 , step 103 comprises:

[0100] Step 201, predictively encode the masked acoustic feature representation sequence through a speech recognition pre-trained sub-model to obtain a first predictive coding sequence;

[0101] Step 202: normalize the first predicted coding sequence by an instance through a normalization layer to obtain a first content vector.

[0102] Specifically, the speech recognition pre-training sub-model is a neural network model combined with a mask prediction mechanism. The speech recognition pre-training sub-model can predict the masked content in the masked acoustic feature representation sequence, and can learn the unmasked acoustic feature representation and the long-range temporal structure of the speech sample data. The masked acoustic feature representation sequence is input into the speech recognition pre-training sub-model for predictive coding, and a first predicted coding sequence can be obtained. The first predicted coding sequence includes content information and also includes speaker information. Therefore, it is necessary to input the first predicted coding sequence into the normalization layer for instance normalization, and a first content vector can be obtained. The first content vector can represent the speaking content in the speech sample data and does not include the speaker information. The speech recognition pre-training sub-model can be a HuBERT (Hidden-Unit BERT) model. The normalization layer is specifically an IN (Instance Normalization) layer.

[0103] The benefit of the above embodiment is that predictive coding is performed first and then instance normalization is performed, which improves the accuracy of obtaining the first content vector.

[0104] In step 104 of some embodiments, the masked acoustic feature representation sequence is content encoded by a preset initial content encoding student model to obtain a second content vector. The initial content encoding student model is a neural network model with poor content encoding effect but a small number of parameters and a fast running speed. The number of parameters of the content encoding teacher model is greater than the number of parameters of the initial content encoding student model. The masked acoustic feature representation sequence is input into the initial content encoding student model for content encoding to obtain a second content vector.

[0105] In some embodiments, the initial content encoding student model includes an attention sub-model and a content representation learning sub-model. Figure 3 , step 104 comprises:

[0106] Step 301, predictively encode the masked acoustic feature representation sequence through the attention sub-model to obtain a second predictive encoding sequence;

[0107] Step 302: Perform content representation on the second predicted coding sequence through the content representation learning sub-model to obtain a second content vector.

[0108] Specifically, the attention sub-model is a neural network model combined with a mask prediction mechanism. The attention sub-model can predict the masked content in the masked acoustic feature representation sequence, and can learn the unmasked acoustic feature representation and the long-range temporal structure of the speech sample data. The masked acoustic feature representation sequence is input into the attention sub-model for predictive coding, and a second predicted coding sequence can be obtained. The second predicted coding sequence includes content information and also includes speaker information. Therefore, it is necessary to input the second predicted coding sequence into the content representation learning sub-model for content representation, and a second content vector can be obtained. The second content vector can represent the speaking content in the speech sample data and does not include the speaker information. The attention sub-model can have a model structure similar to the HuBERT (Hidden-Unit BERT) model, but the parameter amount of the attention sub-model is less than that of the HuBERT model. The content representation learning sub-model can be a recurrent neural network (RNNs).

[0109] The benefit of the above embodiment is that predictive coding is performed first and then content representation is performed, thereby improving the accuracy of the second content vector while improving the content coding vector.

[0110] In step 105 of some embodiments, content distillation loss is calculated based on the first content vector and the second content vector to obtain target loss data. The target loss data is used to characterize the content encoding error between the initial content encoding student model and the content encoding teacher model. The smaller the target loss data, the smaller the content encoding error, which means that the content encoding accuracy of the initial content encoding student model is closer to the content encoding teacher model.

[0111] In one embodiment, step 105 includes:

[0112] Performing difference calculation based on the first content vector and the second content vector to obtain a content coding error;

[0113] Performing hyperbolic cosine calculation on the content coding error to obtain hyperbolic cosine data;

[0114] Perform logarithmic calculation on the hyperbolic cosine data to obtain the target loss data.

[0115] The calculation process of target loss data is shown in the following formula:

[0116] L=L distill =Log(cosh(R1-R2))Formula (1).

[0117] In formula (1), L represents the target loss function, L distill represents content distillation loss data, R1 represents the first content vector, and R2 represents the second content vector.

[0118] The benefit of the above embodiment is that it does not directly use the content coding error as the target loss data, but also introduces a hyperbolic cosine function and a logarithmic function to calculate the content coding error to obtain the target loss data, which can improve the model convergence speed and improve the model training efficiency.

[0119] It should be noted that the target loss data may also be calculated using functions such as a mean square error function and a mean absolute error function, which is not specifically limited in this embodiment.

[0120] In getting L distill After that, you can directly execute step 106, that is, directly use L distill The parameters of the initial content encoding student model are adjusted to obtain the target content encoding student model.

[0121] In one embodiment, referring to Figure 4 Before step 106, the speech synthesis method of the embodiment of the present application further includes: updating the target loss data, specifically including:

[0122] Step 401, prosody encoding the speech sample data through a preset prosody encoding model to obtain a sample prosody vector;

[0123] Step 402, performing speaker recognition on the speech sample data to obtain a sample speaker vector;

[0124] Step 403, decoding is performed according to the second content vector, the sample speaker vector and the sample prosody vector to obtain speech reconstruction data;

[0125] Step 404, performing loss reconstruction according to the speech sample data and the speech reconstruction data to obtain reconstruction loss data;

[0126] Step 405: Update target loss data according to the reconstructed loss data.

[0127] In step 401, the speech sample data may be input into a prosody coding model for prosody coding to obtain a sample prosody vector. The prosody coding model is a neural network model for prosody coding speech. In one embodiment, the prosody coding model may be designed based on the structure of ECAPA-TDNN.

[0128] In one embodiment, the prosodic coding model includes an acoustic feature extraction sub-model and a prosodic feature extraction sub-model. Figure 5 , step 401 comprises:

[0129] Step 501, extracting acoustic features from speech sample data using an acoustic feature extraction sub-model to obtain an initial acoustic vector;

[0130] Step 502, extracting rhythmic features from the initial acoustic vector using a rhythmic feature extraction sub-model to obtain an acoustic rhythmic vector;

[0131] Step 503: The initial acoustic vector and the acoustic prosody vector are fused to obtain a sample prosody vector.

[0132] Specifically, the network structure of the acoustic feature extraction submodel is basically the same as that of the ECAPA-TDNN network, and is mainly used for acoustic feature extraction to obtain an initial acoustic vector. Considering that the emotional rhythm in the initial acoustic vector is difficult to be fully reflected, this embodiment introduces an emotional rhythm extraction branch, namely a rhythm feature extraction submodel. The initial acoustic vector is input into the rhythm feature extraction submodel for rhythm feature extraction to obtain an acoustic rhythm vector. The acoustic rhythm vector is used to characterize the emotional rhythm information in the speech sample data. Emotional rhythm information includes fundamental frequency, energy, glottal period, etc. Then the initial acoustic vector and the acoustic rhythm vector are fused. For example, the vector dimension of the initial acoustic vector is consistent with the vector dimension of the acoustic rhythm vector, and the sample rhythm vector can be obtained by fusion in series. The sample rhythm vector is a feature representation vector with emotional rhythm information.

[0133] The benefit of the above embodiment is that acoustic feature extraction is performed first and then rhythmic feature extraction, and the obtained initial acoustic vector and acoustic rhythmic vector are fused to obtain a sample rhythmic vector, thereby improving the accuracy of obtaining the sample rhythmic vector.

[0134] In step 402, the voice sample data may be subjected to speaker recognition through a voiceprint recognition model to obtain a sample speaker vector. Voiceprint recognition is a biometric recognition technology that performs identity authentication by analyzing an individual's voice features. Voiceprint recognition models typically use deep learning techniques, such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs), to extract and recognize voice features.

[0135] In step 403, the second content vector, the sample speaker vector and the sample prosody vector may be decoded by a decoder to obtain speech reconstruction data. A decoder is a neural network structure in deep learning, usually used in conjunction with an encoder to convert the high-dimensional representation learned by the encoder into a target output. In this embodiment, the decoder may be used to convert the second content vector, the sample speaker vector and the prosody vector into speech reconstruction data.

[0136] In one example, step 403 includes: concatenating the second content vector, the sample speaker vector and the sample prosody vector to obtain a target concatenated vector; and inputting the target concatenated vector into a decoder for decoding to obtain speech reconstruction data.

[0137] In another example, step 403 includes: performing vector summation on the second content vector, the sample speaker vector and the sample prosody vector to obtain a target summation vector; and inputting the target summation vector into a decoder for decoding to obtain speech reconstruction data.

[0138] In step 404, the speech sample data and the speech reconstruction data may be subjected to loss reconstruction using a preset loss function to obtain reconstruction loss data. The loss function may be an absolute error function, a mean square error loss function, or the like.

[0139] In step 405, the target loss data is updated according to the reconstructed loss data.

[0140] In one embodiment, step 405 may include: adding the reconstruction loss data and the content distillation loss data to obtain target loss data. The target loss data may be expressed as follows:

[0141] L=L distill +L recon Formula (2).

[0142] In formula (2), L represents the target loss data, L distill Represents the content distillation loss data, L recon Represents the reconstruction loss data.

[0143] The benefit of the embodiment of the above steps 401 to 405 is that, by updating the target loss data, the influence of the speech reconstruction data on the target loss data is reflected, and the accuracy of the target loss data is further improved.

[0144] In one embodiment, updating the target loss data further includes:

[0145] Multiply the preset content distillation weight and the target loss data to obtain the first loss data;

[0146] The first loss data and the reconstruction loss data are added to obtain target fusion loss data, and the target loss data is updated to the target fusion loss data.

[0147] It should be noted that before the target loss data is updated, the target loss data is the same as the content distillation loss data, that is, the first loss data is actually equal to the product of the content distillation weight and the content distillation loss data.

[0148] The calculation process of the above target loss data is shown in the following formula:

[0149] L=L recon +μ*L distill Formula (3).

[0150] In formula (3), L represents the target loss data, L recon Represents the reconstruction loss data, L distill Represents the content distillation loss data. μ represents the content distillation weight, which is greater than 0 and less than 1.

[0151] The benefit of the above embodiment is that, when updating the target loss data, the importance of the speech reconstruction loss data is fully reflected, so that the accuracy of speech synthesis can be ensured.

[0152] In one embodiment, after step 401, refer to Figure 6 , the speech synthesis method further comprises:

[0153] Step 601, performing emotion recognition on the sample prosody vector to obtain the sample emotion category;

[0154] Step 602, calculating the emotion recognition loss according to the sample emotion category and the preset sample emotion label to obtain the rhythm loss data;

[0155] Step 603: Adjust parameters of the prosody coding model according to the prosody loss data to obtain a target prosody coding model.

[0156] In step 601, the sample prosody vector may be subjected to emotion recognition by an emotion recognition model to obtain a sample emotion category. The emotion recognition model is a deep learning model that is intended to recognize emotion or sentiment information contained in speech data. The emotion recognition model may be a speech emotion recognition model (SER).

[0157] In step 602, the emotion recognition loss calculation can be performed on the sample emotion category and the sample emotion label by using a preset loss function to obtain prosody loss data. The loss function can be an absolute error function, a mean square error loss function, or the like.

[0158] In step 603, the parameters of the prosody coding model are adjusted according to the prosody loss data, and in the parameter adjustment process, the prosody loss data is compared with a preset first loss threshold. If the prosody loss data is greater than the first loss threshold, the parameters of the prosody coding model are continuously adjusted until the prosody loss data is less than or equal to the first loss threshold, and the parameter adjustment is stopped to obtain a target prosody coding model. The target prosody coding model can be used to perform prosody coding on the target speech data that appears below to obtain a target prosody vector.

[0159] The benefit of the above embodiment is that the sample emotion category can be identified based on the sample prosody information, and the prosody loss data can be calculated based on the sample emotion category, so that the parameters of the prosody encoder can be adjusted based on the prosody loss data to obtain a target prosody encoder, thereby improving the encoding performance of the target prosody encoder.

[0160] In step 106 of some embodiments, parameters of the initial content encoding student model may be adjusted according to the target loss data to obtain a target content encoding student model.

[0161] Referring to the above, the target loss data as shown in any one of formulas (1), (2) and (3) can be used to adjust the parameters of the initial content coding student model to obtain the target content coding student model. The target content coding student model is the same as the initial content coding student model in terms of model structure, but the model parameters are different. During the parameter adjustment process, the target loss data is compared with a preset second loss threshold. If the target loss data is greater than the second loss threshold, the parameters of the initial content coding student model are continuously adjusted until the target loss data is less than or equal to the second loss threshold, then the parameter adjustment is stopped to obtain the target content coding student model.

[0162] In step 107 of some embodiments, target voice data is obtained. The method of obtaining the target voice data is similar to the process of obtaining the voice sample data above, and will not be repeated here. The target voice data is used to represent the voice uttered by the target speaker according to the target voice content. The target voice content is used to represent the text information of the target voice data. It should be noted that the target voice data can be a voice composed of Chinese or English, etc.

[0163] In step 108 of some embodiments, the target speech data is content-encoded by the target content encoding student model to obtain a target content vector. Since the target content encoding student model is obtained by adjusting the parameters of the initial content encoding student model through steps 101 to 106, it has a high content encoding capability, so that the target content vector can well represent the content characteristics of the target speech content.

[0164] In step 109 of some embodiments, the target content vector, the preset target speaker vector and the preset target prosody vector can be decoded by a decoder to obtain target synthesized speech data. The decoder has been explained above and will not be repeated here. The target speaker vector can be obtained by voiceprint encoding the speaker speech data through a voiceprint recognition model. The target prosody vector can be obtained by prosody encoding the emotional speech data through a target prosody encoding model. The speaker speech data is speech data used to provide speaker information. The emotional speech data is speech data used to provide emotional prosody information. The speaker speech data and the emotional speech data can be the same speech data or different speech data.

[0165] See also Figure 7 In one embodiment, the speech synthesis method includes the following implementation steps:

[0166] (1) obtaining an acoustic feature representation sequence of speech sample data to obtain an initial acoustic feature representation sequence;

[0167] (2) performing masking processing on the initial acoustic feature representation sequence to obtain a masked acoustic feature representation sequence;

[0168] (3) performing predictive coding on the masked acoustic feature representation sequence through the speech recognition pre-trained sub-model to obtain a first predictive coding sequence, and performing instance normalization on the first predictive coding sequence through a normalization layer to obtain a first content vector;

[0169] (4) performing predictive coding on the masked acoustic feature representation sequence through the attention sub-model to obtain a second predictive coding sequence, and performing content representation on the second predictive coding sequence through the content representation learning sub-model to obtain a second content vector;

[0170] (5) Calculate the content distillation loss based on the first content vector and the second content vector to obtain the content distillation loss data L distill ;

[0171] (6) performing prosody encoding on the speech sample data through a prosody coding model to obtain a sample prosody vector;

[0172] (7) Perform speaker recognition on the speech sample data to obtain a sample speaker vector;

[0173] (8) decoding the second content vector, the sample speaker vector, and the sample prosody vector through a decoder to obtain speech reconstruction data;

[0174] (9) Perform loss reconstruction based on the speech sample data and speech reconstruction data to obtain the reconstructed loss data L recon ;

[0175] (10) Summing the reconstruction loss data and the content distillation loss data to obtain the target loss data;

[0176] (11) adjusting parameters of the initial content encoding student model according to the target loss to obtain a target content encoding student model;

[0177] (12) performing emotion recognition on the sample rhythm vector through an emotion recognition model to obtain the sample emotion category;

[0178] (13) Calculate the emotion recognition loss based on the sample emotion category and the preset sample emotion label to obtain the rhythm loss data L p ;

[0179] (14) According to the rhythm loss data L p The parameters of the prosody coding model are adjusted to obtain a target prosody coding model.

[0180] After obtaining the target content coding student model and the target prosody coding model through the above steps (11) to (14), the target content coding student model can be used to perform content coding on the target speech data to obtain a target content vector, and the target prosody coding model can be used to perform prosody coding on the emotional speech data to obtain a target prosody vector, and the voiceprint recognition model can be used to perform voiceprint recognition on the speaker speech data to obtain a speaker vector. Then, the decoder is used to decode the target content vector, the target speaker vector, and the target prosody vector to obtain the target synthesized speech data.

[0181] The embodiments of the present application have at least the following benefits: 1. More natural voice expression: Through emotional rhythm modeling and conversion, the voice output of intelligent customer service is made more expressive and natural, which can simulate the voice communication characteristics of human customer service and improve the communication experience between users and intelligent customer service. 2. Improve user experience: Introducing emotional rhythm features enables the bank's intelligent customer service system to identify and adapt to the emotional needs of users, better meet the communication needs of users in different scenarios, and improve customer satisfaction.

[0182] See also Figure 8 The embodiment of the present application also provides a speech synthesis device, which can implement the above-mentioned speech synthesis method. Figure 8The module structure block diagram of the speech synthesis device provided in the embodiment of the present application includes: a speech acquisition module 801, a mask processing module 802, a content encoding module 803, a loss calculation module 804, a parameter adjustment module 805 and a speech synthesis module 806. Among them, the speech acquisition module 801 is used to acquire the acoustic feature representation sequence of the speech sample data to obtain the initial acoustic feature representation sequence; the mask processing module 802 is used to perform mask processing on the initial acoustic feature representation sequence to obtain the masked acoustic feature representation sequence; the content encoding module 803 is used to perform content encoding on the masked acoustic feature representation sequence through a preset content encoding teacher model to obtain a first content vector; the content encoding module 803 is also used to perform content encoding on the masked acoustic feature representation sequence through a preset initial content encoding student model to obtain a second content vector; wherein the parameter amount of the content encoding teacher model is greater than the initial content encoding student model. The parameter quantity of the initial content coding student model; the loss calculation module 804 is used to perform content distillation loss calculation according to the first content vector and the second content vector to obtain target loss data; the parameter adjustment module 805 is used to adjust the parameters of the initial content coding student model according to the target loss data to obtain the target content coding student model; the speech synthesis module 806 is used to: obtain target speech data; content encode the target speech data through the target content coding student model to obtain the target content vector; decode according to the target content vector, the preset target speaker vector and the preset target rhythm vector to obtain the target synthesized speech data.

[0183] In one embodiment, the speech synthesis device further includes: a loss updating module, configured to update the target loss data.

[0184] In another embodiment, the speech synthesis device also includes: a prosody coding model adjustment module, which is used to: perform emotion recognition on the sample prosody vector to obtain the sample emotion category; calculate the emotion recognition loss according to the sample emotion category and the preset sample emotion label to obtain the prosody loss data; adjust the parameters of the prosody coding model according to the prosody loss data to obtain the target prosody coding model.

[0185] It should be noted that the specific implementation of the speech synthesis device is basically the same as the specific embodiment of the above-mentioned speech synthesis method, and will not be repeated here.

[0186] The embodiment of the present application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned speech synthesis method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0187] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0188] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0189] The memory 902 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 may store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the speech synthesis method of the embodiments of this application;

[0190] Input / output interface 903, used to implement information input and output;

[0191] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0192] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0193] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0194] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned speech synthesis method.

[0195] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0196] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0197] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0198] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0200] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0201] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0202] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0203] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0206] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Acquiring an acoustic feature representation sequence of the speech sample data to obtain an initial acoustic feature representation sequence; Performing masking processing on the initial acoustic feature representation sequence to obtain a masked acoustic feature representation sequence; Content encoding the masked acoustic feature representation sequence is performed using a preset content encoding teacher model to obtain a first content vector; Content encoding the masked acoustic feature representation sequence is performed using a preset initial content encoding student model to obtain a second content vector; wherein the parameter amount of the content encoding teacher model is greater than the parameter amount of the initial content encoding student model; Calculate content distillation loss according to the first content vector and the second content vector to obtain target loss data; Adjusting parameters of the initial content encoding student model according to the target loss data to obtain a target content encoding student model; Obtain target voice data; Content encoding is performed on the target speech data by using the target content encoding student model to obtain a target content vector; Decoding is performed according to the target content vector, the preset target speaker vector and the preset target prosody vector to obtain target synthesized speech data.

2. The method according to claim 1, characterized in that The content encoding teacher model includes a speech recognition pre-trained sub-model and a normalization layer; The step of performing content encoding on the mask acoustic feature representation sequence by using a preset content encoding teacher model to obtain a first content vector includes: Performing predictive coding on the masked acoustic feature representation sequence through the speech recognition pre-trained sub-model to obtain a first predictive coding sequence; The first predicted coding sequence is instance-normalized by the normalization layer to obtain the first content vector.

3. The method according to claim 1, characterized in that: The initial content encoding student model includes an attention sub-model and a content representation learning sub-model; The step of performing content encoding on the mask acoustic feature representation sequence by using a preset initial content encoding student model to obtain a second content vector includes: Performing predictive coding on the masked acoustic feature representation sequence through the attention sub-model to obtain a second predictive coding sequence; The second predicted coding sequence is content represented by the content representation learning sub-model to obtain the second content vector.

4. The method according to any one of claims 1 to 3, characterized in that: Before adjusting the parameters of the initial content encoding student model according to the target loss data to obtain the target content encoding student model, the method further includes: updating the target loss data, specifically including: Prosody encoding the speech sample data is performed using a preset prosody encoding model to obtain a sample prosody vector; Performing speaker recognition on the speech sample data to obtain a sample speaker vector; Decoding is performed according to the second content vector, the sample speaker vector and the sample prosody vector to obtain speech reconstruction data; Perform loss reconstruction according to the speech sample data and the speech reconstruction data to obtain reconstructed loss data; The target loss data is updated according to the reconstructed loss data.

5. The method according to claim 4, characterized in that The prosody coding model includes an acoustic feature extraction sub-model and a prosody feature extraction sub-model; The prosody encoding of the speech sample data is performed using a preset prosody encoding model to obtain a sample prosody vector; Extracting acoustic features from the speech sample data using the acoustic feature extraction submodel to obtain an initial acoustic vector; Extracting rhythmic features from the initial acoustic vector using the rhythmic feature extraction sub-model to obtain an acoustic rhythmic vector; The initial acoustic vector and the acoustic prosody vector are fused to obtain the sample prosody vector.

6. The method according to claim 4, characterized in that After the prosody encoding of the speech sample data is performed using a preset prosody encoding model to obtain a sample prosody vector, the method further includes: Performing emotion recognition on the sample rhythm vector to obtain a sample emotion category; Emotion recognition loss is calculated based on the sample emotion category and the preset sample emotion label to obtain rhythm loss data; The parameters of the prosody coding model are adjusted according to the prosody loss data to obtain a target prosody coding model; wherein the target prosody coding model is used to perform prosody coding on the target speech data to obtain the target prosody vector.

7. The method according to any one of claims 1 to 3, characterized in that: The performing content distillation loss calculation according to the first content vector and the second content vector to obtain target loss data includes: Performing a difference calculation based on the first content vector and the second content vector to obtain a content coding error; Performing hyperbolic cosine calculation on the content coding error to obtain hyperbolic cosine data; Perform logarithmic calculation on the hyperbolic cosine data to obtain the target loss data.

8. A speech synthesis device, characterized in that: The device comprises: A speech acquisition module, used to acquire an acoustic feature representation sequence of speech sample data to obtain an initial acoustic feature representation sequence; A mask processing module, used for performing mask processing on the initial acoustic feature representation sequence to obtain a masked acoustic feature representation sequence; A content encoding module, configured to perform content encoding on the masked acoustic feature representation sequence through a preset content encoding teacher model to obtain a first content vector; The content encoding module is further used to perform content encoding on the masked acoustic feature representation sequence through a preset initial content encoding student model to obtain a second content vector; wherein the parameter amount of the content encoding teacher model is greater than the parameter amount of the initial content encoding student model; a loss calculation module, configured to perform content distillation loss calculation according to the first content vector and the second content vector to obtain target loss data; A parameter adjustment module, used for adjusting the parameters of the initial content encoding student model according to the target loss data to obtain a target content encoding student model; Speech synthesis module for: Obtain target voice data; Content encoding is performed on the target speech data by using the target content encoding student model to obtain a target content vector; Decoding is performed according to the target content vector, the preset target speaker vector and the preset target prosody vector to obtain target synthesized speech data.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech synthesis method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Audio signal processing method and device, electronic equipment and storage medium

    CN111179961A

  • Speech synthesis method and device, vocoder and electronic equipment

    CN112365875A

  • Model training method and speech synthesis method based on semi-supervised knowledge distillation

    CN116092469A

  • Phoneme identification method and device, electronic equipment and storage medium

    CN116978359A

  • Text data processing method and apparatus

    US20230360634A1