Speech synthesis method and device, computer equipment and readable storage medium

Abstract

The invention relates to the field of artificial intelligence, and can improve the fluency and naturalness of multilingual speech synthesis in the aspect of speech code conversion. Relates to a speech synthesis method and device, equipment and a medium. The method comprises the following steps: acquiring initial feature vectors respectively corresponding to at least two types of initial speech data; calling the trained speech synthesis model; determining a target language expert unit corresponding to each type of initial voice data, and inputting the initial feature vector corresponding to each type of initial voice data into the corresponding target language expert unit for feature extraction to obtain first feature information corresponding to each type of initial voice data; inputting the initial feature vectors corresponding to all the initial voice data into a language-independent expert unit for feature extraction to obtain second feature information; determining target feature information according to the second feature information and the first feature information corresponding to each type of initial voice data; and performing voice synthesis on the target feature information to obtain target voice data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, apparatus, computer device and computer-readable storage medium. Background Art

[0002] Globalization has brought about the need for different languages ​​and speech interactions. Code-switching and accent control are particularly valuable in multilingual text-to-speech systems because they both help improve the authenticity and comprehensibility of speech. Code-switching refers to the use of two or more different languages, dialects, or styles by bilinguals in a bilingual or multilingual communication environment. How to seamlessly synthesize speech in two languages ​​in one sentence is particularly important. Related technologies have problems with poor fluency and naturalness when synthesizing speech in at least two languages. Summary of the invention

[0003] The present application provides a speech synthesis method, a speech synthesis device, a computer device and a computer-readable storage medium, which can effectively improve the fluency and naturalness of speech synthesis in at least two languages.

[0004] In a first aspect, the present application provides a speech synthesis method, the method comprising:

[0005] Obtaining initial feature vectors corresponding to at least two types of initial speech data;

[0006] Calling a trained speech synthesis model, wherein the speech synthesis model includes a mixed expert module, wherein the mixed expert module includes a language-independent expert unit and at least one language expert unit;

[0007] Determine the target language expert unit corresponding to each of the initial speech data, input the initial feature vector corresponding to each of the initial speech data into the corresponding target language expert unit for feature extraction, and obtain the first feature information corresponding to each of the initial speech data;

[0008] Inputting the initial feature vectors corresponding to all the initial speech data into the language-independent expert unit for feature extraction to obtain second feature information;

[0009] Determining target feature information according to the second feature information and the first feature information corresponding to each type of initial voice data;

[0010] Perform speech synthesis on the target feature information to obtain target speech data.

[0011] In a second aspect, the present application further provides a speech synthesis device, the speech synthesis device comprising:

[0012] A feature vector acquisition module, used to acquire initial feature vectors corresponding to at least two types of initial speech data;

[0013] A model calling module, used for calling a trained speech synthesis model, wherein the speech synthesis model comprises a mixed expert module, wherein the mixed expert module comprises a language-independent expert unit and at least one language expert unit;

[0014] A first feature extraction module is used to determine the target language expert unit corresponding to each of the initial speech data, input the initial feature vector corresponding to each of the initial speech data into the corresponding target language expert unit for feature extraction, and obtain the first feature information corresponding to each of the initial speech data;

[0015] A second feature extraction module, used for inputting the initial feature vectors corresponding to all the initial speech data into the language-independent expert unit for feature extraction to obtain second feature information;

[0016] a feature information determination module, configured to determine target feature information according to the second feature information and the first feature information corresponding to each type of initial voice data;

[0017] The speech synthesis module is used to perform speech synthesis on the target feature information to obtain target speech data.

[0018] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor;

[0019] The memory is used to store computer programs;

[0020] The processor is used to execute the computer program and implement the above-mentioned speech synthesis method when executing the computer program.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the speech synthesis method as described above.

[0022] The present application discloses a speech synthesis method, device, computer equipment and readable storage medium. Since different language expert units focus on different languages, by determining the target language expert unit corresponding to each initial speech data, the initial feature vector corresponding to each initial speech data is input into the corresponding target language expert unit for feature extraction, so that the language expert unit can more accurately extract the language-specific features in the initial speech data and integrate the semantic information in the speech; by inputting the initial feature vectors corresponding to all initial speech data into the language-independent expert unit for feature extraction, it is possible to extract feature information that is independent of the language, so as to strengthen the interaction of code conversion context information. Thus, the fluency and naturalness of multi-language speech synthesis in code conversion can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 is a structural schematic diagram of a computer device provided in an embodiment of the present application;

[0025] Figure 2 is a schematic flow chart of a speech synthesis method provided in an embodiment of the present application;

[0026] Figure 3 It is a structural schematic diagram of a speech synthesis model provided in an embodiment of the present application;

[0027] Figure 4 is a schematic flow chart of a training process of a speech synthesis model provided in an embodiment of the present application;

[0028] Figure 5 It is a schematic block diagram of a speech synthesis device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0031] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0032] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0033] The embodiment of the present application provides a speech synthesis method, a speech synthesis device, a computer device and a readable storage medium. The speech synthesis method can be applied to a computer device. By determining the target language expert unit corresponding to each initial speech data, the initial feature vector corresponding to each initial speech data is input into the corresponding target language expert unit for feature extraction, so that the language expert unit can more accurately extract the language-specific features in the initial speech data and integrate the semantic information in the speech; by inputting the initial feature vectors corresponding to all the initial speech data into the language-independent expert unit for feature extraction, it is possible to extract feature information that is not related to the language, so as to strengthen the interaction of code conversion context information. Thereby, the fluency and naturalness of multilingual speech synthesis in code conversion can be improved.

[0034] Exemplarily, the computer device can be a server or a terminal. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal can be an electronic device such as a smartphone, tablet computer, laptop computer, and desktop computer.

[0035] See also Figure 1 , Figure 1 1 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device may include a processor and a memory, wherein the processor and the memory may be connected via a bus, and the bus may be any applicable bus such as an Inter-integrated Circuit (I2C) bus.

[0036] The memory may include a storage medium and an internal memory. The storage medium may be a non-volatile storage medium or a volatile storage medium. The storage medium may store an operating system and a computer program, and the internal memory provides an environment for the operation of the computer program in the storage medium. The computer program includes program instructions, and when the program instructions are executed, the processor may execute the speech synthesis method described in any embodiment.

[0037] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0038] The processor may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or any conventional processor.

[0039] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0040] Acquire initial feature vectors corresponding to at least two types of initial speech data; call a trained speech synthesis model, the speech synthesis model includes a mixed expert module, the mixed expert module includes a language-independent expert unit and at least one language expert unit; determine the target language expert unit corresponding to each type of initial speech data, input the initial feature vector corresponding to each type of initial speech data into the corresponding target language expert unit for feature extraction, and obtain first feature information corresponding to each type of initial speech data; input the initial feature vectors corresponding to all the initial speech data into the language-independent expert unit for feature extraction, and obtain second feature information; determine target feature information based on the second feature information and the first feature information corresponding to each type of initial speech data; perform speech synthesis on the target feature information to obtain target speech data.

[0041] In one embodiment, before implementing the calling of the trained speech synthesis model, the processor is further configured to implement:

[0042] Acquire sample speech data of at least two languages ​​and sample text data corresponding to each sample speech data; vectorize each sample speech data to obtain a sample feature vector corresponding to each sample speech data; input the sample feature vector into at least one language expert unit in the speech synthesis model for feature extraction training to obtain first prediction feature information corresponding to each language expert unit; input the sample feature vector into a language-independent expert unit in the speech synthesis model for feature extraction training to obtain second prediction feature information; perform speech synthesis based on the second prediction feature information and the first prediction feature information corresponding to each language expert unit to obtain predicted speech data corresponding to each language expert unit; calculate the loss function value based on the predicted speech data corresponding to each language expert unit and the sample text data corresponding to the sample speech data to obtain the loss function value of this round of training for each language expert unit; if the loss function value is greater than a preset loss function threshold, adjust the parameters of the speech synthesis model and perform the next round of training until the loss function value obtained is less than or equal to the loss function threshold, terminate the training, and obtain a trained speech synthesis model.

[0043] In one embodiment, when the processor calculates the loss function value according to the predicted speech data corresponding to each language expert unit and the sample text data corresponding to the sample speech data to obtain the loss function value of each language expert unit in this round of training, it is used to implement:

[0044] The loss function value is calculated based on the predicted feature information corresponding to each language expert unit and the sample text data corresponding to the sample feature vector to obtain the initial loss function value corresponding to each language expert unit; the variance regularization calculation is performed on the speaker identity feature vector in the sample feature vector to obtain the variance regularization term; the covariance regularization calculation is performed on the speaker identity feature vector in the sample feature vector to obtain the covariance regularization term; the variance regularization term and the covariance regularization term are added to the initial loss function value to obtain the loss function value of each language expert unit for this round of training.

[0045] In one embodiment, when the processor calculates the loss function value according to the predicted speech data corresponding to each language expert unit and the sample text data corresponding to the sample speech data to obtain the loss function value of each language expert unit in this round of training, it is used to implement:

[0046] The loss function value is calculated based on the predicted feature information corresponding to each language expert unit and the sample text data corresponding to the sample feature vector to obtain the initial loss function value corresponding to each language expert unit; the variance regularization calculation is performed on the accent feature vector in the sample feature vector to obtain the variance regularization term; the covariance regularization calculation is performed on the accent feature vector in the sample feature vector to obtain the covariance regularization term; the variance regularization term and the covariance regularization term are added to the initial loss function value to obtain the loss function value of each language expert unit for this round of training.

[0047] In one embodiment, after performing speech synthesis based on the second prediction feature information and the first prediction feature information corresponding to each language expert unit to obtain the predicted speech data corresponding to each language expert unit, the processor is further configured to implement:

[0048] Extracting the first speaker feature information from the sample speech data and extracting the second speaker feature information from the predicted speech data; calculating the similarity between the first speaker feature information and the second speaker feature information; if the similarity is less than a preset value, adjusting the parameters of the speech synthesis model and performing the next round of training until the obtained similarity is greater than or equal to the preset value.

[0049] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features of the embodiments can be combined with each other. Figure 2 , Figure 2 is a schematic flow chart of a speech synthesis method provided in an embodiment of the present application. Figure 2 As shown, the speech synthesis method may include steps S101 to S106.

[0050] Step S101: Obtain initial feature vectors corresponding to at least two types of initial speech data.

[0051] The speech synthesis method provided in the embodiment of the present application can be applied to the scenario of speech synthesis of speech data in two or more languages, ensuring that the generated speech is consistent with the required accent, thereby improving the fluency and naturalness of multilingual speech synthesis in code conversion.

[0052] Exemplarily, a vectorization model can be used to vectorize at least two types of initial speech data to obtain initial feature vectors corresponding to each type of initial speech data. The vectorization model may include but is not limited to a BERT (Bidirectional Encoder Representations from Transformer) model, a word2vec model, a glove model, an ELMo model, and the like. For example, each type of initial speech data can be input into a BERT model for vectorization to obtain a corresponding initial feature vector. The specific vectorization process can be referred to in the relevant technology and is not limited here.

[0053] In the embodiment of the present application, initial feature vectors corresponding to at least two types of initial speech data may be obtained as inputs to a speech synthesis model.

[0054] Step S102: calling a trained speech synthesis model, where the speech synthesis model includes a hybrid expert module, and the hybrid expert module includes a language-independent expert unit and at least one language expert unit.

[0055] Exemplarily, the trained speech synthesis model can be called to perform speech synthesis on the initial speech data.

[0056] In the embodiment of the present application, the speech synthesis model uses a Mixture-of-Language-Experts (MoLE) module as an encoder. Since the processing of different languages ​​can be completed by different experts, each expert focuses on a specific language field, it is possible to extract language-specific features and integrate semantic information within the discourse. The basic network architecture of the speech synthesis model can adopt an end-to-end speech synthesis model (such as FastSpeech2, VITS), or other encoder-decoder structured speech synthesis models.

[0057] It should be noted that speech data contains different features such as language, accent, rhythm, emotion, etc., but these multiple features are mixed and coupled together. The purpose of using the hybrid expert module is to decouple different features through multi-expert learning, so as to achieve the purpose of controlling the synthesis effect.

[0058] See also Figure 3 , Figure 3 Schematic diagram of a speech synthesis model provided in an embodiment of the present application. Figure 3As shown, the speech synthesis model includes a gating network and a hybrid expert module, and the hybrid expert module includes a language-independent expert unit and at least one language expert unit, for example, a language 1 expert unit, a language 2 expert unit, ... Among them, the gating network is responsible for activating the language expert unit that is most suitable for processing the input feature vector, and the experts here are divided into language-independent experts and language experts. The language-independent expert unit and each language expert unit are an independent neural network, such as a feedforward network (FFN), or other more complex network structures. The language-independent expert mainly learns the features in the speech that are not related to the language (such as emotion, tone, mood, etc.); there are multiple language experts (generally corresponding to different languages), and they mainly learn the pronunciation features corresponding to different languages ​​(accent, acoustic rhythm, phonemes, etc.).

[0059] Step S103, determining the target language expert unit corresponding to each type of initial speech data, inputting the initial feature vector corresponding to each type of initial speech data into the corresponding target language expert unit for feature extraction, and obtaining the first feature information corresponding to each type of initial speech data.

[0060] In some embodiments, the target language expert unit corresponding to each type of initial speech data may be determined through a gating network.

[0061] It should be noted that the gating network (also called a router) can use a similar softmax gating function. For each language expert unit, the gating network predicts the probability value of the language expert unit for each speech input (token, word unit), thereby generating the best output for a given speech input. It can be understood that the gating network does not calculate the output of all language expert units, but calculates the output of the top K language expert units (the output predicted by them), and K can generally be taken as 1 or 2.

[0062] Exemplarily, for each language expert unit, a probability value corresponding to the language expert unit can be predicted for each initial speech data through a gated network, and the top K language expert units are used as target language expert units corresponding to the corresponding initial speech data.

[0063] Exemplarily, the initial feature vector corresponding to each initial speech data can be input into the corresponding target language expert unit for feature extraction to obtain the first feature information corresponding to each initial speech data. The first feature information is feature information related to the language, and the first feature information may include pronunciation features such as accent, acoustic rhythm, and phoneme.

[0064] For example, for initial speech data in Mandarin, the initial feature vector corresponding to the initial speech data can be input into the target language expert unit for processing Mandarin for feature extraction to obtain the first feature information corresponding to the initial speech data; for another example, for initial speech data in Cantonese, the initial feature vector corresponding to the initial speech data can be input into the target language expert unit for processing Cantonese for feature extraction to obtain the first feature information corresponding to the initial speech data.

[0065] In the above embodiment, since different language expert units focus on different languages, by determining the target language expert unit corresponding to each type of initial speech data, the initial feature vector corresponding to each type of initial speech data is input into the corresponding target language expert unit for feature extraction. This enables the language expert unit to more accurately extract language-specific features in the initial speech data and integrate the semantic information in the speech.

[0066] Step S104: input the initial feature vectors corresponding to all the initial speech data into the language-independent expert unit for feature extraction to obtain second feature information.

[0067] Exemplarily, the initial feature vectors corresponding to all the initial speech data can be input into the language-independent expert unit for feature extraction to obtain the second feature information, wherein the second feature information is language-independent feature information and may include features such as emotion, tone, and voice.

[0068] It should be noted that by inputting the initial feature vectors corresponding to all the initial speech data into the language-independent expert unit for feature extraction, feature information independent of the language can be extracted, thereby enhancing the interaction of code-switching context information.

[0069] Step S105: Determine target feature information according to the second feature information and the first feature information corresponding to each type of initial voice data.

[0070] Exemplarily, the second feature information may be concatenated with the first feature information corresponding to each type of initial voice data to obtain the target feature information. For example, the first feature information may be concatenated after the second feature information, or the second feature information may be concatenated after the first feature information.

[0071] Step S106: Perform speech synthesis on the target feature information to obtain target speech data.

[0072] Exemplarily, the speech synthesis model may further include a vocoder, and the target feature information may be input into the vocoder in the speech synthesis model to perform speech synthesis to obtain target speech data.

[0073] It should be noted that a vocoder is a tool for converting acoustic parameters into speech waveforms. Exemplarily, the vocoder may include but is not limited to a WaveGlow model, a Parallel WaveNet model, a Parallel WaveGan model, a MelGan model, and the like.

[0074] Exemplarily, the vocoder performs speech synthesis on the target feature information, and can perform phase recovery processing on the target feature information based on the Griffin-Lim algorithm, and then restore the speech waveform through short-time Fourier transform (ISTFT), thereby obtaining the target speech data. It should be noted that the Griffin-Lim algorithm needs to use the amplitude spectrum and phase spectrum to generate the reconstructed speech signal.

[0075] In the above embodiment, target feature information is determined based on the second feature information and the first feature information corresponding to each initial speech data, and speech synthesis is performed on the target feature information. Since the first feature information integrates the semantic information in the speech, the second feature information can enhance the interaction of code-conversion context information, thereby improving the fluency and naturalness of multi-language speech synthesis in code-conversion.

[0076] In an embodiment of the present application, before calling the trained speech synthesis model, the initial speech synthesis model may also be iteratively trained. The training process of the speech synthesis model will be described in detail below.

[0077] See also Figure 4 , Figure 4 FIG. 1 is a schematic flow chart of a training process of a speech synthesis model provided in an embodiment of the present application. Figure 4 As shown, the training process of the speech synthesis model may include steps S201 to S207.

[0078] Step S201: Acquire sample speech data in at least two languages ​​and sample text data corresponding to each sample speech data.

[0079] Step S202: vectorize each type of sample speech data to obtain a sample feature vector corresponding to each type of sample speech data.

[0080] Exemplarily, a vectorization model can be used to vectorize each sample speech data to obtain a sample feature vector corresponding to each sample speech data. The vectorization model may include but is not limited to a BERT (Bidirectional Encoder Representations from Transformer) model, a word2vec model, a glove model, an ELMo model, and the like. For example, each sample speech data can be input into a BERT model for vectorization to obtain a sample feature vector corresponding to each sample speech data. The specific vectorization process can be referred to the relevant technology and is not limited here.

[0081] Step S203: input the sample feature vector into at least one language expert unit in the speech synthesis model for feature extraction training to obtain first prediction feature information corresponding to each language expert unit.

[0082] Exemplarily, the sample feature vector corresponding to each sample speech data can be input into at least one language expert unit in the speech synthesis model for feature extraction training to obtain the first prediction feature information corresponding to each language expert unit and the prediction probability corresponding to the first prediction feature information. The specific process of feature extraction training can be referred to in the relevant technology, and the specific process is not described here.

[0083] Step S204: input the sample feature vector into the language-independent expert unit in the speech synthesis model for feature extraction training to obtain second prediction feature information.

[0084] Exemplarily, all sample feature vectors can be input into the language-independent expert unit in the speech synthesis model for feature extraction training to obtain the second prediction feature information and the prediction probability corresponding to the second prediction feature information. The specific process of feature extraction training can be referred to in the relevant technology, and the specific process is not described here.

[0085] Step S205: Perform speech synthesis based on the second prediction feature information and the first prediction feature information corresponding to each language expert unit to obtain predicted speech data corresponding to each language expert unit.

[0086] Exemplarily, the second prediction feature information may be concatenated with the first prediction feature information corresponding to each language expert unit, and then speech synthesis may be performed on the concatenated total feature information to obtain the predicted speech data corresponding to each language expert unit.

[0087] Step S206: Calculate the loss function value based on the predicted speech data corresponding to each language expert unit and the sample text data corresponding to the sample speech data to obtain the loss function value of each language expert unit in this round of training.

[0088] Exemplarily, based on the loss function, the loss function value can be calculated according to the predicted speech data corresponding to each language expert unit and the sample text data corresponding to the sample speech data, so as to obtain the loss function value of each language expert unit in this round of training. Among them, the loss function may include but is not limited to 0-1 loss function, absolute value loss function, logarithmic loss function, square loss function and exponential loss function, etc. For example, based on the absolute value loss function, the loss function value can be calculated according to the predicted speech data corresponding to each language expert unit and the sample text data corresponding to the sample speech data, so as to obtain the loss function value of each language expert unit in this round of training. Among them, the specific process of calculating the loss function value can be referred to the relevant technology, which will not be described in detail here.

[0089] Step S207: If the loss function value is greater than the preset loss function threshold, adjust the parameters of the speech synthesis model and perform the next round of training until the loss function value obtained is less than or equal to the loss function threshold. The training is terminated to obtain the trained speech synthesis model.

[0090] Exemplarily, when the loss function value is greater than a preset loss function threshold, the parameters of the speech synthesis model are adjusted, and the next round of training is performed until the loss function value obtained is less than or equal to the loss function threshold, and the training is terminated to obtain a trained speech synthesis model. The preset loss function threshold can be set according to actual conditions, and the specific value is not limited here.

[0091] Exemplarily, the parameters of the speech synthesis model can be adjusted by a convergence algorithm such as a gradient descent algorithm, a Newton algorithm, a conjugate gradient method or a Cauchy-Newton method. The specific process of adjusting the parameters is not limited here. It is understandable that by adjusting the parameters of the speech synthesis model according to the convergence algorithm, the speech synthesis model can be converged quickly, thereby improving the training efficiency.

[0092] In the above-mentioned embodiment, by inputting sample feature vectors into at least one language expert unit in the speech synthesis model for feature extraction training, the language expert unit can learn the pronunciation features corresponding to different languages, thereby improving the ability of the language expert unit to extract language-specific features in the initial speech data and integrate the semantic information in the speech; by inputting sample feature vectors into the language-independent expert unit in the speech synthesis model for feature extraction training, the language-independent expert unit can learn features in the speech that are not related to the language, thereby strengthening the interaction of code-conversion context information.

[0093] Since the speaker's identity and accent are highly correlated, in order to enhance the speaker's similarity and accent controllability, the speaker's identity and accent need to be separated and decoupled. In an embodiment of the present application, a loss function with a variance-covariance regularization term can be used to control the decoupling of the speaker's identity and accent during the training of the speech synthesis model. By introducing VICReg as a loss function term, VICReg is a regularization method based on variance, covariance, and correlation, which is used for self-supervised training of the joint embedding architecture, which can avoid representation collapse and overfitting in model training.

[0094] In some embodiments, a loss function value is calculated based on the predicted speech data corresponding to each language expert unit and the sample text data corresponding to the sample speech data to obtain the loss function value of each language expert unit in this round of training. It can also include: calculating the loss function value based on the predicted feature information corresponding to each language expert unit and the sample text data corresponding to the sample feature vector to obtain the initial loss function value corresponding to each language expert unit; performing variance regularization calculation on the speaker identity feature vector in the sample feature vector to obtain the variance regularization term; performing covariance regularization calculation on the speaker identity feature vector in the sample feature vector to obtain the covariance regularization term; adding the variance regularization term and the covariance regularization term to the initial loss function value to obtain the loss function value of each language expert unit in this round of training.

[0095] The speaker identity category in the sample speech data can be encoded by a preset speaker identity encoder to obtain a speaker identity feature vector. It should be noted that the speaker identity encoder is a learnable lookup table for encoding the speaker identity category into a speaker identity feature vector.

[0096] In the embodiment of the present application, the variance regularization term is defined as the hinge loss function of the standard deviation of e along dimension d to ensure that each component of e∈R is differentiable, where R is a set of n embedding vectors with a dimension of n×d. The variance regularization calculation formula is as follows:

[0097]

[0098] Where, L var is the variance regularization term; e represents the embedding vector, which can be a speaker identity feature vector or an accent feature vector; var(·) is the variance measure, d represents the dimension of each embedding vector, γ represents the fixed target value of the standard deviation, and ε is an manually set empirical value to prevent numerical instability.

[0099] Exemplarily, based on the above variance regularization calculation formula, variance regularization calculation may be performed on the speaker identity feature vector in the sample feature vector to obtain a variance regularization term.

[0100] The covariance regularization term is defined as the non-diagonal elements of the covariance matrix C(e) approaching zero, which requires that there should be no redundant information between the components in an embedded vector. The covariance regularization calculation formula is as follows:

[0101]

[0102] Where, L covar is the variance regularization term; i, j are row and column indices, respectively, corresponding to a row and a column in the set table (dimension is n×d) composed of n embedded vectors.

[0103] Exemplarily, based on the above covariance regularization calculation formula, covariance regularization calculation may be performed on the speaker identity feature vector in the sample feature vector to obtain a covariance regularization term.

[0104] Exemplarily, after calculating the variance regularization term and the covariance regularization term corresponding to the speaker identity feature vector, the variance regularization term and the covariance regularization term can be added to the initial loss function value to obtain the loss function value of each language expert unit in this round of training.

[0105] In the above embodiment, by performing variance regularization calculation on the speaker identity feature vector in the sample feature vector to obtain the variance regularization term, and performing covariance regularization calculation on the speaker identity feature vector in the sample feature vector to obtain the covariance regularization term, and adding the variance regularization term and the covariance regularization term to the initial loss function value, it is possible to decouple the speaker identity from the accent during the training of the speech synthesis model, thereby avoiding representation collapse and overfitting of the speech synthesis model during the training process.

[0106] In some embodiments, a loss function value is calculated based on the predicted speech data corresponding to each language expert unit and the sample text data corresponding to the sample speech data to obtain the loss function value of each language expert unit in this round of training. It can also include: calculating the loss function value based on the predicted feature information corresponding to each language expert unit and the sample text data corresponding to the sample feature vector to obtain the initial loss function value corresponding to each language expert unit; performing variance regularization calculation on the accent feature vector in the sample feature vector to obtain the variance regularization term; performing covariance regularization calculation on the accent feature vector in the sample feature vector to obtain the covariance regularization term; adding the variance regularization term and the covariance regularization term to the initial loss function value to obtain the loss function value of each language expert unit in this round of training.

[0107] The accent feature vector is obtained by encoding the accent category in the sample speech data through a preset accent encoder. It should be noted that the accent encoder is a learnable lookup table for encoding the accent category into an accent feature vector.

[0108] Exemplarily, based on the above variance regularization calculation formula, the accent feature vector in the sample feature vector can be subjected to variance regularization calculation to obtain a variance regularization term. Based on the above covariance regularization calculation formula, the speaker identity feature vector in the sample feature vector can be subjected to covariance regularization calculation to obtain a covariance regularization term. The specific process of variance regularization calculation and covariance regularization calculation can be referred to the relevant technology and will not be described in detail here.

[0109] Exemplarily, after the variance regularization term and the covariance regularization term corresponding to the accent feature vector are calculated, the variance regularization term and the covariance regularization term can be added to the initial loss function value to obtain the loss function value of each language expert unit in this round of training.

[0110] In the above embodiment, by performing variance regularization calculation on the accent feature vector in the sample feature vector to obtain the variance regularization term, and performing covariance regularization calculation on the accent feature vector in the sample feature vector to obtain the covariance regularization term, and adding the variance regularization term and the covariance regularization term to the initial loss function value, the accent can be decoupled from the speaker identity during the training of the speech synthesis model, thereby avoiding representation collapse and overfitting of the speech synthesis model during the training process. In addition, the speech synthesis model can also generate the pronunciation of a real person and generate speech with an accent.

[0111] In some embodiments, after performing speech synthesis based on the second prediction feature information and the first prediction feature information corresponding to each language expert unit to obtain the predicted speech data corresponding to each language expert unit, it can also include: extracting the first speaker feature information in the sample speech data, and extracting the second speaker feature information in the predicted speech data; calculating the similarity between the first speaker feature information and the second speaker feature information; if the similarity is less than a preset value, adjusting the parameters of the speech synthesis model and performing the next round of training until the obtained similarity is greater than or equal to the preset value.

[0112] It should be noted that in an embodiment of the present application, in order to alleviate the problem of inconsistent speaker timbre caused by incomplete decoupling characteristics in synthesized speech, it is necessary to introduce a speaker consistency loss function (SCL, SpeakerConsistencyLoss). Specifically, a pre-trained speaker encoder model (for example: Wav2vec 2.0, HuBERT and WavLM, etc.) can be used to extract speaker-related feature embeddings from real audio (i.e., sample speech data) and synthesized audio (i.e., predicted speech data). SCL aims to maximize the cosine similarity between the two embeddings. Cosine similarity is the cosine value of the angle between two n-dimensional vectors in n-dimensional space, which is used to calculate the similarity of two vectors. The closer the cosine value is to 1, the closer the angle is to 0 degrees, that is, the more similar the two vectors are.

[0113] Exemplarily, the calculation formula of the speaker consistency loss function is as follows:

[0114]

[0115] Where α is a positive weight, and the specific value can be set according to the actual situation; cos_sim is a retrieval based on cosine similarity, and φ is a function for extracting speaker features from a given audio; g i Represents sample speech data, s i Represents predicted speech data.

[0116] Exemplarily, a speaker encoder model can be used to extract the first speaker feature information in the sample speech data, and to extract the second speaker feature information in the predicted speech data; the similarity between the first speaker feature information and the second speaker feature information is calculated according to a similarity algorithm; if the similarity is less than a preset value, the parameters of the speech synthesis model are adjusted, and the next round of training is performed until the obtained similarity is greater than or equal to the preset value. The similarity algorithm can be cosine similarity, or can be a similarity algorithm such as Euclidean distance, Jaccard similarity coefficient, and Pearson correlation coefficient, which is not limited in this application.

[0117] In the above embodiment, by extracting the first speaker feature information from the sample speech data and extracting the second speaker feature information from the predicted speech data, the similarity between the first speaker feature information and the second speaker feature information is calculated, and when the similarity is less than a preset value, the parameters of the speech synthesis model are adjusted and the next round of training is performed until the obtained similarity is greater than or equal to the preset value. This can effectively improve the timbre similarity of the speech output by the speech synthesis model, thereby enhancing the speaker similarity and accent controllability.

[0118] See also Figure 5 , Figure 5The embodiment of the present application further provides a schematic block diagram of a speech synthesis device 1000, wherein the speech synthesis device 1000 is used to execute the aforementioned speech synthesis method. The speech synthesis device 1000 may be configured in a computer device.

[0119] like Figure 5 As shown, the speech synthesis device 1000 includes: a feature vector acquisition module 1001, a model calling module 1002, a first feature extraction module 1003, a second feature extraction module 1004, a feature information determination module 1005 and a speech synthesis module 1006.

[0120] The feature vector acquisition module 1001 is used to acquire initial feature vectors corresponding to at least two types of initial speech data.

[0121] The model calling module 1002 is used to call the trained speech synthesis model, where the speech synthesis model includes a mixed expert module, which includes a language-independent expert unit and at least one language expert unit.

[0122] The first feature extraction module 1003 is used to determine the target language expert unit corresponding to each initial speech data, input the initial feature vector corresponding to each initial speech data into the corresponding target language expert unit for feature extraction, and obtain the first feature information corresponding to each initial speech data.

[0123] The second feature extraction module 1004 is used to input the initial feature vectors corresponding to all the initial speech data into the language-independent expert unit for feature extraction to obtain second feature information.

[0124] The feature information determination module 1005 is used to determine target feature information according to the second feature information and the first feature information corresponding to each type of initial voice data.

[0125] The speech synthesis module 1006 is used to perform speech synthesis on the target feature information to obtain target speech data.

[0126] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0127] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement any speech synthesis method provided in the embodiment of the present application.

[0128] For example, the program is loaded by the processor and may execute the following steps:

[0129] Acquire initial feature vectors corresponding to at least two types of initial speech data; call a trained speech synthesis model, the speech synthesis model includes a gated network and a hybrid expert module, the hybrid expert module includes a language-independent expert unit and at least one language expert unit; determine the target language expert unit corresponding to each type of initial speech data, input the initial feature vector corresponding to each type of initial speech data into the corresponding target language expert unit for feature extraction, and obtain first feature information corresponding to each type of initial speech data; input the initial feature vectors corresponding to all the initial speech data into the language-independent expert unit for feature extraction, and obtain second feature information; determine target feature information based on the second feature information and the first feature information corresponding to each type of initial speech data; perform speech synthesis on the target feature information to obtain target speech data.

[0130] The computer-readable storage medium may be an internal storage unit of the computer device of the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital card (Secure Digital Card, SD Card), a flash card (Flash Card), etc. equipped on the computer device.

[0131] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0132] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0133] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A speech synthesis method, characterized in that: include: Obtaining initial feature vectors corresponding to at least two types of initial speech data; Calling a trained speech synthesis model, wherein the speech synthesis model includes a mixed expert module, wherein the mixed expert module includes a language-independent expert unit and at least one language expert unit; Determine the target language expert unit corresponding to each of the initial speech data, input the initial feature vector corresponding to each of the initial speech data into the corresponding target language expert unit for feature extraction, and obtain the first feature information corresponding to each of the initial speech data; Inputting the initial feature vectors corresponding to all the initial speech data into the language-independent expert unit for feature extraction to obtain second feature information; Determining target feature information according to the second feature information and the first feature information corresponding to each type of initial voice data; Perform speech synthesis on the target feature information to obtain target speech data.

2. The speech synthesis method according to claim 1, characterized in that: The first characteristic information is characteristic information related to a language, and the second characteristic information is characteristic information unrelated to a language.

3. The speech synthesis method according to claim 1, characterized in that: Before calling the trained speech synthesis model, the method further includes: Acquire sample speech data in at least two languages ​​and sample text data corresponding to each of the sample speech data; Vectorizing each of the sample speech data to obtain a sample feature vector corresponding to each of the sample speech data; Inputting the sample feature vector into at least one language expert unit in the speech synthesis model for feature extraction training to obtain first prediction feature information corresponding to each language expert unit; Inputting the sample feature vector into the language-independent expert unit in the speech synthesis model for feature extraction training to obtain second prediction feature information; Performing speech synthesis according to the second prediction feature information and the first prediction feature information corresponding to each of the language expert units to obtain predicted speech data corresponding to each of the language expert units; Calculate the loss function value according to the predicted speech data corresponding to each of the language expert units and the sample text data corresponding to the sample speech data, and obtain the loss function value of each of the language expert units in this round of training; If the loss function value is greater than the preset loss function threshold, the parameters of the speech synthesis model are adjusted and the next round of training is performed until the loss function value obtained is less than or equal to the loss function threshold. The training is terminated to obtain the trained speech synthesis model.

4. The speech synthesis method according to claim 3, characterized in that: The loss function value is calculated according to the predicted speech data corresponding to each of the language expert units and the sample text data corresponding to the sample speech data to obtain the loss function value of each of the language expert units in this round of training, and further includes: Calculating the loss function value based on the prediction feature information corresponding to each of the language expert units and the sample text data corresponding to the sample feature vector to obtain an initial loss function value corresponding to each of the language expert units; Performing variance regularization calculation on the speaker identity feature vector in the sample feature vector to obtain a variance regularization term; Performing covariance regularization calculation on the speaker identity feature vector in the sample feature vector to obtain a covariance regularization term; The variance regularization term and the covariance regularization term are added to the initial loss function value to obtain the loss function value of each language expert unit in this round of training.

5. The speech synthesis method according to claim 3, characterized in that: The loss function value is calculated according to the predicted speech data corresponding to each of the language expert units and the sample text data corresponding to the sample speech data to obtain the loss function value of each of the language expert units in this round of training, and further includes: Calculating the loss function value based on the prediction feature information corresponding to each of the language expert units and the sample text data corresponding to the sample feature vector to obtain an initial loss function value corresponding to each of the language expert units; Performing variance regularization calculation on the accent feature vector in the sample feature vector to obtain a variance regularization term; Performing covariance regularization calculation on the accent feature vector in the sample feature vector to obtain a covariance regularization term; The variance regularization term and the covariance regularization term are added to the initial loss function value to obtain the loss function value of each language expert unit in this round of training.

6. The speech synthesis method according to claim 5, characterized in that: The accent feature vector is obtained by encoding the accent category in the sample speech data using a preset accent encoder.

7. The speech synthesis method according to claim 3, characterized in that: After performing speech synthesis according to the second prediction feature information and the first prediction feature information corresponding to each of the language expert units to obtain the predicted speech data corresponding to each of the language expert units, the method further includes: Extracting first speaker feature information from the sample speech data, and extracting second speaker feature information from the predicted speech data; Calculating the similarity between the first speaker feature information and the second speaker feature information; If the similarity is less than a preset value, the parameters of the speech synthesis model are adjusted and the next round of training is performed until the obtained similarity is greater than or equal to the preset value.

8. A speech synthesis device, characterized in that: include: A feature vector acquisition module, used to acquire initial feature vectors corresponding to at least two types of initial speech data; A model calling module, used for calling a trained speech synthesis model, wherein the speech synthesis model comprises a mixed expert module, wherein the mixed expert module comprises a language-independent expert unit and at least one language expert unit; A first feature extraction module is used to determine the target language expert unit corresponding to each of the initial speech data, input the initial feature vector corresponding to each of the initial speech data into the corresponding target language expert unit for feature extraction, and obtain the first feature information corresponding to each of the initial speech data; A second feature extraction module, used for inputting the initial feature vectors corresponding to all the initial speech data into the language-independent expert unit for feature extraction to obtain second feature information; a feature information determination module, configured to determine target feature information according to the second feature information and the first feature information corresponding to each type of initial voice data; The speech synthesis module is used to perform speech synthesis on the target feature information to obtain target speech data.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the speech synthesis method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the speech synthesis method according to any one of claims 1 to 7.

Images