Training of a speech synthesis model and speech synthesis method, device, apparatus and medium

Abstract

The application relates to an artificial intelligence technology and proposes a speech synthesis model training and speech synthesis method, device, equipment and medium. The method comprises the following steps: acquiring a data set containing a plurality of text-speech pairs; training a to-be-trained speech synthesis model by using the data set, acquiring synthesized speech predicted by a to-be-trained synthesis model of each target training node; calculating a reconstruction loss and a feature gap between sample speech and synthesized speech of each target training node; judging whether high-frequency noise exists in each synthesized speech; ranking the synthesis effect of the target synthesized speech according to the reconstruction loss and the feature gap; and determining the speech synthesis model corresponding to the synthesized speech with the optimal synthesis effect as the trained speech synthesis model according to the sorting result and an additional screening rule. The application realizes comprehensive screening of the optimal speech synthesis model in multiple dimensions of the reconstruction loss, the feature gap and whether high-frequency noise exists in the synthesized speech of the model of the training node.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a speech synthesis model training method and device, a speech synthesis method and device, equipment and a medium. BACKGROUND

[0002] Speech synthesis technology is a text-to-speech technology (TTS). The prior art uses a voiceprint recognition module based on an x-vector for speaker vector embedding. The speaker vector is input into a TTS speech synthesis system to obtain synthesized speech. That is, by inputting a reference speech of any target speaker and any specified text, the voice of the target speaker reading the text can be synthesized.

[0003] In the training process of the prior art, the final trained model is often determined after the model reaches convergence. Since the performance of the model is only considered in one dimension, the final model obtained may not be the best model in terms of speech synthesis effect. Moreover, the speech synthesis system needs to be trained by large-scale multi-speaker speech corpus data. The synthesis effect depends on high-quality large-scale speech data, which is difficult to collect and expensive. SUMMARY

[0004] To solve the technical problems that the converged model in the prior art is not necessarily the optimal model and the cost of obtaining training samples is high, the present application provides a speech synthesis model training method and device, a speech synthesis method and device, equipment and a medium, which mainly aims to select the optimal model in the training process and reduce the cost of obtaining samples.

[0005] To achieve the above-mentioned purpose, the present application provides a speech synthesis model training method, which comprises:

[0006] obtaining a data set, the data set comprising a plurality of text-speech pairs, each text-speech pair comprising a sample speech and a corresponding sample text;

[0007] training a to-be-trained speech synthesis model using the data set, and obtaining a synthesized speech predicted by the to-be-trained synthesis model of each target training node;

[0008] calculating a reconstruction loss and a feature gap between the sample speech and the synthesized speech corresponding to each target training node;

[0009] determining whether the synthesized speech corresponding to each target training node contains high-frequency noise;

[0010] ranking the synthesis effect of the target synthesized speech according to the reconstruction loss and the feature gap, wherein the target synthesized speech is the synthesized speech that does not contain high-frequency noise or the synthesized speech that contains high-frequency noise within a preset range;

[0011] According to the sorting result and the additional screening rule, a speech synthesis model corresponding to a target synthesized speech with the optimal synthesis effect is determined as the trained speech synthesis model.

[0012] In addition, to achieve the above-mentioned purpose, the present application also provides a speech synthesis method, which comprises:

[0013] obtaining a graph embedding vector of the text to be synthesized as a target graph embedding vector;

[0014] encoding the target graph embedding vector by using a graph encoder included in the trained speech synthesis model obtained by the training method of any one of the preceding speech synthesis models, to obtain a target hidden representation;

[0015] inputting the target hidden representation into a random duration predictor included in the trained speech synthesis model to perform phoneme duration prediction, to obtain a target predicted duration of a target phoneme;

[0016] converting the target hidden representation into a mean value and a variance of a target prior distribution by using a projection layer included in the trained speech synthesis model;

[0017] sampling a target latent variable from the mean value and the variance of the target prior distribution in combination with the target predicted duration;

[0018] encoding the target latent variable by using a flow-based decoder included in the trained speech synthesis model and synthesizing a target synthesized waveform in parallel, to obtain a target synthesized speech.

[0019] In addition, to achieve the above-mentioned purpose, the present application also provides a training device of a speech synthesis model, which comprises:

[0020] a data acquisition module, configured to acquire a data set, the data set comprising a plurality of text-speech pairs, each text-speech pair comprising a sample speech and a corresponding sample text;

[0021] a training module, configured to train a to-be-trained speech synthesis model by using the data set, and to acquire a synthesized speech predicted by the to-be-trained synthesis model of each target training node;

[0022] a calculation module, configured to calculate a reconstruction loss and a feature gap between the sample speech and the synthesized speech corresponding to each target training node;

[0023] a noise judgment module, configured to judge whether the synthesized speech corresponding to each target training node has high-frequency noise;

[0024] a sorting module, configured to sort the synthesis effects of the target synthesized speeches according to the reconstruction loss and the feature gap, wherein the target synthesized speeches are synthesized speeches without high-frequency noise or synthesized speeches with high-frequency noise within a preset range.

[0025] The selection module is configured to determine, according to the sorting result and an additional screening rule, a speech synthesis model corresponding to a target synthesized speech with the optimal synthesis effect as the trained speech synthesis model.

[0026] To achieve the above object, the present application further provides a computer device, which comprises a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, and the processor executes the steps of the speech synthesis model training method of any one of the preceding embodiments or the steps of the speech synthesis method of any one of the preceding embodiments when executing the computer readable instructions.

[0027] To achieve the above object, the present application further provides a computer readable storage medium, which stores computer readable instructions, and the computer readable instructions are executed by a processor to make the processor execute the steps of the speech synthesis model training method of any one of the preceding embodiments or the steps of the speech synthesis method of any one of the preceding embodiments.

[0028] The speech synthesis model training method and the speech synthesis method, device, equipment and medium provided by the present application comprehensively screen the optimal speech synthesis model from multiple dimensions of the reconstruction loss of the model of each training node, the feature gap, and whether the synthesized speech has high-frequency noise. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The figure is a flowchart of the speech synthesis model training method in an embodiment of the present application;

[0030] Figure 2 The figure is a structural block diagram of the speech synthesis model training device in an embodiment of the present application;

[0031] Figure 3 The figure is an internal structural block diagram of the computer device in an embodiment of the present application.

[0032] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0033] To make the object, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0034] Figure 1 FIG. 1 is a flowchart of a method for training a speech synthesis model according to an embodiment of the present application. As shown in FIG. 1, the method for training the speech synthesis model includes the following steps S100-S600. Figure 1

[0035] S100: Obtain a data set, the data set including a plurality of text-speech pairs, each text-speech pair including a sample speech and a corresponding sample text.

[0036] Specifically, the sample text of each text-speech pair in the data set is the speaking content in the corresponding sample speech.

[0037] The data set can be a user data set constructed from target user speech. The data set can also include a first data set and a second data set. The first data set is an open source data set, which is an existing data set in the prior art. The second data set is a user data set constructed from target user speech. The number of samples contained in the first data set is much larger than the number of samples contained in the second data set. Since the second solution uses an open source data set, the construction cost of training samples can be saved.

[0038] S200: Train the speech synthesis model to be trained using the data set, and obtain the synthesized speech predicted by the speech synthesis model to be trained of each target training node.

[0039] Specifically, the sample text and the corresponding sample speech in any one of the text-speech pairs in the data set are obtained. The alignment relationship between the sample text and the corresponding sample speech is obtained through monotonic alignment search. The first duration information of the phonemes contained in the sample text is obtained according to the alignment relationship.

[0040] The sample text is encoded by a text encoder or a graph encoder to obtain a corresponding hidden representation. The hidden representation is taken as an input of a random duration predictor, and the duration of the phonemes contained in the sample text is predicted by the random duration predictor to obtain second duration information of the phonemes.

[0041] A first loss function is calculated according to the difference between the first duration information and the second duration information, and the model parameters of the random duration predictor are iteratively updated according to the first loss function.

[0042] The first linear spectrum corresponding to the sample speech is encoded by a posterior encoder to obtain a posterior latent variable.

[0043] The posterior latent variable is decoded by a decoder to obtain a synthesized waveform corresponding to the synthesized speech.

[0044] ​The second loss function of the model is obtained, and the model parameters of modules such as the decoder and posterior encoder are iteratively updated based on the second loss function. The second loss function includes the reconstruction loss function. The first MEL spectrogram corresponding to the sample speech is obtained, the synthesized waveform of the synthesized speech is converted into the second MEL spectrogram, and the reconstruction loss function is calculated based on the difference between the first MEL spectrogram and the second MEL spectrogram.

[0045] By arbitrarily selecting the next text-speech pair, the above steps are repeated to continuously optimize the model parameters of the random duration predictor, decoder, and posterior encoder until the convergence condition is met.

[0046] Each iteration of the model training process constitutes a training node. Each iteration of the decoder generates a synthesized waveform corresponding to the synthesized speech.

[0047] S300: Calculate the reconstruction loss and feature gap between the sample speech and the synthesized speech corresponding to each target training node.

[0048] Specifically, the reconstruction loss is calculated based on the difference between the first and second Mel spectrograms. Feature gap refers to the feature difference between the sample speech and the corresponding synthesized speech. Specifically, feature gap is the speech feature difference, such as the difference between MFCC features, i.e., MCD (Mel cepstral distortion).

[0049] S400: Determine whether there is high-frequency noise in the synthesized speech corresponding to each target training node.

[0050] Specifically, audio frequencies above half the sampling rate are considered high-frequency noise. For example, with a sampling rate of 22kHz, frequencies above 11kHz are considered noise. Whether synthesized speech contains high-frequency noise can be detected using existing techniques.

[0051] S500: The synthesis effect of the target synthesized speech is ranked according to the reconstruction loss and feature gap, wherein the target synthesized speech is synthesized speech without high-frequency noise or synthesized speech with high-frequency noise within a preset range.

[0052] Specifically, the target synthesized speech is synthesized speech that contains no high-frequency noise or whose proportion of high-frequency noise is within a preset range. If the duration of high-frequency noise in the synthesized speech exceeds the preset range, it is determined to be non-target synthesized speech.

[0053] Reconstruction loss, feature gap, and the presence of high-frequency noise are multiple dimensions used to evaluate the synthesis quality of the synthesized speech generated at each training node. Theoretically, the less high-frequency noise, the smaller the reconstruction loss, and the smaller the feature gap, the better the synthesis quality.

[0054] For target synthesized speech, the smaller the reconstruction loss and feature difference, the better the synthesis effect. The sorting can be done in ascending or descending order based on the synthesis effect.

[0055] S600: Based on the ranking results and additional filtering rules, the speech synthesis model corresponding to the target synthesized speech with the best synthesis effect is determined to be the trained speech synthesis model.

[0056] Specifically, the additional selection rules can be the human evaluation of the synthesis effect of the target synthesized speech by the listeners. The optimal target synthesized speech is selected based on the ranking results and the additional selection rules, and the speech synthesis model corresponding to the optimal target synthesized speech is used as the final trained speech synthesis model.

[0057] Alternatively, you can directly select the speech synthesis model corresponding to the best-performing speech from the target synthesized speech based on the ranking results as the final trained speech synthesis model.

[0058] This application enables the comprehensive selection of the optimal speech synthesis model by considering multiple dimensions, including the reconstruction loss of the model at each training node, feature gap, and the presence of high-frequency noise in the synthesized speech.

[0059] In one embodiment, step S600 specifically includes:

[0060] Based on the ranking results, target synthesized speech that does not rank higher than a preset position is selected as candidate speech.

[0061] Obtain the listening evaluation results of the listening testers on the synthesis effect of the candidate speech;

[0062] Based on the listening test evaluation results, the speech synthesis model corresponding to the candidate speech with the best synthesis effect is selected as the trained speech synthesis model.

[0063] Specifically, the target synthesized speech is sorted in descending order according to its synthesis effect. Target synthesized speech not exceeding a preset ranking is selected as candidate speech. For example, the top 5 target synthesized speech in descending order is selected as candidate speech.

[0064] The selected candidate speech samples are sent to listeners for evaluation, and the listeners' evaluation results are received. The speech synthesis model corresponding to the candidate speech sample with the best evaluation result is selected as the final trained speech synthesis model.

[0065] The final trained speech synthesis model may not be the final speech synthesis model at the point of convergence. It may be an intermediate speech synthesis model obtained by adjusting the model parameters at a certain training node. Although the intermediate speech synthesis model has not reached the convergence condition, its synthesis effect has been comprehensively evaluated through multiple dimensions. Therefore, it is regarded as the final trained speech synthesis model.

[0066] In one embodiment, the dataset includes a first dataset and a second dataset;

[0067] Step S200 specifically includes:

[0068] Obtain a first dataset, wherein the first dataset includes a first number of first text-speech pairs, and each first text-speech pair includes a first sample speech and a corresponding first sample text.

[0069] The first dataset is used to train the speech synthesis model in the first iteration until the first convergence condition is met, thus obtaining the basic speech synthesis model.

[0070] Obtain a second dataset generated from the target user's speech, wherein the second dataset includes a second number of second text-speech pairs, each second text-speech pair including a second sample speech and a corresponding second sample, and the first number is greater than the second number.

[0071] The basic speech synthesis model is trained in a second iteration using the second dataset until the second convergence condition is met. The synthesized speech predicted by the basic synthesis model at each target training node during the second iteration training process is obtained.

[0072] Specifically, the first dataset is an open-source dataset, which is a dataset already available in existing technologies. More specifically, the first dataset is a standardized corpus of the target language and gender from the open-source dataset. The first dataset includes at least 10 hours of speech data recorded in a quiet environment.

[0073] The second dataset is a user dataset constructed from the speech of the target user. More specifically, given a recorded text, a certain number of speech data points of the target speaker (target user) are collected, such as at least 30 or 40, etc., and the specific number is not limited in this application.

[0074] The first dataset contains a significantly larger number of samples than the second dataset. Since the second approach uses an open-source dataset, it saves on the cost of constructing training samples.

[0075] Phase 1 training:

[0076] Obtain the first sample text and its corresponding first sample speech from any first text-speech pair in the first dataset. Obtain the alignment relationship between the first sample text and its corresponding first sample speech through monotonic alignment search. Based on the alignment relationship, obtain the first duration information of the phonemes contained in the first sample text.

[0077] The first sample text is encoded using a text encoder or graph encoder in the speech synthesis model to be trained to obtain the corresponding hidden representation. The hidden representation is then used as input to a stochastic duration predictor in the speech synthesis model to be trained. The stochastic duration predictor in the speech synthesis model to be trained predicts the duration of the phonemes contained in the first sample text to obtain the second duration information of the phonemes.

[0078] The first loss function is calculated based on the difference between the first duration information and the second duration information, and the model parameters of the random duration predictor in the speech synthesis model to be trained are iteratively updated based on the first loss function.

[0079] The posterior encoder in the speech synthesis model to be trained is used to encode the first linear spectrum corresponding to the first sample speech to obtain the posterior latent variable.

[0080] The posterior latent variables are decoded by the decoder in the speech synthesis model to be trained, and the synthesized waveform corresponding to the first synthesized speech is obtained.

[0081] The second loss function of the model is obtained, and the model parameters of modules such as the decoder and posterior encoder are iteratively updated based on the second loss function. The second loss function includes a reconstruction loss function. The first MEL spectrogram corresponding to the first sample speech is obtained, the synthesized waveform of the first synthesized speech is converted into the second MEL spectrogram, and the reconstruction loss function is calculated based on the difference between the first MEL spectrogram and the second MEL spectrogram.

[0082] By arbitrarily selecting the next first text-speech pair, the above steps are performed to continuously optimize the model parameters of the random duration predictor, decoder, and posterior encoder in the speech synthesis model to be trained until the first convergence condition is met, thus obtaining the basic speech synthesis model.

[0083] Second training phase:

[0084] Obtain the second sample text and its corresponding second sample speech from any second text-speech pair in the second dataset. Obtain the alignment relationship between the second sample text and its corresponding second sample speech through monotonic alignment search. Based on the alignment relationship, obtain the first duration information of the phonemes contained in the second sample text.

[0085] The second sample text is encoded using a text encoder or graph encoder in the basic speech synthesis model to obtain the corresponding hidden representation. This hidden representation is then used as input to a stochastic duration predictor in the basic speech synthesis model. The stochastic duration predictor then predicts the duration of the phonemes contained in the second sample text to obtain the second duration information of the phonemes.

[0086] The first loss function is calculated based on the difference between the first duration information and the second duration information, and the model parameters of the random duration predictor in the basic speech synthesis model are iteratively updated based on the first loss function.

[0087] The posterior encoder in the basic speech synthesis model is used to encode the first linear spectrum corresponding to the second sample speech to obtain the posterior latent variable.

[0088] The synthesized waveform corresponding to the second synthesized speech is obtained by decoding the posterior latent variables using the decoder in the basic speech synthesis model.

[0089] The second loss function of the model is obtained, and the model parameters of modules such as the decoder and posterior encoder are iteratively updated based on the second loss function. The second loss function includes the reconstruction loss function. The first MEL spectrogram corresponding to the second sample speech is obtained, the synthesized waveform of the second synthesized speech is converted into the second MEL spectrogram, and the reconstruction loss function is calculated based on the difference between the first MEL spectrogram and the second MEL spectrogram.

[0090] By arbitrarily selecting the next second text-speech pair, the above steps are performed to continuously optimize the model parameters of the random duration predictor, decoder, and posterior encoder in the basic speech synthesis model until the second convergence condition is met.

[0091] Obtain the synthesized waveform corresponding to the synthesized speech generated by the decoder in the basic speech synthesis model during each iteration of model training in the second training phase. That is, the target training node is the training node of the second training node.

[0092] The loss function of the model, the maximum log-likelihood function, is shown in the following formula (1):

[0093]

[0094] Where m is the number of sample points, p G This represents the probability of generating sample points based on the base generative model. Let be the log-likelihood function of Gauss. |W k | represents the log-likelihood function obtained from the variable transformation theorem, #layers represents the number of layers, det represents the determinant, and W represents the log-likelihood function. k These are the model parameters.

[0095] This embodiment, based on a basic speech generation model, collects a small amount of data from the target speaker to quickly clone the speaker's voice, achieving good transfer learning and enabling rapid model convergence. It realizes a speech synthesis system with a small data volume. It provides TTS services for any text from the target speaker; the model training time is short, allowing for rapid deployment to new speakers; and it can be reused for any speaker, enabling rapid cloning and replication of multiple speakers' voices.

[0096] Furthermore, in the process of transfer learning, the transfer performance of the model is generally not directly related to the model's loss function. Therefore, this application can comprehensively evaluate the speech synthesis model obtained during training through multiple dimensions such as reconstruction loss, MCD, and whether high-frequency noise is generated, so as to achieve intelligent selection of the optimal model.

[0097] Furthermore, by training the basic speech synthesis model using a second dataset generated from the speech of different target users, a final speech synthesis model for synthesizing speech from different target users can be generated. This achieves the training of a speech synthesis model for any target speaker.

[0098] In one embodiment, step S200 specifically includes:

[0099] Obtain the graph embedding vector of the sample text;

[0100] The graph embedding vectors are encoded using the graph encoder contained in the speech synthesis model to be trained to obtain the hidden representation;

[0101] The hidden representation is used as input to the projection layer, and the mean and variance of the prior distribution are constructed using the projection layer.

[0102] Obtain the first linear spectrum of the sample speech corresponding to the sample text;

[0103] The posterior latent variables are obtained by encoding the first linear spectrum using the posterior encoder contained in the speech synthesis model to be trained.

[0104] The posterior latent variables are decoded using the decoder contained in the speech synthesis model to be trained, and the synthesized waveform corresponding to the synthesized speech is obtained.

[0105] The posterior latent variables are parameterized using the standardized flow to obtain the parameterized latent variables.

[0106] The alignment between parameterized latent variables and the mean and variance of the prior distribution is estimated by using monotonic alignment search, thus obtaining the alignment relationship between sample text and corresponding sample speech.

[0107] Obtain the first duration information of the phonemes corresponding to the sample text based on the alignment relationship;

[0108] The hidden representation is used as the input to the random duration predictor contained in the speech synthesis model to be trained. The random duration predictor is used to predict the duration of the corresponding phonemes in the sample text to obtain the second duration information of the phonemes.

[0109] The first loss function is calculated based on the first duration information and the second duration information. The model parameters of the random duration predictor are then iteratively updated based on the first loss function until the convergence condition is met.

[0110] Specifically, the projection layer is a linear projection layer used to construct the mean and variance of the prior distribution using the hidden representation. More specifically, the projection layer contains an affine function, which is used to project the hidden representation to obtain the mean and variance of the prior distribution corresponding to the sample text.

[0111] A portion of the sequence from the posterior latent variables is used as input to the decoder in the speech synthesis model to be trained. The decoder decodes the portion of the sequence from the posterior latent variables to obtain the synthesized waveform in the waveform domain corresponding to the synthesized speech. The synthesized waveform is the synthesized speech.

[0112] The alignment relation, or alignment matrix, represents the alignment between text and sound spectrum. Normalizing flow increases the encoder's expressive power. The Monotonic Alignment Search (MAS) algorithm (an alignment search algorithm) obtains the optimal alignment matrix between the mean and variance of the predicted prior distribution after text encoding and the normal distribution of the posterior latent variables after passing through the flow. Based on the alignment relation, the duration labels of the phonemes contained in the sample text can be obtained; these duration labels represent the first duration information of the phonemes.

[0113] This embodiment also trains a random duration predictor, which predicts the phoneme durations contained in the text to match duration tags calculated from alignment relationships. The phoneme duration prediction capability of the random duration predictor is corrected and trained based on alignment relationships obtained from monotonic alignment search.

[0114] In addition, the reconstruction loss function can be calculated based on the first and second MEL spectra, the gradient can be obtained based on the reconstruction loss function, and the model parameters of other modules of the model can be iteratively updated based on the gradient.

[0115] In addition, this embodiment uses a linear spectrum as the input to the posterior encoder. Compared with the Mel spectrum, the linear spectrum is high-resolution information, which can improve the performance of the speech synthesis model.

[0116] In one embodiment, the method further includes: obtaining a first mel spectrogram of the sample speech corresponding to the sample text, and obtaining a second mel spectrogram corresponding to the synthesized waveform;

[0117] Step S300 specifically includes:

[0118] The reconstruction loss is calculated based on the first MEL spectrogram of the sample speech and the second MEL spectrogram of the corresponding synthesized speech.

[0119] The first MFCC feature sequence of the sample speech is obtained, and the second MFCC feature sequence of the synthesized speech is obtained based on the synthesized waveform of the corresponding synthesized speech.

[0120] Align the first MFCC feature sequence with the second MFCC feature sequence to obtain the feature difference between the sample speech and the corresponding synthesized speech.

[0121] Specifically, the objective evaluation standard for speech conversion is MCD (Mel Cepstral Distortion), which represents the difference between the MFCC features of the converted speech and the MFCC features of the original speech. Measuring MCD requires aligning the MFCC feature sequences of the converted and original speech.

[0122] The reconstruction loss function in this embodiment is shown in Equation (2):

[0123]

[0124] in, Let m represent the predicted MEL spectrum, i.e., the second MEL spectrum, and m represent the true MEL spectrum, i.e., the first MEL spectrum. || represents the L1 regularization loss, and |||| represents the L2 regularization loss.

[0125] In one embodiment, the graph embedding vector includes multiple nodes;

[0126] The graph embedding vectors are encoded using the graph encoder contained in the speech synthesis model to be trained, resulting in hidden representations including:

[0127] Message computation is performed on each node to obtain the message computation information corresponding to each node.

[0128] For each node, message aggregation is performed on the message computation information of all neighboring nodes to obtain the node aggregation information of the corresponding node, and the node aggregation information is used as the hidden representation.

[0129] Specifically, the graph encoder uses text graph embeddings as input and is constructed from GNN networks such as GAT, GCN, or GGNN. The encoding process of the graph encoder consists of two steps: message computation and message aggregation. Taking the GCN model used in this proposal as an example:

[0130] (1) Message computation

[0131] Message computation is performed on each node, and the message computation information of each node is shown in formula (3):

[0132]

[0133] (2) Message aggregation

[0134] The message information of all neighboring nodes corresponding to each node is summed, and then a nonlinear activation function is applied to obtain the node aggregation information of that node as shown in formula (4):

[0135]

[0136] The mathematical expression of the GCN model is shown in formula (5):

[0137]

[0138] in, The node aggregation information for the current node is defined by σ, where σ is the sigmoid activation function, u represents neighboring nodes, N(v) is the set of neighboring nodes of the current node, and W represents the set of neighboring nodes of the current node. (l) The parameters of the GCN model learned by the model. Aggregate node information for the previous node of the current node. The message calculation information for the current node.

[0139] This application also provides a speech synthesis method, the method comprising:

[0140] Obtain the graph embedding vector of the text to be synthesized as the target graph embedding vector;

[0141] The target graph embedding vector is encoded using the graph encoder contained in the trained speech synthesis model obtained according to the training method of any of the previous speech synthesis models to obtain the target hidden representation;

[0142] The target hidden representation is input into the random duration predictor contained in the trained speech synthesis model to predict the phoneme duration, and the target predicted duration of the target phoneme is obtained.

[0143] The hidden representation of the target is transformed into the mean and variance of the prior distribution of the target through the projection layer contained in the trained speech synthesis model;

[0144] The latent variables of the target are sampled from the mean and variance of the target prior distribution by combining the target prediction duration;

[0145] The target synthesized speech is obtained by encoding the target latent variables and synthesizing the target synthesized waveform in parallel using the stream-based decoder contained in the trained speech synthesis model.

[0146] Specifically, during the inference process, the graph encoder and the random duration predictor predict the statistical information of the prior distribution and alignment. Then, latent variables are sampled from the prior distribution, transformed by a stream-based decoder, and the target synthesized waveform corresponding to the synthesized speech is generated in parallel. The target synthesized speech is then obtained based on the target synthesized waveform. This embodiment can construct a TTS system for a target speaker, and can complete TTS speech synthesis for any given text.

[0147] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0148] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0149] Figure 3 This is a structural block diagram of a training device for a speech synthesis model according to one embodiment of this application. (Reference) Figure 3 The device includes:

[0150] Data acquisition module 100 is used to acquire a dataset, which includes multiple text-speech pairs, each of which includes sample speech and corresponding sample text.

[0151] The training module 200 is used to train the speech synthesis model to be trained using the dataset and to obtain the synthesized speech predicted by the synthesis model to be trained for each target training node.

[0152] The calculation module 300 is used to calculate the reconstruction loss and feature gap between the sample speech and the synthesized speech corresponding to each target training node;

[0153] The noise detection module 400 is used to determine whether there is high-frequency noise in the synthesized speech corresponding to each target training node;

[0154] The sorting module 500 is used to sort the synthesis results of synthesized speech without high-frequency noise based on reconstruction loss and feature difference.

[0155] The selection module 600 is used to determine the speech synthesis model corresponding to the target synthesized speech with the best synthesis effect as the trained speech synthesis model based on the sorting results and additional filtering rules.

[0156] In one embodiment, the selection module 600 specifically includes:

[0157] The first filtering module is used to select target synthesized speech whose synthesis effect ranking does not exceed a preset ranking as candidate speech based on the sorting results.

[0158] The listening evaluation module is used to obtain the listening evaluation results of the listening testers on the synthesis effect of the candidate speech;

[0159] The second screening module is used to select the speech synthesis model corresponding to the candidate speech with the best synthesis effect as the trained speech synthesis model based on the listening test evaluation results.

[0160] In one embodiment, the dataset includes a first dataset and a second dataset;

[0161] Training module 200 specifically includes:

[0162] The first data acquisition module is used to acquire a first dataset, wherein the first dataset includes a first number of first text-speech pairs, and each first text-speech pair includes a first sample speech and a corresponding first sample text.

[0163] The first training module is used to perform the first iteration of training on the speech synthesis model to be trained using the first dataset until the first convergence condition is met, thereby obtaining the basic speech synthesis model.

[0164] The second data acquisition module is used to acquire a second dataset generated from the target user's speech. The second dataset includes a second number of second text-to-speech pairs, each pair including a second sample speech and a corresponding second sample. The first number is greater than the second number.

[0165] The second training module is used to perform a second iteration of training on the basic speech synthesis model using the second dataset until the second convergence condition is met, and to obtain the synthesized speech predicted by the basic synthesis model for each target training node during the second iteration of training.

[0166] In one embodiment, the training module 200 specifically includes:

[0167] The first transformation module is used to obtain the graph embedding vector of the sample text;

[0168] The first encoding module is used to encode the graph embedding vector using the graph encoder contained in the speech synthesis model to be trained, so as to obtain the hidden representation;

[0169] The first projection module is used to take the hidden representation as input to the projection layer and use the projection layer to construct the mean and variance of the prior distribution.

[0170] The second conversion module is used to obtain the first linear spectrum of the sample speech corresponding to the sample text;

[0171] The second encoding module is used to encode the first linear spectrum through the posterior encoder contained in the speech synthesis model to be trained, so as to obtain the posterior latent variables.

[0172] The decoding module is used to decode the posterior latent variables using the decoder contained in the speech synthesis model to be trained, so as to obtain the synthesized waveform corresponding to the synthesized speech.

[0173] The first standard flow module is used to parameterize the posterior latent variables using the standardized flow to obtain the parameterized latent variables.

[0174] The alignment estimation module is used to estimate the alignment between parameterized latent variables and the mean and variance of the prior distribution using monotonic alignment search, so as to obtain the alignment relationship between the sample text and the corresponding sample speech.

[0175] The duration acquisition module is used to obtain the first duration information of the phonemes corresponding to the sample text based on the alignment relationship;

[0176] The first duration prediction module is used to take the hidden representation as input to the random duration predictor contained in the speech synthesis model to be trained, and use the random duration predictor to predict the duration of the phonemes corresponding to the sample text to obtain the second duration information of the phonemes.

[0177] The parameter update module is used to calculate the first loss function based on the first duration information and the second duration information, and to iteratively update the model parameters of the random duration predictor based on the first loss function until the convergence condition is met.

[0178] In one embodiment, the device further includes:

[0179] The third conversion module is used to obtain the first mel spectrogram of the sample speech corresponding to the sample text and the second mel spectrogram corresponding to the synthesized waveform.

[0180] The calculation module 300 specifically includes:

[0181] The reconstruction loss calculation module is used to calculate the reconstruction loss based on the first mel spectrogram of the sample speech and the second mel spectrogram of the corresponding synthesized speech.

[0182] The feature extraction module is used to obtain the first MFCC feature sequence of the sample speech, and to obtain the second MFCC feature sequence of the synthesized speech based on the synthesized waveform of the corresponding synthesized speech.

[0183] The feature gap calculation module is used to align the first MFCC feature sequence with the second MFCC feature sequence to obtain the feature gap between the sample speech and the corresponding synthesized speech.

[0184] In one embodiment, the graph embedding vector includes multiple nodes;

[0185] The first encoding module specifically includes:

[0186] The message computation module is used to perform message computation on each node and obtain the message computation information corresponding to each node.

[0187] The message aggregation module is used to aggregate the message calculation information of all neighboring nodes corresponding to each node, obtain the node aggregation information of the corresponding node, and use the node aggregation information as a hidden representation.

[0188] This application also provides a speech synthesis device, the device comprising:

[0189] The fourth conversion module is used to obtain the graph embedding vector of the text to be synthesized as the target graph embedding vector;

[0190] The third encoding module is used to encode the target graph embedding vector using the graph encoder contained in the trained speech synthesis model to obtain the target hidden representation;

[0191] The second duration prediction module is used to input the target hidden representation into the random duration predictor contained in the trained speech synthesis model to predict the phoneme duration and obtain the target predicted duration of the target phoneme.

[0192] The second projection module is used to convert the target hidden representation into the mean and variance of the target prior distribution through the projection layer contained in the trained speech synthesis model.

[0193] The sampling module is used to sample latent variables of the target from the mean and variance of the target prior distribution, in conjunction with the target prediction time.

[0194] The synthesis module is used to encode the target latent variables and synthesize the target synthesized waveform in parallel using the stream-based decoder contained in the trained speech synthesis model to obtain the target synthesized speech.

[0195] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0196] The terms "first" and "second" in the above-mentioned modules / units are only used to distinguish different modules / units and are not intended to specify which module / unit has a higher priority or any other limiting meaning. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The module divisions appearing in this application are merely logical divisions; in actual applications, different division methods may be used.

[0197] Specific limitations regarding the training device for the speech synthesis model can be found in the limitations on the training method for the speech synthesis model above, and will not be repeated here. Each module in the aforementioned training device for the speech synthesis model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0198] Figure 3 This is a block diagram of the internal structure of a computer device according to an embodiment of this application. Figure 3 As shown, the computer device includes a processor, memory, network interface, input device, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes a storage medium and internal memory. The storage medium can be non-volatile or volatile. The storage medium stores an operating system and may also store computer-readable instructions. When executed by the processor, these instructions enable the processor to implement a training method for a speech synthesis model, or a speech synthesis method. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the storage medium. The internal memory may also store computer-readable instructions, which, when executed by the processor, enable the processor to execute a training method for a speech synthesis model, or a speech synthesis method. The network interface of the computer device is used for communication with an external server via a network connection. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0199] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions (e.g., a computer program) stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, it implements the training method for the speech synthesis model described in the above embodiment (e.g., ...). Figure 1 The steps S100 to S600 shown, and other extensions and related steps of the method, or the steps of the speech synthesis method. Alternatively, when the processor executes computer-readable instructions, it implements the functions of each module / unit of the training device for the speech synthesis model in the above embodiments (e.g., Figure 2 The functions of modules 100 to 600 shown, or the functions of each module / unit of the speech synthesis device. To avoid repetition, they will not be described again here.

[0200] A processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of a computer device, connecting all parts of the computer device through various interfaces and lines.

[0201] Memory can be used to store computer-readable instructions and / or modules. The processor implements various functions of the computer device by running or executing the computer-readable instructions and / or modules stored in memory, and by accessing data stored in memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as audio data, video data, etc.).

[0202] The memory can be integrated into the processor or set up separately from the processor.

[0203] Those skilled in the art will understand that Figure 3The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0204] In one embodiment, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When executed by a processor, the computer-readable instructions implement the training method of the speech synthesis model in the above embodiment (e.g., Figure 1 The steps S100 to S600 shown, and other extensions and related steps of the method, or the steps of the speech synthesis method. Alternatively, when computer-readable instructions are executed by a processor, they implement the functions of each module / unit of the training device for the speech synthesis model in the above embodiments (e.g., Figure 2 The functions of modules 100 to 600 shown, or the functions of each module / unit of the speech synthesis device, will not be described here to avoid repetition.

[0205] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium, and when executed, they can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0206] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0207] The sequence numbers of the embodiments in this application are merely for description and do not represent the superiority or inferiority of the embodiments. Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0208] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A training method for a speech synthesis model, characterized in that, The method includes: Obtain a dataset, which includes multiple text-speech pairs, each of which includes sample speech and corresponding sample text; The dataset is used to train the speech synthesis model to be trained, and the synthesized speech predicted by the speech synthesis model to be trained for each target training node is obtained. Each iteration of training in the model training process is a training node. Calculate the reconstruction loss and feature gap between the sample speech and the synthesized speech corresponding to each target training node; Determine whether there is high-frequency noise in the synthesized speech corresponding to each target training node; audio with a sampling rate higher than half is considered high-frequency noise. The synthesis effect of the target synthesized speech is ranked according to the reconstruction loss and feature gap, wherein the target synthesized speech is synthesized speech without high-frequency noise or synthesized speech with high-frequency noise within a preset range. Based on the ranking results and additional screening rules, the speech synthesis model corresponding to the target synthesized speech with the best synthesis effect is determined as the trained speech synthesis model. The additional screening rules are the human evaluation of the synthesis effect of the target synthesized speech by the listeners.

2. The method according to claim 1, characterized in that, The speech synthesis model corresponding to the target synthesized speech with the best synthesis effect, determined according to the ranking results and additional filtering rules, is a trained speech synthesis model, including: Based on the ranking results, target synthesized speech that does not rank higher than a preset position is selected as candidate speech. Obtain the listening evaluation results of the listening test personnel on the synthesis effect of the candidate speech; Based on the listening test evaluation results, the speech synthesis model corresponding to the candidate speech with the best synthesis effect is selected as the trained speech synthesis model.

3. The method according to claim 1, characterized in that, The dataset includes a first dataset and a second dataset; The step of training the speech synthesis model to be trained using the dataset and obtaining the synthesized speech predicted by the speech synthesis model to be trained at each target training node includes: Obtain a first dataset, wherein the first dataset includes a first number of first text-speech pairs, each first text-speech pair includes a first sample speech and a corresponding first sample text, and the first dataset is a standardized corpus of the target language and target gender in an open source dataset; The first dataset is used to train the speech synthesis model to be trained in the first iteration until the first convergence condition is met, thus obtaining the basic speech synthesis model. Obtain a second dataset generated from the target user's speech, wherein the second dataset includes a second number of second text-to-speech pairs, each second text-to-speech pair including a second sample speech and a corresponding second sample, and the first number is greater than the second number. The basic speech synthesis model is trained in a second iteration using the second dataset until the second convergence condition is met, and the synthesized speech predicted by the basic speech synthesis model at each target training node during the second iteration training process is obtained.

4. The method according to claim 1, characterized in that, The step of training the speech synthesis model to be trained using the dataset and obtaining the synthesized speech predicted by the speech synthesis model to be trained at each target training node includes: Obtain the graph embedding vector of the sample text; The graph embedding vector is encoded using the graph encoder contained in the speech synthesis model to be trained to obtain the hidden representation; The hidden representation is used as input to the projection layer, and the mean and variance of the prior distribution are constructed using the projection layer. Obtain the first linear spectrum of the sample speech corresponding to the sample text; The first linear spectrum is encoded by the posterior encoder contained in the speech synthesis model to be trained to obtain the posterior latent variable; The posterior latent variables are decoded using the decoder contained in the speech synthesis model to be trained, and the synthesized waveform corresponding to the synthesized speech is obtained. The posterior latent variables are parameterized using a standardized flow to obtain parameterized latent variables. The alignment between the parameterized latent variables and the mean and variance of the prior distribution is estimated using monotonic alignment search to obtain the alignment relationship between the sample text and the corresponding sample speech. The first duration information of the phonemes corresponding to the sample text is obtained according to the alignment relationship; The hidden representation is used as the input to the random duration predictor included in the speech synthesis model to be trained. The random duration predictor is used to predict the duration of the phonemes corresponding to the sample text to obtain the second duration information of the phonemes. The first loss function is calculated based on the first duration information and the second duration information. The model parameters of the random duration predictor are iteratively updated based on the first loss function until the convergence condition is met.

5. The method according to claim 4, characterized in that, The method further includes: Obtain the first mel spectrogram of the sample speech corresponding to the sample text, and obtain the second mel spectrogram corresponding to the synthesized waveform; The calculation of the reconstruction loss and feature gap between the sample speech and the synthesized speech corresponding to each target training node includes: The reconstruction loss is calculated based on the first mel spectrogram of the sample speech and the second mel spectrogram of the corresponding synthesized speech. The first MFCC feature sequence of the sample speech is obtained, and the second MFCC feature sequence of the synthesized speech is obtained based on the synthesized waveform of the corresponding synthesized speech. Align the first MFCC feature sequence with the second MFCC feature sequence to obtain the feature difference between the sample speech and the corresponding synthesized speech.

6. The method according to claim 4, characterized in that, The graph embedding vector includes multiple nodes; The process of encoding the graph embedding vector using the graph encoder contained in the speech synthesis model to be trained, to obtain the hidden representation, includes: Message computation is performed on each node to obtain message computation information corresponding to each node. For each node, message aggregation is performed on the message calculation information of all neighboring nodes to obtain the node aggregation information of the corresponding node, and the node aggregation information is used as a hidden representation.

7. A speech synthesis method, characterized in that, The method includes: Obtain the graph embedding vector of the text to be synthesized as the target graph embedding vector; The target graph embedding vector is encoded using the graph encoder contained in the trained speech synthesis model obtained by the training method of any one of claims 1-6 to obtain the target hidden representation; The target hidden representation is input into the random duration predictor contained in the trained speech synthesis model to predict the phoneme duration, thereby obtaining the target predicted duration of the target phoneme. The target hidden representation is converted into the mean and variance of the target prior distribution through the projection layer contained in the trained speech synthesis model; The latent variable of the target is sampled from the mean and variance of the prior distribution of the target, in conjunction with the target prediction duration; The target latent variables are encoded and the target synthesized waveform is synthesized in parallel using the stream-based decoder contained in the trained speech synthesis model to obtain the target synthesized speech.

8. A training device for a speech synthesis model, characterized in that, The device includes: The data acquisition module is used to acquire a dataset, which includes multiple text-speech pairs, each of which includes sample speech and corresponding sample text. The training module is used to train the speech synthesis model to be trained using the dataset, and to obtain the synthesized speech predicted by the speech synthesis model to be trained for each target training node. Each iteration of training during the model training process is a training node. The calculation module is used to calculate the reconstruction loss and feature gap between the sample speech and the synthesized speech corresponding to each target training node; The noise detection module is used to determine whether there is high-frequency noise in the synthesized speech corresponding to each target training node. Audio with a sampling rate higher than half is considered as high-frequency noise. The sorting module is used to sort the synthesis effect of the target synthesized speech according to the reconstruction loss and feature gap, wherein the target synthesized speech is synthesized speech without high-frequency noise or synthesized speech with high-frequency noise within a preset range. The selection module is used to determine the speech synthesis model corresponding to the target synthesized speech with the best synthesis effect based on the ranking results and additional filtering rules. The additional filtering rules are the human evaluation of the synthesis effect of the target synthesized speech by the listeners.

9. A computer device, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-readable instructions, it performs the steps of the training method for the speech synthesis model as described in any one of claims 1-6, or, when the processor executes the computer-readable instructions, it performs the steps of the speech synthesis method as described in claim 7.

10. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, When the computer-readable instructions are executed by the processor, the processor performs the steps of the training method for the speech synthesis model as described in any one of claims 1-6, or the processor performs the steps of the speech synthesis method as described in claim 7.

Images