Speech synthesis method and device, computer equipment and storage medium

Abstract

The invention relates to a speech synthesis method and device, computer equipment and a storage medium. The method comprises the steps of obtaining to-be-synthesized speaker voice data, and extracting speaker features from the to-be-synthesized speaker voice data; obtaining text data, converting the text data into a phoneme sequence, and inserting a position code into the phoneme sequence to obtain a sorted phoneme sequence; inputting the text data and the sorted phoneme sequence into a graph encoder to obtain text encoding features output by the graph encoder; splicing the speaker features and the text coding features to obtain spliced features, and inputting the spliced features into a variational adapter for duration prediction to obtain predicted features; and inserting position codes into the prediction features to obtain sorted prediction features, inputting the sorted prediction features into a decoder to obtain a to-be-synthesized Mel spectrogram output by the decoder, and converting the to-be-synthesized Mel spectrogram into speaker synthesized audio. By adopting the method, the accuracy of speech synthesis can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech synthesis method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of speech processing technology, text-to-speech (TTS) technology has emerged, which can also be called speech synthesis technology. As an important part of intelligent question-answering systems, this technology has made great progress in the past two decades. In recent years, neural network methods and sequence-to-sequence models have enabled the development of TTS to a certain level. Although neural network-based TTS has shown competitive performance, there is still room for improvement in speech synthesis accuracy. In traditional speaker speech synthesis or cloning methods, the use of speaker encoders to extract speaker embedding vectors may face the problem of inaccurate extraction of speaker embedding vectors, resulting in low accuracy and poor effect of speech synthesis. Summary of the invention

[0003] Based on this, it is necessary to provide a speech synthesis method, apparatus, computer device and storage medium that can improve the accuracy of speech synthesis in response to the above technical problems.

[0004] In a first aspect of the present application, a speech synthesis method is provided, which includes: obtaining speech data of a speaker to be synthesized, and extracting speaker features from the speech data of the speaker to be synthesized; obtaining text data, converting the text data into a phoneme sequence, and inserting position codes into the phoneme sequence to obtain a sorted phoneme sequence; inputting the text data and the sorted phoneme sequence into a graph encoder to obtain text encoding features output by the graph encoder; wherein the graph encoder encodes the grammatical relationship of the text data based on the grammatical graph of the text data, and performs text encoding on the sorted phoneme sequence in combination with a graph attention mechanism; concatenating the speaker features with the text encoding features to obtain concatenated features, and inputting the concatenated features into a variational adapter for duration prediction to obtain predicted features; inserting position codes into the predicted features to obtain sorted predicted features, and inputting the sorted predicted features into a decoder to obtain a mel-spectrogram to be synthesized output by the decoder, and converting the mel-spectrogram to be synthesized into speaker synthesized audio.

[0005] In some embodiments, extracting speaker features from speaker speech data to be synthesized includes: extracting speaker embedding vectors from the speaker speech data to be synthesized using a speaker embedding extraction network; constructing an affinity graph based on the speaker embedding vectors and sample embedding vectors of training samples of the speaker embedding extraction network; wherein the affinity graph uses embedding vectors as nodes and uses at least one neighbor embedding vector that is the nearest neighbor to each embedding vector determined by cosine similarity as an edge; and clustering the nodes in the affinity graph based on a graph convolutional network to generate clusters of speakers to be synthesized.

[0006] In some embodiments, clustering processing is performed on nodes in an affinity graph based on a graph convolutional network to generate speaker clusters to be synthesized, including: clustering the nodes in the affinity graph to generate multiple speaker cluster proposals to be synthesized; a cluster detection unit based on the graph convolutional network extracts clustering features of each speaker cluster proposal to be synthesized, and identifies candidate speaker cluster proposals to be synthesized from the generated multiple speaker cluster proposals to be synthesized according to the clustering features; a cluster segmentation unit based on graph convolution predicts the probability value of each node in the candidate speaker cluster proposal to be synthesized, and removes nodes with probability values ​​less than a preset threshold as abnormal nodes from the nodes of the candidate speaker cluster proposal to be synthesized; and sorts each node in the candidate speaker cluster proposal to be synthesized according to the size of the probability value of each node, and selects at least one node with a probability value greater than the preset threshold to generate a speaker cluster proposal to be synthesized.

[0007] In some embodiments, the nodes in the affinity graph are clustered to generate multiple speaker cluster proposals to be synthesized, including: setting edge weights between nodes in the affinity graph to obtain multiple super nodes; performing clustering operations based on each super node, using the centroid of each super node as a node and the relationship between the centroids as an edge, to generate multiple speaker cluster proposals to be synthesized.

[0008] In some embodiments, the method further includes: assigning pseudo labels to speaker clusters to be synthesized; optimizing network parameters of a speaker embedding extraction network through a denoising loss function to denoise the pseudo labels generated during clustering processing, and re-inputting the denoised pseudo labels into the speaker embedding extraction network for iterative training.

[0009] In some embodiments, text data and a sorted phoneme sequence are input into a graph encoder to obtain text encoding features output by the graph encoder, including: generating a syntax tree corresponding to the text data based on the text data; parsing the grammatical relationship between words in the syntax tree, using phonemes as nodes and the grammatical relationship between phonemes as edges to generate a syntax graph; based on a bidirectional GRU network, bidirectionally encoding the grammatical relationship between two phonemes to obtain a relation encoding vector; wherein the relation encoding vector includes a forward relation encoding vector and a backward relation encoding vector; calculating an attention score based on the graph attention mechanism of the graph encoder and the relation encoding vector; encoding the sorted phoneme sequence according to the attention score to obtain text encoding features output by the graph encoder.

[0010] In some embodiments, an attention score is calculated based on a graph attention mechanism and a relation encoding vector of a graph encoder, including: capturing an addressing relationship based on basic grammatical content in relation encoding based on a graph attention mechanism of a graph encoder; calculating a forward relation bias between phonemes based on a forward relation encoding vector; controlling a backward relation bias between phonemes based on a backward relation encoding vector; calculating a comprehensive bias based on the forward relation encoding vector and the backward relation encoding vector; and obtaining an attention score based on the addressing relationship, the forward relation bias, the backward relation bias, and the comprehensive bias.

[0011] In some embodiments, based on a bidirectional GRU network, the grammatical relationship between two phonemes is bidirectionally encoded to obtain a relationship encoding vector, including: when two phonemes belong to the same word, based on the bidirectional GRU network and using a self-looping edge coding algorithm, the grammatical relationship between the two phonemes is bidirectionally encoded; when the two phonemes belong to different words, based on the bidirectional GRU network, the grammatical relationship between the words to which the two phonemes belong is bidirectionally encoded.

[0012] In a second aspect of the present application, a speech synthesis device is provided, the device comprising:

[0013] The speaker encoding module is used to obtain the speech data of the speaker to be synthesized and extract the speaker features from the speech data of the speaker to be synthesized;

[0014] A front-end processing module is used to obtain text data, convert the text data into a phoneme sequence, and insert a position code into the phoneme sequence to obtain a sorted phoneme sequence;

[0015] A graph encoding module, used for inputting text data and a sorted phoneme sequence into a graph encoder to obtain text encoding features output by the graph encoder;

[0016] A variational adaptation module is used to concatenate speaker features with text encoding features to obtain concatenated features, and input the concatenated features into a variational adapter for duration prediction to obtain predicted features;

[0017] The decoding and synthesis module is used to insert the position code into the prediction feature to obtain the sorted spliced ​​feature, input the sorted spliced ​​feature into the decoder, obtain the Mel-spectrogram to be synthesized output by the decoder, and convert the Mel-spectrogram to be synthesized into the speaker's synthesized audio.

[0018] According to a third aspect of the present application, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the speech synthesis method of any embodiment of the first aspect are implemented.

[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the speech synthesis method of any embodiment of the first aspect described above.

[0020] The above-mentioned speech synthesis method, device, computer equipment and storage medium use a graph encoder to encode the grammatical relationship of the text data based on the grammatical graph of the text data, and combine the graph attention mechanism to encode the sorted phoneme sequence corresponding to the text data to obtain text encoding features, and splice the text encoding features processed by the graph encoder with the speaker features to perform feature prediction and Mel-spectrogram prediction, thereby realizing the synthesis or cloning of the speaker's synthesized audio. The method adopts the graph encoder, introduces the grammatical relationship encoding based on the grammatical graph combined with the self-attention mechanism for text encoding, which can improve the accuracy of text encoding, thereby improving the accuracy of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of a speech synthesis method in some embodiments;

[0022] Figure 2 A schematic diagram of the connection structure of each network unit and the data flow direction for implementing the speech synthesis method in some embodiments;

[0023] Figure 3 A schematic diagram of data flow for extracting speaker features based on a graph speaker encoder in some embodiments;

[0024] Figure 4 A schematic diagram of data flow for performing a text feature encoder based on a graph encoder in some embodiments;

[0025] Figure 5 is a schematic diagram of the structure of a decoder in some embodiments;

[0026] Figure 6 A schematic diagram of the structure of a variational adapter in some embodiments;

[0027] Figure 7is a structural block diagram of a speech synthesis device in some embodiments;

[0028] Figure 8 1 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] The speech synthesis method provided in the present application can be applied to a computer device, which can be a server or a terminal. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, portable wearable devices, and intelligent terminal devices such as vehicle terminals, smart speakers, and smart robots. The server can be implemented as an independent server or a server cluster composed of multiple servers.

[0031] In some embodiments, Figure 1 As shown, a speech synthesis method is provided, which is described by taking the method applied to an intelligent terminal device as an example, and includes the following steps:

[0032] Step S102: obtaining speech data of the speaker to be synthesized, and extracting speaker features from the speech data of the speaker to be synthesized.

[0033] The speech data of the speaker to be synthesized refers to the speech data of the speaker to be synthesized or cloned as a sample. The speaker feature refers to the feature vector that can represent the speaker's speech information.

[0034] Specifically, the speaker features can be extracted from the input speaker speech data to be synthesized by a graph speaker encoder. Exemplarily, the speaker embedding can be extracted from the input speaker speech data to be synthesized by a speaker encoder in any traditional method as the speaker feature, or by a speaker feature extraction method based on a graph speaker encoder described in the embodiments below in this application.

[0035] Step S104: Acquire text data, convert the text data into a phoneme sequence, and insert a position code into the phoneme sequence to obtain a sorted phoneme sequence.

[0036] Specifically, the text data can be preprocessed by the front-end processing module. For example, it can be processed by text normalization, grapheme-to-phoneme conversion, and for Chinese text data, it can also be preprocessed by polyphone classification, prosody prediction, etc., and finally converted into a phoneme sequence.

[0037] Furthermore, position codes can be inserted into the phoneme sequence. Inserting position codes can be understood as concatenating the phoneme sequence with the position codes. Because for any language, the position and order of words in a sentence are very important and can affect the semantics of the sentence. Therefore, by inserting position codes, the input phoneme sequence can carry position information and generate a sorted phoneme sequence, so that the subsequent model network can find out its position characteristics.

[0038] Step S106: Input the text data and the sorted phoneme sequence into the graph encoder to obtain the text encoding features output by the graph encoder; wherein the graph encoder encodes the grammatical relationship of the text data based on the grammatical graph of the text data, and performs text encoding on the sorted phoneme sequence in combination with the graph attention mechanism.

[0039] Specifically, the sorted phoneme sequence corresponding to the preprocessed text data and the text data can be input into the graph encoder together. The graph encoder can include a relation encoding unit and a text encoder, wherein the relation encoding unit is used to perform grammatical relation encoding on the text data based on the grammatical graph of the text data, and the text encoder is used to combine the grammatical relation encoding with the graph attention mechanism to perform text encoding on the sorted phoneme sequence.

[0040] The graph encoder incorporates grammatical relation encoding into the graph attention mechanism to indicate character relations. The graph encoder aims to convert the input phoneme sequence and relation encoding into the corresponding syntax-driven character embedding sequence (text encoding feature).

[0041] Step S108: concatenate the speaker feature with the text encoding feature to obtain a concatenated feature, and input the concatenated feature into a variational adapter for duration prediction to obtain a predicted feature.

[0042] Specifically, the speaker features of the speaker to be synthesized extracted in step S102 and the text encoding features of the text data obtained in step S106 may be concatenated, and the concatenated features obtained after concatenation may be input into the variational adapter for duration prediction to obtain predicted features.

[0043] Step S110: inserting position codes into the prediction features to obtain sorted prediction features, inputting the sorted prediction features into a decoder to obtain a mel-spectrogram to be synthesized output by the decoder, and converting the mel-spectrogram to be synthesized into speaker synthesized audio.

[0044] Specifically, the position code is concatenated to the prediction feature to obtain the sorted prediction feature with the position code inserted, the sorted prediction feature is decoded by a decoder to predict the Mel-spectrogram to be synthesized of the speaker to be synthesized, and then the Mel-spectrogram to be synthesized is converted into the audio of the speaker to be synthesized by a vocoder based on the predicted Mel-spectrogram to be synthesized.

[0045] The above-mentioned speech synthesis method adopts a graph encoder to encode the grammatical relationship of the text data based on the grammatical graph of the text data, and combines the graph attention mechanism to encode the sorted phoneme sequence corresponding to the text data to obtain text encoding features, and splices the text encoding features processed by the graph encoder with the speaker features to perform feature prediction and prediction of the mel-spectrogram, thereby realizing the synthesis or cloning of the speaker synthesized audio. By adopting this method, a graph encoder is introduced, and the grammatical relationship encoding based on the grammatical graph is combined with the self-attention mechanism for text encoding, which can improve the accuracy of text encoding, thereby improving the accuracy of speech synthesis.

[0046] Below, reference Figure 2 As shown, Figure 2 The following is a schematic diagram showing the connection structure of each network unit and the data flow for implementing the speech synthesis method in some application examples. Specifically,

[0047] Step 1: Input the speech data of the speaker to be synthesized into the speaker encoder to extract the speaker features of the speaker to be synthesized or cloned.

[0048] Step 2: The input text data enters the front-end processing module, undergoes text regularization, character-phonetic conversion, Chinese characters are classified into polyphones, and rhythm prediction, and finally converted into a phoneme sequence. Furthermore, the phoneme sequence can be processed by character embedding.

[0049] Step 3: Insert the positional encoding into the output after step 2.

[0050] Step 4: Input the text data and the output of step 3 into the graph encoder.

[0051] Step 5: The features output from step 4 are concatenated with the speaker features extracted from step 1 and fed into the variational adapter for duration prediction.

[0052] Step 6: Insert the position code into the output features of step 5 and enter the decoder.

[0053] Step 7: Generate the Mel-spectrogram to be synthesized or cloned through decoder prediction.

[0054] Step 9: Convert the Mel-spectrogram to the audio of the speaker to be synthesized or cloned through griffin-lim or using a neural vocoder.

[0055] In some embodiments, extracting speaker features from speaker speech data to be synthesized includes: extracting speaker embedding vectors from the speaker speech data to be synthesized using a speaker embedding extraction network; constructing an affinity graph based on the speaker embedding vectors and sample embedding vectors of training samples of the speaker embedding extraction network; wherein the affinity graph uses embedding vectors as nodes and uses at least one neighbor embedding vector that is the nearest neighbor to each embedding vector determined by cosine similarity as an edge; and clustering the nodes in the affinity graph based on a graph convolutional network to generate clusters of speakers to be synthesized.

[0056] Exemplarily, the speaker embedding extraction network can be trained in a supervised manner using historical speaker speech data as samples.

[0057] For example, the affinity graph, G = (V, E) is a relationship graph, where the node V represents the input sample and the edge E represents the similarity between the data. Based on the embedding vector extracted from the pre-trained speaker embedding extraction network, each sample (or the embedding vector corresponding to the sample) is regarded as a node, and the cosine similarity is used to find the K nearest neighbor samples for each sample. By connecting the neighbor nodes, an affinity graph containing all samples and the affinity relationship between samples is constructed.

[0058] By constructing an affinity graph, we can build an affinity relationship network between the currently extracted speaker embedding vector and the sample embedding vector of the training sample. Based on the constructed affinity graph and clustering processing using a graph convolutional network, the accuracy of speaker feature extraction can be further improved.

[0059] In some embodiments, clustering nodes in the affinity graph based on a graph convolutional network to generate speaker clusters to be synthesized includes:

[0060] 1. Generate cluster proposals

[0061] Specifically, by clustering the nodes in the affinity graph, a plurality of cluster proposals of speakers to be synthesized are generated. The cluster proposals of speakers to be synthesized are subgraphs of the affinity graph, which are generated based on super nodes, and the super nodes contain a small number of nodes that are closely related to each other.

[0062] In some embodiments, the nodes in the affinity graph are clustered to generate multiple speaker cluster proposals to be synthesized, including: setting edge weights between nodes in the affinity graph to obtain multiple super nodes; performing clustering operations based on each super node, using the centroid of each super node as a node and the relationship between the centroids as an edge, to generate multiple speaker cluster proposals to be synthesized.

[0063] Specifically, a set of supernodes can be generated by setting various thresholds (preliminary clustering) on ​​the edge weights of the affinity graph. Although the samples in the same supernode may be the same speaker, each speaker may correspond to multiple supernodes, so the supernodes need to be further clustered. In this way, a higher-level graph based on supernodes is constructed, with the centroids of supernodes as nodes and the relationships between centroids as edges.

[0064] 2. Cluster detection based on graph convolution

[0065] Specifically, the cluster detection unit based on the graph convolutional network extracts the cluster features of each speaker cluster proposal to be synthesized, and identifies the candidate speaker cluster proposal to be synthesized from the generated multiple speaker cluster proposals to be synthesized according to the cluster features. Through the detection of the cluster detection unit, the possibility of whether the proposal is correct is determined, and high-quality speaker cluster proposals to be synthesized can be identified as candidate speaker cluster proposals to be synthesized.

[0066] Exemplarily, a graph convolutional network is used to extract the features of each proposal. The calculation formula of each GCN (Graph Convolutional Network) layer can be expressed as:

[0067]

[0068] in, represents the diagonal matrix. i represents the speaker cluster proposal to be synthesized, H (k) (P i ) represents the feature vector of the kth layer. (k) represents the matrix used to transform the vector, and σ is the ReLU nonlinear activation function. During the training process, GCN is optimized by minimizing the objective function of the mean square error (MSE) between the true score and the predicted score. Then, high-quality clusters are selected from the generated multiple speaker cluster proposals to be synthesized as candidate speaker cluster proposals to be synthesized.

[0069] 3. Clustering segmentation based on graph convolution

[0070] Specifically, the cluster segmentation unit based on graph convolution predicts the probability value of each node in the candidate speaker cluster proposal to be synthesized, and removes the nodes with probability values ​​less than a preset threshold as abnormal nodes from the nodes of the candidate speaker cluster proposal to be synthesized.

[0071] More specifically, even if GCN identifies high-quality clusters, they may contain some outliers (abnormal nodes) that need to be eliminated. Outliers can be excluded from cluster proposals by using a cluster split unit. During cluster detection, each sample in a candidate cluster proposal consists of a set of feature vectors (representing a node), an affinity matrix, and a binary vector indicating whether the node is positive. The cluster split unit is then trained using node binary cross entropy as the loss function. The cluster split unit outputs a predicted probability value for each node to indicate how likely it is that it is a true member rather than an abnormal node. In the cluster split unit, abnormal nodes are removed from cluster proposals.

[0072] 4. De-overlapping

[0073] Specifically, the nodes in the candidate speaker cluster proposals to be synthesized are sorted according to the probability values ​​of the nodes, and at least one node whose probability value is greater than a preset threshold is selected to generate the speaker cluster proposal to be synthesized.

[0074] More specifically, there should be no overlap between different clusters. The “de-overlapping” process uses the scores based on GCN predictions and sorts them in descending order. The nodes with the highest GCN scores are selected to form cluster proposals, dividing the unlabeled dataset into appropriate clusters.

[0075] In the above-mentioned embodiment, by performing cluster detection, cluster segmentation and de-overlapping processing based on graph convolution on the generated multiple cluster proposals of speakers to be synthesized, high-quality clusters of speakers to be synthesized can be obtained, thereby improving the output accuracy of the graph speaker encoder. In some embodiments, the method further includes the following steps:

[0076] 5. Label denoising

[0077] Specifically, corresponding speaker labels are assigned to the speaker clusters to be synthesized; the network parameters of the speaker embedding extraction network are optimized through the denoising loss function to denoise the speaker labels generated during the clustering process, and the denoised speaker labels are re-input into the speaker embedding extraction network for iterative training.

[0078] In the self-supervised learning of speaker recognition, pseudo-labels can be used as supervision symbols. Due to the unreliability of pseudo-labels, speaker recognition models may not be able to effectively utilize pseudo-labels. The noise of pseudo-labels has a significant impact on the performance of speaker embedding vectors obtained by speaker encoders trained on large datasets. The pseudo-labels assigned by clustering will introduce noise and be quickly memorized by the speaker embedding extraction network. Therefore, in order to avoid the speaker embedding extraction network fitting noisy data, the parameters of the speaker embedding extraction network can be updated through an improved loss function, the formula of which is:

[0079]

[0080] Where B is the minimum batch size, Represents sample x i Classified as the true label y i The posterior probability of is the sample x i The posterior probability of being classified as the predicted label. Represents x i The predicted label of α t ∈[0,1] is and The confidence weight for the t-th training iteration between α and α, which determines whether the loss function depends more on the true label or the predicted label. t Dynamic increase, the formula is:

[0081] α t =α T (t / T) λ

[0082] Among them, α T ∈[0,1] indicates the final iteration α t The dynamically increasing confidence weights mean that the loss tends to depend more on the predicted label as the prediction becomes more accurate. By dynamically learning the labels of samples, we can prevent the network from overfitting to incorrect samples.

[0083] To better understand the operation flow of the image speaker encoder in the above embodiment, please refer to Figure 3 As shown, Figure 3 A schematic diagram of data flow for extracting speaker features based on a graph speaker encoder in some embodiments is shown.

[0084] In some embodiments, text data and a sorted phoneme sequence are input into a graph encoder to obtain text encoding features output by the graph encoder, including: generating a syntax tree corresponding to the text data based on the text data; parsing the grammatical relationship between words in the syntax tree, using phonemes as nodes and the grammatical relationship between phonemes as edges to generate a syntax graph; based on a bidirectional GRU network, bidirectionally encoding the grammatical relationship between two phonemes to obtain a relation encoding vector; wherein the relation encoding vector includes a forward relation encoding vector and a backward relation encoding vector; calculating an attention score based on the graph attention mechanism of the graph encoder and the relation encoding vector; encoding the sorted phoneme sequence according to the attention score to obtain text encoding features output by the graph encoder.

[0085] Specifically, the graph encoder may include a text encoding unit and a relation encoding unit.

[0086] The relation encoding unit converts the syntax tree of the input text data into a syntax graph describing the global relationship between the involved input data.

[0087] Syntax tree, also known as syntactic dependency parse tree, is one of the traditional methods to describe the linguistic dependency relationship between words. In the tree structure, only directly related words in the sentence are connected. In order to mine the grammatical relationship between two words in a sentence, the topology of the syntax tree is extended to establish a fully connected communication. The syntax graph can be used as an extension of the syntax tree.

[0088] The grammar graph proposed in this application is an extension of the grammar tree, which turns a unidirectional connection into a bidirectional connection by adding a reverse connection. In addition, a self-loop edge is introduced, and each word has a specific label. In this way, the words (or phonemes) in a sentence can be represented by nodes, and their connection relationships are represented by edges. Through bidirectional connections, a word can directly receive and send information to any other word, whether they are directly related or not.

[0089] To model the relationship between two nodes (phonemes), the relationship between a pair of nodes is described as the shortest relationship path between them. A recurrent neural network with a gated recurrent unit (GRU) can be used to convert the relationship sequence into a distributed representation.

[0090] For example, the shortest relationship path sp between node i and node j is i→j It is represented as [sp1,...,sp t ,...,sp n+1 ]=[e(i,k1),e(k1,k2),...,e(k n ,j)], where e(·,·) represents the edge label and k1:n is the relay node. Bidirectional GRU is used to encode the path relationship:

[0091]

[0092]

[0093] in, represents the forward relation encoding vector, Represents the backward relation encoding vector, GRU f Represents the forward GRU network, GRU b represents the backward GRU network, and the last hidden states of the forward and backward GRU networks are concatenated to form the final relational encoding r ij =[s n+1 ; s0]. The final relation encoding represents the linguistic relationship between two words (or phonemes). By constructing the relation encoding, the graph encoder model in speech synthesis is provided with a global view on how to collect and distribute information.

[0094] The graph encoder in the embodiment of the present application incorporates the grammatical relation encoding into the self-attention mechanism to indicate the relationship between characters (or phonemes). The purpose of the graph encoder is to convert the input character embedding sequence (or phoneme sequence) and the relation encoding into a corresponding syntax-driven character embedding sequence (or phoneme sequence). This method incorporates the explicit relational representation between two nodes in the grammar graph into the graph attention calculation, forming a grammar-aware graph attention mechanism, which can be referred to as grammar-aware graph attention. Furthermore, the final encoded representation can also be calculated by stacking multiple grammar-aware graph attention and feedforward layer modules. In each module, the character embedding sequence is updated based on all other character embedding sequences and corresponding relational encodings. The encoding result of the character embedding sequence of the last module can be fed to the variational adapter and decoder to generate acoustic features.

[0095] In some embodiments, based on a bidirectional GRU network, the grammatical relationship between two phonemes is bidirectionally encoded to obtain a relationship encoding vector, including: when two phonemes belong to the same word, based on the bidirectional GRU network and using a self-looping edge coding algorithm, the grammatical relationship between the two phonemes is bidirectionally encoded; when the two phonemes belong to different words, based on the bidirectional GRU network, the grammatical relationship between the words to which the two phonemes belong is bidirectionally encoded.

[0096] In this embodiment, the basic unit of a sentence in text data may be a phoneme tag, and the relationship coding between words in NLP may be extended to the relationship coding between phonemes. When two phonemes belong to the same word, the grammatical relationship between the two phonemes may be bidirectionally encoded using a self-looping edge coding algorithm. When two phonemes belong to different words, the grammatical relationship between the words to which the two phonemes belong may be bidirectionally encoded based on a bidirectional GRU network. By extending the relationship coding between phonemes to the relationship coding between words, the coding accuracy may be improved.

[0097] In some embodiments, bidirectional encoding of the grammatical relationship between two phonemes may be encoding the relationship based on the shortest path between the two phonemes.

[0098] In some embodiments, an attention score is calculated based on a graph attention mechanism and a relation encoding vector of a graph encoder, including: capturing an addressing relationship based on basic grammatical content in relation encoding based on a graph attention mechanism of a graph encoder; calculating a forward relation bias between phonemes based on a forward relation encoding vector; controlling a backward relation bias between phonemes based on a backward relation encoding vector; calculating a comprehensive bias based on the forward relation encoding vector and the backward relation encoding vector; and obtaining an attention score based on the addressing relationship, the forward relation bias, the backward relation bias, and the comprehensive bias.

[0099] Specifically, in order to encode the connection direction between phonemes when calculating attention, the relationship encoding vector r is first ij Divided into forward relation encoding vector r i→j and the backward relation encoding vector r j→i ; [r i→j ; r j→i ]=W rrij .

[0100] Then, we use the syntax-aware graph attention mechanism to calculate the attention score, which is based on the phoneme representation and its bidirectional relationship as follows:

[0101]

[0102] Among them, (a) represents the addressing relation that captures the basic grammatical content. (b) The forward relation bias between phonemes is calculated based on the forward relation encoding vector. (c) The backward relation bias between phonemes is controlled based on the backward relation encoding vector; (d) The comprehensive bias (encoding universal relation bias) is obtained based on the forward relation encoding vector and the backward relation encoding vector.

[0103] Below, you can refer to Figure 4 As shown, Figure 4 A schematic diagram of data flow for performing a text feature encoder based on a graph encoder in some embodiments is shown.

[0104] In some application examples, Stanza can be used to extract syntactic dependency parse trees from text data based on graph encoders. The relation encoding unit is based on bidirectional GRU to encode relations in both directions. N = 6 encoding blocks are used in the text encoding unit of the graph encoder. The multi-head attention in the graph attention unit in the graph encoder and decoder is 4 heads. The input text data is first converted into a 256-dimensional character embedding sequence and then used as the input of the encoder. The decoder generates a series of 80-channel Mel-spectrogram acoustic features as output. Mel-spectrogram acoustic features with a frame size of 50ms and a frame shift of 12.5ms are extracted and further normalized to zero mean and unit variance as reference targets. The training algorithm can be optimized using Adam, with β1 = 0.9 and β2 = 0.98. The same learning rate schedule as in trandformer is used. This method combines all the different shortest paths in the training and test batches, and then encodes them into vector representations through the relation encoding unit.

[0105] For example, the structure of the decoder can refer to Figure 5 As shown, Figure 5 A schematic diagram of the structure of a decoder in some embodiments is shown.

[0106] For example, the variational adapter can refer to Figure 6 As shown, Figure 6 A schematic diagram of the structure of a variational adapter in some embodiments is shown.

[0107] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0108] In some embodiments, Figure 7 As shown, a speech synthesis device is provided, including: a graph speaker encoding module 710, a front-end processing module 720, a graph encoding module 730, a variational adaptation module 740 and a decoding synthesis module 750, wherein:

[0109] Figure speaker encoding module 710, used to obtain the speech data of the speaker to be synthesized and extract speaker features from the speech data of the speaker to be synthesized;

[0110] The front-end processing module 720 is used to obtain text data, convert the text data into a phoneme sequence, and insert a position code into the phoneme sequence to obtain a sorted phoneme sequence;

[0111] A graph encoding module 730, configured to input text data and the sorted phoneme sequence into a graph encoder to obtain text encoding features output by the graph encoder;

[0112] The variational adaptation module 740 is used to concatenate the speaker feature with the text encoding feature to obtain a concatenated feature, and input the concatenated feature into the variational adapter for duration prediction to obtain a predicted feature;

[0113] The decoding and synthesis module 750 is used to insert the position code into the prediction feature to obtain the sorted spliced ​​feature, input the sorted spliced ​​feature into the decoder, obtain the Mel-spectrogram to be synthesized output by the decoder, and convert the Mel-spectrogram to be synthesized into the speaker synthesized audio.

[0114] In some embodiments, the graph speaker encoding module 710 uses a speaker embedding extraction network to extract a speaker embedding vector from the speech data of the speaker to be synthesized; an affinity graph is constructed based on the speaker embedding vector and the sample embedding vector of the training sample of the speaker embedding extraction network; wherein the affinity graph uses the embedding vector as a node and at least one neighbor embedding vector of each embedding vector determined by cosine similarity as an edge; the nodes in the affinity graph are clustered based on the graph convolutional network to generate a cluster of speakers to be synthesized.

[0115] In some embodiments, the graph speaker encoding module 710 clusters the nodes in the affinity graph to generate multiple speaker cluster proposals to be synthesized; the cluster detection unit based on the graph convolution network extracts the clustering features of each speaker cluster proposal to be synthesized, and identifies the candidate speaker cluster proposal to be synthesized from the generated multiple speaker cluster proposals to be synthesized according to the clustering features; the cluster segmentation unit based on the graph convolution predicts the probability value of each node in the candidate speaker cluster proposal to be synthesized, and removes the nodes with probability values ​​less than a preset threshold as abnormal nodes from the nodes of the candidate speaker cluster proposal to be synthesized; the nodes in the candidate speaker cluster proposal to be synthesized are sorted according to the size of the probability value of each node, and at least one node with a probability value greater than the preset threshold is selected to generate the speaker cluster proposal to be synthesized.

[0116] In some embodiments, the graph speaker encoding module 710 obtains multiple super nodes by setting the edge weights between nodes in the affinity graph; performs clustering operations based on each super node, takes the centroid of each super node as the node and the relationship between the centroids as the edge, and generates multiple speaker cluster proposals to be synthesized.

[0117] In some embodiments, the graph speaker encoding module 710 is further used to assign pseudo labels to the speaker clusters to be synthesized; optimize the network parameters of the speaker embedding extraction network through a denoising loss function to denoise the pseudo labels generated during the clustering process, and re-input the denoised pseudo labels into the speaker embedding extraction network for iterative training.

[0118] In some embodiments, the graph encoding module 730 generates a syntax tree corresponding to the text data based on the text data; parses the grammatical relationship between words in the syntax tree, uses phonemes as nodes and the grammatical relationship between phonemes as edges to generate a syntax graph; based on a bidirectional GRU network, bidirectionally encodes the grammatical relationship between two phonemes to obtain a relationship encoding vector; wherein the relationship encoding vector includes a forward relationship encoding vector and a backward relationship encoding vector; calculates an attention score based on the graph attention mechanism of the graph encoder and the relationship encoding vector; encodes the sorted phoneme sequence according to the attention score to obtain the text encoding features output by the graph encoder.

[0119] In some embodiments, the graph encoding module 730 captures the addressing relationship based on the basic grammatical content in the relation encoding based on the graph attention mechanism of the graph encoder; calculates the forward relation bias between phonemes according to the forward relation encoding vector; controls the backward relation bias between phonemes according to the backward relation encoding vector; calculates the comprehensive bias based on the forward relation encoding vector and the backward relation encoding vector; and obtains the attention score according to the addressing relationship, the forward relation bias, the backward relation bias and the comprehensive bias.

[0120] In some embodiments, when two phonemes belong to the same word, the graph encoding module 730 performs bidirectional encoding on the grammatical relationship between the two phonemes based on a bidirectional GRU network and using a self-looping edge coding algorithm; when the two phonemes belong to different words, the graph encoding module 730 performs bidirectional encoding on the grammatical relationship between the words to which the two phonemes belong respectively based on a bidirectional GRU network.

[0121] For the specific definition of the speech synthesis device, please refer to the definition of the speech synthesis method above, which will not be repeated here. Each module in the above-mentioned speech synthesis device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0122] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech synthesis method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0123] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0124] In some embodiments, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: obtaining speech data of a speaker to be synthesized, extracting speaker features from the speech data of the speaker to be synthesized; obtaining text data, converting the text data into a phoneme sequence, and inserting position codes into the phoneme sequence to obtain a sorted phoneme sequence; inputting the text data and the sorted phoneme sequence into a graph encoder to obtain text encoding features output by the graph encoder; wherein the graph encoder encodes the grammatical relationship of the text data based on the grammatical graph of the text data, and performs text encoding on the sorted phoneme sequence in combination with a graph attention mechanism; concatenating the speaker features with the text encoding features to obtain concatenated features, inputting the concatenated features into a variational adapter for duration prediction, and obtaining predicted features; inserting position codes into the predicted features to obtain sorted predicted features, inputting the sorted predicted features into a decoder, obtaining a mel-spectrogram to be synthesized output by the decoder, and converting the mel-spectrogram to be synthesized into speaker synthesized audio.

[0125] In some embodiments, when the processor executes the computer program, the following steps are also implemented: using a speaker embedding extraction network to extract a speaker embedding vector from the speech data of the speaker to be synthesized; constructing an affinity graph based on the speaker embedding vector and the sample embedding vector of the training sample of the speaker embedding extraction network; wherein the affinity graph uses the embedding vector as a node and at least one neighbor embedding vector that is the nearest neighbor of each embedding vector determined by cosine similarity as an edge; clustering the nodes in the affinity graph based on a graph convolutional network to generate a cluster of speakers to be synthesized.

[0126] In some embodiments, when the processor executes the computer program, the following steps are also implemented: clustering the nodes in the affinity graph to generate multiple speaker cluster proposals to be synthesized; a cluster detection unit based on a graph convolution network extracts cluster features of each speaker cluster proposal to be synthesized, and identifies candidate speaker cluster proposals to be synthesized from the generated multiple speaker cluster proposals to be synthesized according to the cluster features; a cluster segmentation unit based on graph convolution predicts the probability value of each node in the candidate speaker cluster proposals to be synthesized, and removes nodes with probability values ​​less than a preset threshold as abnormal nodes from the nodes of the candidate speaker cluster proposals to be synthesized; and sorting each node in the candidate speaker cluster proposals to be synthesized according to the size of the probability value of each node, and selecting at least one node with a probability value greater than the preset threshold to generate a speaker cluster proposal to be synthesized.

[0127] In some embodiments, when the processor executes the computer program, the following steps are also implemented: obtaining multiple super nodes by setting the edge weights between nodes in the affinity graph; performing clustering operations based on each super node, using the centroid of each super node as the node and the relationship between the centroids as the edge, to generate multiple clustering proposals for speakers to be synthesized.

[0128] In some embodiments, when the processor executes the computer program, the following steps are also implemented: assigning pseudo labels to the speaker clusters to be synthesized; optimizing the network parameters of the speaker embedding extraction network through a noise reduction loss function to denoise the pseudo labels generated during the clustering process, and re-inputting the denoised pseudo labels into the speaker embedding extraction network for iterative training.

[0129] In some embodiments, the processor further implements the following steps when executing the computer program: generating a syntax tree corresponding to the text data according to the text data; parsing the grammatical relationship between words in the syntax tree, using phonemes as nodes and the grammatical relationship between phonemes as edges to generate a syntax graph; based on a bidirectional GRU network, bidirectionally encoding the grammatical relationship between two phonemes to obtain a relationship encoding vector; wherein the relationship encoding vector includes a forward relationship encoding vector and a backward relationship encoding vector; calculating an attention score based on the graph attention mechanism of the graph encoder and the relationship encoding vector; encoding the sorted phoneme sequence according to the attention score to obtain the text encoding features output by the graph encoder.

[0130] In some embodiments, the processor further implements the following steps when executing the computer program: a graph attention mechanism based on a graph encoder captures an addressing relationship based on basic grammatical content in a relational encoding; a forward relation bias between phonemes is calculated based on a forward relation encoding vector; a backward relation bias between phonemes is controlled based on a backward relation encoding vector; a comprehensive bias is calculated based on the forward relation encoding vector and the backward relation encoding vector; and an attention score is obtained based on the addressing relationship, the forward relation bias, the backward relation bias, and the comprehensive bias.

[0131] In some embodiments, when the processor executes the computer program, the following steps are also implemented: when two phonemes belong to the same word, the grammatical relationship between the two phonemes is bidirectionally encoded based on the bidirectional GRU network and using a self-looping edge coding algorithm; when the two phonemes belong to different words, the grammatical relationship between the words to which the two phonemes belong is bidirectionally encoded based on the bidirectional GRU network.

[0132] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining speech data of a speaker to be synthesized, extracting speaker features from the speech data of the speaker to be synthesized; obtaining text data, converting the text data into a phoneme sequence, and inserting position codes into the phoneme sequence to obtain a sorted phoneme sequence; inputting the text data and the sorted phoneme sequence into a graph encoder to obtain text encoding features output by the graph encoder; wherein the graph encoder encodes the grammatical relationship of the text data based on the grammatical graph of the text data, and performs text encoding on the sorted phoneme sequence in combination with a graph attention mechanism; concatenating the speaker features with the text encoding features to obtain concatenated features, inputting the concatenated features into a variational adapter for duration prediction, and obtaining predicted features; inserting position codes into the predicted features to obtain sorted predicted features, inputting the sorted predicted features into a decoder, obtaining a mel-spectrogram to be synthesized output by the decoder, and converting the mel-spectrogram to be synthesized into speaker synthesized audio.

[0133] In some embodiments, when the computer program is executed by the processor, the following steps are also implemented: using a speaker embedding extraction network to extract a speaker embedding vector from the speech data of the speaker to be synthesized; constructing an affinity graph based on the speaker embedding vector and the sample embedding vector of the training sample of the speaker embedding extraction network; wherein the affinity graph uses the embedding vector as a node and at least one neighbor embedding vector that is the nearest neighbor of each embedding vector determined by cosine similarity as an edge; clustering the nodes in the affinity graph based on a graph convolutional network to generate a cluster of speakers to be synthesized.

[0134] In some embodiments, when the computer program is executed by the processor, the following steps are also implemented: clustering the nodes in the affinity graph to generate multiple speaker cluster proposals to be synthesized; a cluster detection unit based on a graph convolutional network extracts cluster features of each speaker cluster proposal to be synthesized, and identifies candidate speaker cluster proposals to be synthesized from the generated multiple speaker cluster proposals to be synthesized based on the cluster features; a cluster segmentation unit based on graph convolution predicts the probability value of each node in the candidate speaker cluster proposals to be synthesized, and removes nodes with probability values ​​less than a preset threshold as abnormal nodes from the nodes of the candidate speaker cluster proposals to be synthesized; sorting each node in the candidate speaker cluster proposals to be synthesized according to the size of the probability value of each node, and selecting at least one node with a probability value greater than the preset threshold to generate a speaker cluster proposal to be synthesized.

[0135] In some embodiments, when the computer program is executed by the processor, the following steps are also implemented: multiple super nodes are obtained by setting the edge weights between nodes in the affinity graph; clustering operations are performed based on each super node, with the centroid of each super node as the node and the relationship between the centroids as the edge, to generate multiple clustering proposals of speakers to be synthesized.

[0136] In some embodiments, when the computer program is executed by the processor, the following steps are also implemented: assigning pseudo labels to the speaker clusters to be synthesized; optimizing the network parameters of the speaker embedding extraction network through a noise reduction loss function to reduce the noise of the pseudo labels generated during the clustering process, and re-inputting the denoised pseudo labels into the speaker embedding extraction network for iterative training.

[0137] In some embodiments, when the computer program is executed by the processor, the following steps are also implemented: generating a syntax tree corresponding to the text data based on the text data; parsing the grammatical relationship between words in the syntax tree, using phonemes as nodes and the grammatical relationship between phonemes as edges to generate a syntax graph; based on a bidirectional GRU network, bidirectionally encoding the grammatical relationship between two phonemes to obtain a relationship encoding vector; wherein the relationship encoding vector includes a forward relationship encoding vector and a backward relationship encoding vector; calculating an attention score based on the graph attention mechanism of the graph encoder and the relationship encoding vector; encoding the sorted phoneme sequence according to the attention score to obtain the text encoding features output by the graph encoder.

[0138] In some embodiments, when the computer program is executed by a processor, the following steps are also implemented: a graph attention mechanism based on a graph encoder captures an addressing relationship based on basic grammatical content in a relational encoding; a forward relation bias between phonemes is calculated based on a forward relation encoding vector; a backward relation bias between phonemes is controlled based on a backward relation encoding vector; a comprehensive bias is calculated based on the forward relation encoding vector and the backward relation encoding vector; and an attention score is obtained based on the addressing relationship, the forward relation bias, the backward relation bias, and the comprehensive bias.

[0139] In some embodiments, when the computer program is executed by a processor, the following steps are also implemented: when two phonemes belong to the same word, the grammatical relationship between the two phonemes is bidirectionally encoded based on a bidirectional GRU network and using a self-looping edge coding algorithm; when the two phonemes belong to different words, the grammatical relationship between the words to which the two phonemes belong is bidirectionally encoded based on a bidirectional GRU network.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A speech synthesis method, the method comprising: Acquire speech data of a speaker to be synthesized, and extract speaker features from the speech data of the speaker to be synthesized; Acquire text data, convert the text data into a phoneme sequence, and insert a position code into the phoneme sequence to obtain a sorted phoneme sequence; Inputting the text data and the sorted phoneme sequence into a graph encoder to obtain text encoding features output by the graph encoder; wherein the graph encoder performs grammatical relationship encoding on the text data based on a grammatical graph of the text data, and performs text encoding on the sorted phoneme sequence in combination with a graph attention mechanism; Concatenating the speaker feature with the text encoding feature to obtain a concatenated feature, and inputting the concatenated feature into a variational adapter for duration prediction to obtain a predicted feature; Inserting position codes into the prediction features to obtain sorted prediction features, inputting the sorted prediction features into a decoder to obtain a mel-spectrogram to be synthesized output by the decoder, and converting the mel-spectrogram to be synthesized into speaker synthesized audio.

2. The method according to claim 1, characterized in that The extracting speaker features from the speaker speech data to be synthesized includes: Extracting a speaker embedding vector from the to-be-synthesized speaker speech data using a speaker embedding extraction network; Constructing an affinity graph based on the speaker embedding vector and the sample embedding vector of the training sample of the speaker embedding extraction network; wherein the affinity graph has the embedding vector as a node and at least one neighbor embedding vector that is the nearest neighbor of each embedding vector determined by cosine similarity as an edge; The nodes in the affinity graph are clustered based on a graph convolutional network to generate clusters of speakers to be synthesized.

3. The method according to claim 2, characterized in that The clustering process of the nodes in the affinity graph based on the graph convolutional network to generate clusters of speakers to be synthesized includes: Clustering the nodes in the affinity graph to generate a plurality of cluster proposals of speakers to be synthesized; Extracting cluster features of each of the speaker cluster proposals to be synthesized by a cluster detection unit based on the graph convolutional network, and identifying a candidate speaker cluster proposal to be synthesized from the generated multiple speaker cluster proposals to be synthesized according to the cluster features; The cluster segmentation unit based on graph convolution predicts the probability value of each node in the candidate speaker cluster proposal to be synthesized, and removes the node with a probability value less than a preset threshold as an abnormal node from the node of the candidate speaker cluster proposal to be synthesized; The nodes in the candidate speaker cluster proposal to be synthesized are sorted according to the probability values ​​of the nodes, and at least one node whose probability value is greater than a preset threshold is selected to generate the speaker cluster proposal to be synthesized.

4. The method according to claim 3, characterized in that The clustering of nodes in the affinity graph to generate a plurality of speaker cluster proposals to be synthesized includes: A plurality of super nodes are obtained by setting edge weights between nodes in the affinity graph; A clustering operation is performed based on each of the super nodes, with the centroids of each of the super nodes as nodes and the relationships between the centroids as edges, to generate a plurality of cluster proposals of speakers to be synthesized.

5. The method according to claim 2, characterized in that: The method further comprises: Assigning pseudo labels to the cluster of speakers to be synthesized; The network parameters of the speaker embedding extraction network are optimized by a denoising loss function to denoise the pseudo labels generated during the clustering process, and the denoised pseudo labels are re-input into the speaker embedding extraction network for iterative training.

6. The method according to claim 1, characterized in that The step of inputting the text data and the sorted phoneme sequence into a graph encoder to obtain text encoding features output by the graph encoder comprises: Generate a syntax tree corresponding to the text data according to the text data; Parsing the grammatical relationship between words in the grammar tree, using phonemes as nodes and the grammatical relationship between phonemes as edges, to generate a grammar graph; Based on the bidirectional GRU network, the grammatical relationship between the two phonemes is bidirectionally encoded to obtain a relation encoding vector; wherein the relation encoding vector includes a forward relation encoding vector and a backward relation encoding vector; Calculating an attention score based on the graph attention mechanism of the graph encoder and the relation encoding vector; The sorted phoneme sequence is encoded according to the attention score to obtain text encoding features output by the graph encoder.

7. The method according to claim 6, characterized in that Calculating an attention score based on the graph attention mechanism of the graph encoder and the relation encoding vector, including: A graph attention mechanism based on the graph encoder captures the addressing relations in the relation encoding based on the basic grammatical content; Calculating the forward relation deviation between phonemes according to the forward relation encoding vector; Controlling the backward relation deviation between phonemes according to the backward relation encoding vector; Calculating a comprehensive deviation based on the forward relation encoding vector and the backward relation encoding vector; The attention score is obtained according to the addressing relationship, the forward relationship deviation, the backward relationship deviation and the comprehensive deviation.

8. The method according to claim 6, characterized in that The bidirectional GRU network is based on which the grammatical relationship between two phonemes is bidirectionally encoded to obtain a relationship encoding vector, including: When two phonemes belong to the same word, the grammatical relationship between the two phonemes is bidirectionally encoded based on the bidirectional GRU network and the self-loop edge coding algorithm; When two phonemes belong to different words, the grammatical relationship between the words to which the two phonemes belong is bidirectionally encoded based on the bidirectional GRU network.

9. A speech synthesis device, characterized in that: The device comprises: A speaker encoding module is used to obtain the speech data of the speaker to be synthesized and extract speaker features from the speech data of the speaker to be synthesized; A front-end processing module is used to obtain text data, convert the text data into a phoneme sequence, and insert a position code into the phoneme sequence to obtain a sorted phoneme sequence; A graph encoding module, used for inputting the text data and the sorted phoneme sequence into a graph encoder to obtain text encoding features output by the graph encoder; A variational adaptation module, used for concatenating the speaker feature with the text encoding feature to obtain a concatenated feature, and inputting the concatenated feature into a variational adapter for duration prediction to obtain a predicted feature; The decoding and synthesis module is used to insert the position code into the predicted feature to obtain the sorted spliced ​​feature, input the sorted spliced ​​feature into the decoder to obtain the Mel-spectrogram to be synthesized output by the decoder, and convert the Mel-spectrogram to be synthesized into the speaker synthesized audio.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images