Speech synthesis method and device, computer device and storage medium

Abstract

The application relates to a speech synthesis method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining to-be-synthesized speaker voice data, extracting speaker features from the to-be-synthesized speaker voice data; obtaining text data, converting the text data into a phoneme sequence, inserting position coding into the phoneme sequence, and obtaining a sorted phoneme sequence; inputting the text data and the sorted phoneme sequence into a graph encoder to obtain text coding features output by the graph encoder; splicing the speaker features and the text coding features to obtain spliced features, inputting the spliced features into a variational adapter to perform duration prediction, and obtaining predicted features; inserting position coding into the predicted features to obtain sorted predicted features, inputting the sorted predicted features into a decoder to obtain to-be-synthesized mel-frequency spectrum output by the decoder, and converting the to-be-synthesized mel-frequency spectrum into synthesized speaker audio. The method can improve the accuracy of speech synthesis.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech synthesis method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of speech processing technology, Text-to-Speech (TTS) technology, also known as speech synthesis technology, has emerged. As a crucial component of intelligent question-answering systems, this technology has achieved tremendous progress over the past two decades. In recent years, neural network methods and sequence-to-sequence models have enabled TTS to reach a certain level of development. Although neural network-based TTS has demonstrated competitive performance, its accuracy in speech synthesis still needs improvement. Traditional speaker speech synthesis or cloning methods, which use speaker encoders to extract speaker embedding vectors, may encounter the problem of inaccurate speaker embedding vector extraction, leading to low accuracy and poor performance in speech synthesis. Summary of the Invention

[0003] Therefore, it is necessary to provide a speech synthesis method, apparatus, computer equipment, and storage medium that can improve the accuracy of speech synthesis in order to address the aforementioned technical problems.

[0004] A first aspect of this application provides a speech synthesis method, comprising: acquiring speech data of a speaker to be synthesized; extracting speaker features from the speech data of the speaker to be synthesized; acquiring text data; converting the text data into a phoneme sequence; inserting positional codes into the phoneme sequence to obtain a sorted phoneme sequence; inputting the text data and the sorted phoneme sequence into a graph encoder to obtain text-encoded features output by the graph encoder; wherein the graph encoder performs syntactic relation encoding on the text data based on a syntactic graph of the text data, and performs text encoding on the sorted phoneme sequence in conjunction with a graph attention mechanism; concatenating the speaker features with the text-encoded features to obtain concatenated features; inputting the concatenated features into a variational adapter for duration prediction to obtain predicted features; inserting positional codes into the predicted features to obtain sorted predicted features; inputting the sorted predicted features into a decoder to obtain a Mel spectrogram to be synthesized output by the decoder; and converting the Mel spectrogram to be synthesized into synthesized speaker audio.

[0005] In some embodiments, extracting speaker features from the speech data of the speaker to be synthesized includes: extracting speaker embedding vectors from the speech data of the speaker to be synthesized using a speaker embedding extraction network; constructing an affinity graph based on the speaker embedding vectors and the sample embedding vectors of the training samples of the speaker embedding extraction network; wherein, in the affinity graph, the embedding vectors are used as nodes and at least one nearest neighbor embedding vector determined by cosine similarity is used as an edge; and clustering the nodes in the affinity graph based on a graph convolutional network to generate a cluster of speakers to be synthesized.

[0006] In some embodiments, clustering of nodes in an affinity graph based on a graph convolutional network to generate a cluster of speakers to be synthesized includes: clustering nodes in the affinity graph to generate multiple cluster proposals for speakers to be synthesized; a clustering detection unit based on a graph convolutional network extracts clustering features of each cluster proposal for speakers to be synthesized, and identifies candidate cluster proposals for speakers to be synthesized from the multiple generated cluster proposals based on the clustering features; a clustering segmentation unit based on a graph convolutional network predicts the probability value of each node in the candidate cluster proposals for speakers to be synthesized, and removes nodes with probability values ​​less than a preset threshold as abnormal nodes from the nodes in the candidate cluster proposals for speakers to be synthesized; and sorts each node according to the probability value of each node in the candidate cluster proposals for speakers to be synthesized, and selects at least one node with a probability value greater than a preset threshold to generate a cluster proposal for speakers to be synthesized.

[0007] In some embodiments, clustering nodes in the affinity graph to generate multiple clustering proposals for synthesized speakers includes: obtaining multiple supernodes by setting edge weights between nodes in the affinity graph; performing clustering operations based on each supernode, using the centroid of each supernode as the node and the relationship between the centroids as the edge, to generate multiple clustering proposals for synthesized speakers.

[0008] In some embodiments, the method further includes: assigning pseudo-labels to the speaker clusters to be synthesized; optimizing the network parameters of the speaker embedding extraction network using a denoising loss function to denoise the pseudo-labels generated during the clustering process; and re-inputting the denoised pseudo-labels into the speaker embedding extraction network for iterative training.

[0009] In some embodiments, text data and a sorted phoneme sequence are input into a graph encoder to obtain text encoding features output by the graph encoder, including: generating a syntax tree corresponding to the text data; parsing the grammatical relationships between words in the syntax tree, using phonemes as nodes and the grammatical relationships between phonemes as edges to generate a syntax graph; bidirectionally encoding the grammatical relationships between two phonemes based on a bidirectional GRU network to obtain a relation encoding vector; wherein the relation encoding vector includes a forward relation encoding vector and a backward relation encoding vector; calculating an attention score based on the graph attention mechanism of the graph encoder and the relation encoding vector; and encoding the sorted phoneme sequence according to the attention score to obtain text encoding features output by the graph encoder.

[0010] In some embodiments, calculating an attention score based on a graph attention mechanism of a graph encoder and relational encoding vectors includes: capturing addressing relations based on basic grammatical content in relational encoding using a graph attention mechanism of a graph encoder; calculating forward relational deviations between phonemes based on forward relational encoding vectors; controlling backward relational deviations between phonemes based on backward relational encoding vectors; calculating a combined deviation based on forward and backward relational encoding vectors; and obtaining an attention score based on addressing relations, forward relational deviations, backward relational deviations, and combined deviations.

[0011] In some embodiments, based on a bidirectional GRU network, the grammatical relationship between two phonemes is bidirectionally encoded to obtain a relationship encoding vector, including: when the two phonemes belong to the same word, the grammatical relationship between the two phonemes is bidirectionally encoded based on a bidirectional GRU network and using a self-circulating edge coding algorithm; when the two phonemes belong to different words, the grammatical relationship between the words to which the two phonemes belong is bidirectionally encoded based on a bidirectional GRU network.

[0012] A second aspect of this application provides a speech synthesis apparatus, the apparatus comprising:

[0013] The image-based speaker encoding module is used to acquire the speech data of the speaker to be synthesized and extract speaker features from the speech data of the speaker to be synthesized.

[0014] The front-end processing module is used to acquire text data, convert the text data into a phoneme sequence, and insert position codes into the phoneme sequence to obtain a sorted phoneme sequence.

[0015] The graph encoding module is used to input text data and a sorted phoneme sequence into the graph encoder to obtain the text encoding features output by the graph encoder.

[0016] The variational adaptation module is used to concatenate speaker features with text encoding features to obtain concatenated features. The concatenated features are then input into the variational adapter for duration prediction to obtain predicted features.

[0017] The decoding and synthesis module is used to insert positional codes into the predicted features to obtain sorted and concatenated features. The sorted and concatenated features are then input into the decoder to obtain the Mel spectrogram to be synthesized from the decoder output. The Mel spectrogram to be synthesized is then converted into synthesized audio of the speaker.

[0018] A third aspect of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the speech synthesis method of any embodiment of the first aspect described above.

[0019] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the speech synthesis method of any embodiment of the first aspect described above.

[0020] The aforementioned speech synthesis method, apparatus, computer equipment, and storage medium employ a graph encoder. Based on the syntax graph of the text data, it encodes the syntactic relations of the text data. Combined with a graph attention mechanism, it encodes the sorted phoneme sequences corresponding to the text data to obtain text-encoded features. These text-encoded features, processed by the graph encoder, are then concatenated with speaker features to perform feature prediction and Mel-spectrum prediction, thereby achieving the synthesis or cloning of the speaker's synthesized audio. This method, by introducing a graph encoder and employing syntax graph-based syntactic relation encoding combined with a self-attention mechanism for text encoding, improves the accuracy of text encoding, thus enhancing the accuracy of speech synthesis. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the speech synthesis method in some embodiments;

[0022] Figure 2 This is a schematic diagram showing the connection structure of each network unit and the data flow in some embodiments of the speech synthesis method;

[0023] Figure 3 This is a schematic diagram illustrating the data flow for extracting speaker features based on a graph speaker encoder in some embodiments;

[0024] Figure 4 This is a schematic diagram of the data flow for text feature encoding based on a graph encoder in some embodiments;

[0025] Figure 5 This is a schematic diagram of the decoder structure in some embodiments;

[0026] Figure 6 This is a schematic diagram of the variational adapter in some embodiments;

[0027] Figure 7Here are some block diagrams of the speech synthesis apparatus in the embodiments;

[0028] Figure 8 This is a diagram showing the internal structure of a computer device in some embodiments. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] The speech synthesis method provided in this application can be applied to computer devices, which can be servers or terminals. The terminals can be, but are not limited to, various personal computers, laptops, smartphones, tablets, portable wearable devices, as well as in-vehicle terminals, smart speakers, smart robots, and other intelligent terminal devices. The server can be a standalone server or a server cluster composed of multiple servers.

[0031] In some embodiments, such as Figure 1 As shown, a speech synthesis method is provided. Taking the application of this method to a smart terminal device as an example, the method includes the following steps:

[0032] Step S102: Obtain the speech data of the speaker to be synthesized, and extract the speaker features from the speech data of the speaker to be synthesized.

[0033] The speaker speech data to be synthesized refers to the speech data of the speaker to be synthesized or cloned as a sample. Speaker features refer to feature vectors that can characterize the speaker's speech information.

[0034] Specifically, speaker features can be extracted from the input speech data to be synthesized using a graph speaker encoder. For example, speaker embeddings can be extracted as speaker features from the input speech data to be synthesized using any conventional speaker encoder, or the speaker feature extraction method based on a graph speaker encoder described in the embodiments below of this application can be used.

[0035] Step S104: Obtain text data, convert the text data into a phoneme sequence, and insert position codes into the phoneme sequence to obtain a sorted phoneme sequence.

[0036] Specifically, text data can be preprocessed through front-end processing modules, such as text normalization, grapheme-to-phoneme conversion, and for Chinese text data, polyphone classification, prosody prediction, etc., and finally converted into phoneme sequences.

[0037] Furthermore, positional encoding can be inserted into the phoneme sequence. Inserting positional encoding can be understood as concatenating the phoneme sequence with positional encoding. Since the position and order of words in a sentence are very important for any language and can affect the semantics of the sentence, by inserting positional encoding, the input phoneme sequence can carry positional information to generate a sorted phoneme sequence, so that the subsequent model network can find its positional characteristics.

[0038] Step S106: Input the text data and the sorted phoneme sequence into the graph encoder to obtain the text encoding features output by the graph encoder; wherein, the graph encoder encodes the syntactic relations of the text data based on the syntactic graph of the text data, and combines the graph attention mechanism to encode the sorted phoneme sequence.

[0039] Specifically, the pre-processed text data and its corresponding sorted phoneme sequence can be input together into the graph encoder. The graph encoder can include a relation encoding unit and a text encoder. The relation encoding unit is used to encode the grammatical relations of the text data based on the grammatical graph of the text data, and the text encoder is used to combine the grammatical relation encoding with the graph attention mechanism to encode the sorted phoneme sequence into text.

[0040] Graph encoders incorporate syntactic relation encodings into a graph attention mechanism to indicate character relationships. The goal of a graph encoder is to convert the input phoneme sequence and relation encodings into corresponding syntactic-driven character embedding sequences (text encoding features).

[0041] Step S108: Concatenate the speaker features with the text encoding features to obtain concatenated features. Input the concatenated features into the variational adapter for duration prediction to obtain predicted features.

[0042] Specifically, the speaker features of the speaker to be synthesized extracted in step S102 can be concatenated with the text encoding features of the text data obtained in step S106. The concatenated features are then input into the variational adapter for duration prediction to obtain the predicted features.

[0043] Step S110: Insert positional encoding into the predicted features to obtain sorted predicted features. Input the sorted predicted features into the decoder to obtain the Mel spectrogram to be synthesized output by the decoder. Convert the Mel spectrogram to be synthesized into speaker synthesized audio.

[0044] Specifically, the positional codes are concatenated to the predicted features to obtain sorted predicted features with the positional codes inserted. The sorted predicted features are then decoded using a decoder to predict the Mel spectrogram of the speaker to be synthesized. Based on the predicted Mel spectrogram, a vocoder is used to convert the Mel spectrogram into the audio of the speaker to be synthesized.

[0045] The aforementioned speech synthesis method employs a graph encoder, which encodes the syntactic relations of the text data based on a syntactic graph. It then combines this with a graph attention mechanism to encode the sorted phoneme sequences corresponding to the text data, obtaining text-encoded features. These text-encoded features, processed by the graph encoder, are concatenated with speaker features to perform feature prediction and Mel-spectrum prediction, thereby achieving the synthesis or cloning of the speaker's synthesized audio. This method, by introducing a graph encoder and using syntactic relation encoding based on a syntactic graph combined with a self-attention mechanism for text encoding, improves the accuracy of text encoding, thus enhancing the accuracy of speech synthesis.

[0046] Below, for reference Figure 2 As shown, Figure 2 The diagram illustrates the network unit connection structure and data flow of speech synthesis methods in several application examples. Specifically,

[0047] Step 1: Input speech data of the speaker to be synthesized into a speech encoder to extract speaker features of the speaker to be synthesized or cloned.

[0048] The second step involves the input text data entering the front-end processing module. After undergoing text regularization, phonetic conversion, and classification of polyphonic characters and prosody prediction for Chinese characters, it is ultimately converted into a phoneme sequence. Furthermore, character embedding can be performed on the phoneme sequence.

[0049] Step 3: Insert positional codes into the output after step 2.

[0050] Step 4: Combine the text data with the output and input graph encoder from step 3.

[0051] Step 5: The features output from Step 4 are concatenated with the speaker features extracted in Step 1 and then fed into the variational adapter for duration prediction.

[0052] Step 6: Insert positional encoding into the output features of step 5 and then feed it into the decoder.

[0053] Step 7: Generate the Mel spectrogram to be synthesized or cloned using the decoder prediction.

[0054] Step 9: Convert the Mel spectrogram into the speaker audio to be synthesized or cloned using griffin-lim or a neural vocoder.

[0055] In some embodiments, extracting speaker features from the speech data of the speaker to be synthesized includes: extracting speaker embedding vectors from the speech data of the speaker to be synthesized using a speaker embedding extraction network; constructing an affinity graph based on the speaker embedding vectors and the sample embedding vectors of the training samples of the speaker embedding extraction network; wherein, in the affinity graph, the embedding vectors are used as nodes and at least one nearest neighbor embedding vector determined by cosine similarity is used as an edge; and clustering the nodes in the affinity graph based on a graph convolutional network to generate a cluster of speakers to be synthesized.

[0056] For example, a speaker embedding extraction network can be trained in a supervised manner using historical speaker speech data as samples.

[0057] For example, the affinity graph, G = (V, E), is a relationship graph where nodes V represent input samples and edges E represent the similarity between data points. Based on the embedding vectors extracted from a pre-trained speaker embedding extraction network, each sample (or its corresponding embedding vector) is treated as a node, and cosine similarity is used to find the K nearest neighbors for each sample. By connecting the neighboring nodes, an affinity graph containing all samples and the affinity relationships between them is constructed.

[0058] By constructing an affinity graph, a network of affinity relationships can be built between the currently extracted speaker embedding vector and the sample embedding vectors of the training samples. Based on the constructed affinity graph, clustering processing using graph convolutional networks can be performed to further improve the accuracy of speaker feature extraction.

[0059] In some embodiments, clustering of nodes in the affinity graph based on a graph convolutional network generates a cluster of speakers to be synthesized, including:

[0060] 1. Generate clustering proposals

[0061] Specifically, multiple speaker clustering proposals are generated by clustering the nodes in the affinity graph. Each speaker clustering proposal is a subgraph of the affinity graph, generated based on supernodes, which contain a small number of closely related nodes.

[0062] In some embodiments, clustering nodes in the affinity graph to generate multiple clustering proposals for synthesized speakers includes: obtaining multiple supernodes by setting edge weights between nodes in the affinity graph; performing clustering operations based on each supernode, using the centroid of each supernode as the node and the relationship between the centroids as the edge, to generate multiple clustering proposals for synthesized speakers.

[0063] Specifically, a set of supernodes can be generated by setting various thresholds (preliminary clustering) on ​​the edge weights of the affinity graph. Although samples in the same supernode may belong to the same speaker, each speaker may correspond to multiple supernodes. Therefore, further clustering of the supernodes is required. This constructs a higher-level graph based on supernodes, with the centroids of the supernodes as nodes and the relationships between centroids as edges.

[0064] 2. Clustering detection based on graph convolution

[0065] Specifically, a clustering detection unit based on a graph convolutional network extracts clustering features from each proposed speaker clustering scheme. Based on these features, candidate proposed speaker clustering schemes are identified from the generated multiple proposed schemes. Through the detection by the clustering detection unit, the likelihood of a scheme being correct is determined, and high-quality proposed speaker clustering schemes are identified as candidate proposed speaker clustering schemes.

[0066] For example, a graph convolutional network is used to extract features for each proposal. The computation formula for each GCN (Graph Convolutional Network) layer can be expressed as:

[0067]

[0068] in, P represents the angle matrix. i H represents the proposed speaker clustering scheme. (k) (P i θ represents the feature vector of the k-th layer. (k) Let represent the matrix used for the transformation vector, and σ be the ReLU nonlinear activation function. During training, GCN is optimized by minimizing the mean squared error (MSE) between the true and predicted scores. Then, high-quality clusters are selected as candidate speaker clustering proposals from the generated multiple proposed clusters.

[0069] 3. Clustering and segmentation based on graph convolution

[0070] Specifically, the graph convolution-based clustering segmentation unit predicts the probability value of each node in the candidate speaker clustering proposal and removes nodes with probability values ​​less than a preset threshold as abnormal nodes from the nodes of the candidate speaker clustering proposal.

[0071] More specifically, even if GCN identifies high-quality clusters, they may still contain outliers (abnormal nodes) that need to be removed. Outliers can be eliminated from cluster proposals by using a clustering splitting unit. During cluster detection, each sample in a candidate cluster proposal consists of a set of feature vectors (representing a node), an affinity matrix, and a binary vector indicating whether the node is positive. The clustering splitting unit is then trained using the node's binary cross-entropy as the loss function. The clustering splitting unit outputs a predicted probability value for each node, indicating how likely it is to be a true member rather than an anomalous node. In the clustering splitting unit, anomalous nodes are removed from the cluster proposal.

[0072] 4. De-overlap processing

[0073] Specifically, the nodes are sorted according to their probability values ​​in the candidate speaker clustering proposals, and at least one node with a probability value greater than a preset threshold is selected to generate a speaker clustering proposal.

[0074] More specifically, there should be no overlap between different clusters. The "de-overlap" process uses scores based on GCN predictions and sorts them in descending order. The nodes with the highest GCN scores are selected to form clustering proposals, dividing the unlabeled dataset into appropriate clusters.

[0075] The above embodiments, by performing graph convolution-based clustering detection, cluster segmentation, and de-overlapping processing on multiple generated speaker clustering proposals, can obtain high-quality speaker clusters, thereby improving the output accuracy of the graph speaker encoder. In some embodiments, the method further includes the following steps:

[0076] 5. Tag noise reduction

[0077] Specifically, corresponding speaker labels are assigned to the clusters of speakers to be synthesized; the network parameters of the speaker embedding extraction network are optimized by using a noise reduction loss function to reduce the noise of the speaker labels generated during the clustering process; and the noise-reduced speaker labels are re-input into the speaker embedding extraction network for iterative training.

[0078] In self-supervised learning of speaker recognition, pseudo-labels can be used as supervisory symbols. However, due to the unreliability of pseudo-labels, speaker recognition models may not effectively utilize them. The noise of pseudo-labels significantly impacts the performance of speaker embedding vectors obtained by speaker encoders trained on large datasets. Pseudo-labels assigned through clustering introduce noise, which is quickly memorized by the speaker embedding extraction network. Therefore, to avoid the speaker embedding extraction network fitting noisy data, the parameters of the speaker embedding extraction network can be updated using an improved loss function, the formula of which is:

[0079]

[0080] Where B is the minimum batch size. Indicates sample x i Classified as real label y i The posterior probability, It is sample x i The posterior probability of being classified as the predicted label. x represents i The predicted label. α t ∈[0,1] is and The confidence weights in the t-th training iteration determine whether the loss function relies more on the true label or the predicted label. α t Dynamically increases, the formula is:

[0081] α t =α T (t / T) λ

[0082] Where, α T ∈[0,1] indicates that α is in the final iteration. t The value of . Dynamically increasing confidence weights means that the loss tends to rely more on the predicted labels as the predictions become increasingly accurate. By dynamically learning the labels of the samples, it is possible to prevent the network from overfitting to incorrect samples.

[0083] To better understand the workflow of the image-based speaker encoder in the above embodiments, please refer to the following. Figure 3 As shown, Figure 3 The diagram illustrates the data flow for extracting speaker features based on a graph speaker encoder in some embodiments.

[0084] In some embodiments, text data and a sorted phoneme sequence are input into a graph encoder to obtain text encoding features output by the graph encoder, including: generating a syntax tree corresponding to the text data; parsing the grammatical relationships between words in the syntax tree, using phonemes as nodes and the grammatical relationships between phonemes as edges to generate a syntax graph; bidirectionally encoding the grammatical relationships between two phonemes based on a bidirectional GRU network to obtain a relation encoding vector; wherein the relation encoding vector includes a forward relation encoding vector and a backward relation encoding vector; calculating an attention score based on the graph attention mechanism of the graph encoder and the relation encoding vector; and encoding the sorted phoneme sequence according to the attention score to obtain text encoding features output by the graph encoder.

[0085] Specifically, a graph encoder may include text encoding units and relation encoding units.

[0086] The relational encoding unit converts the syntax tree of the input text data into a syntax graph that describes the global relationships between the input data involved.

[0087] A syntax tree, or parse tree, is a traditional method for describing linguistic dependencies between words. In a tree structure, only directly related words in a sentence are connected. To uncover the grammatical relationships between two words in a sentence, the topology of the syntax tree can be expanded to establish fully connected communication; a syntax graph can serve as an extension of the syntax tree.

[0088] The syntax graph proposed in this application is an extension of the syntax tree, transforming unidirectional connections into bidirectional connections by adding back connections. Furthermore, self-looping edges are introduced, and each word has a specific label. Thus, words (or phonemes) in a sentence can be represented by nodes, and their connections can be represented by edges. Through bidirectional connections, a word can directly receive and send information to any other word, regardless of whether they are directly related.

[0089] To model the relationship between two nodes (phonemes), the relationship between node pairs is described as the shortest path between them. A recurrent neural network with gated recurrent units (GRUs) can be used to transform the sequence of relationships into a distributed representation.

[0090] For example, the shortest relational path sp between node i and node j i→j Represented as [sp1,...,sp t ,...,sp n+1 ]=[e(i,k1),e(k1,k2),...,e(k n ,j)], where e(·,·) represents the edge label, and k1:n are relay nodes. A bidirectional GRU is used for path relationship encoding:

[0091]

[0092]

[0093] in, This represents the forward relation encoding vector. Represents the backward relation encoding vector, GRU f This indicates a forward-facing GRU network, where GRU... b This represents a backward GRU network where the final hidden states of the forward and backward GRU networks are concatenated to form the final relation code r. ij =[s n+1 [s0]. The final relational encoding represents the linguistic relationship between two words (or phonemes). Constructing relational encoding provides a global view of how information is collected and distributed in graph encoder models for speech synthesis.

[0094] The graph encoder in this embodiment incorporates grammatical relation encoding into a self-attention mechanism to indicate relationships between characters (or phonemes). The graph encoder aims to convert the input character embedding sequence (or phoneme sequence) and relation encoding into a corresponding syntactic-driven character embedding sequence (or phoneme sequence). This method incorporates the explicit relation representation between two nodes in the grammatical graph into the graph attention computation, constituting a grammar-aware graph attention mechanism, which can be simply referred to as grammar-aware graph attention. Furthermore, the final encoded representation can be computed by stacking multiple grammar-aware graph attention and feedforward layer modules. In each module, the character embedding sequence is updated based on all other character embedding sequences and their corresponding relation encodings. The encoding result of the character embedding sequence from the last module can be fed into a variational adapter and decoder to generate acoustic features.

[0095] In some embodiments, based on a bidirectional GRU network, the grammatical relationship between two phonemes is bidirectionally encoded to obtain a relationship encoding vector, including: when the two phonemes belong to the same word, the grammatical relationship between the two phonemes is bidirectionally encoded based on a bidirectional GRU network and using a self-circulating edge coding algorithm; when the two phonemes belong to different words, the grammatical relationship between the words to which the two phonemes belong is bidirectionally encoded based on a bidirectional GRU network.

[0096] In this embodiment, the basic unit of a sentence in the text data can be a phoneme tag. The word-to-word relationship encoding in NLP can be extended to the phoneme-to-phoneme relationship encoding. When two phonemes belong to the same word, the self-circulating edge coding algorithm can be used to encode the grammatical relationship between the two phonemes bidirectionally. When two phonemes belong to different words, the grammatical relationship between the words to which the two phonemes belong is bidirectionally encoded based on a bidirectional GRU network. By extending the word-to-word relationship encoding to the phoneme-to-phoneme relationship encoding, the accuracy of the encoding can be improved.

[0097] In some embodiments, bidirectional encoding of the grammatical relationship between two phonemes can be based on the shortest path between the two phonemes.

[0098] In some embodiments, calculating an attention score based on a graph attention mechanism of a graph encoder and relational encoding vectors includes: capturing addressing relations based on basic grammatical content in relational encoding using a graph attention mechanism of a graph encoder; calculating forward relational deviations between phonemes based on forward relational encoding vectors; controlling backward relational deviations between phonemes based on backward relational encoding vectors; calculating a combined deviation based on forward and backward relational encoding vectors; and obtaining an attention score based on addressing relations, forward relational deviations, backward relational deviations, and combined deviations.

[0099] Specifically, in order to encode the connection direction between phonemes when calculating attention, the relation encoding vector r is first... ij Divided into forward relational encoding vectors r i→j and backward relation encoding vector r j→i ;[r i→j ;r j→i ] = W rrij .

[0100] Then, a grammar-aware graph attention mechanism is used to calculate the attention score, which is based on the phoneme representation and its bidirectional relational representation as follows:

[0101]

[0102] Wherein, (a) represents the addressing relation for capturing basic grammatical content; (b) calculates the forward relation deviation between phonemes based on the forward relation encoding vector; (c) controls the backward relation deviation between phonemes based on the backward relation encoding vector; and (d) the combined deviation (encoding general relation deviation) obtained based on the forward relation encoding vector and the backward relation encoding vector.

[0103] Below, you can refer to Figure 4 As shown, Figure 4 The diagram illustrates the data flow of text feature encoders based on graph encoders in some embodiments.

[0104] In some application examples, text data based on graph encoders can be used to extract syntactic dependency parse trees using Stanza. The relation encoding unit performs relation encoding in both directions based on a bidirectional GRU. The text encoding unit of the graph encoder uses N=6 encoding blocks. The multi-head attention in the graph attention units of both the graph encoder and decoder consists of 4 heads. The input text data is first converted into a 256-dimensional character embedding sequence, which is then used as input to the encoder. The decoder generates a series of 80-channel Mel-spectral acoustic features as output. Mel-spectral acoustic features are extracted for frames of 50ms size and frame shift of 12.5ms, and further normalized to zero mean and unit variance as reference targets. The Adam optimization training algorithm can be used, with β1=0.9 and β2=0.98. The same learning rate table as in Trandformer is used. This method combines all distinct shortest paths in the training and test batches and then encodes them into vector representations through the relation encoding unit.

[0105] For example, the structure of the decoder can be referred to Figure 5 As shown, Figure 5 A schematic diagram of the decoder structure is shown in some embodiments.

[0106] For example, a variational adapter can be referenced Figure 6 As shown, Figure 6 A schematic diagram of the variational adapter is shown in some embodiments.

[0107] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0108] In some embodiments, such as Figure 7 As shown, a speech synthesis device is provided, including: a graph speaker encoding module 710, a front-end processing module 720, a graph encoding module 730, a variational adaptation module 740, and a decoding and synthesis module 750, wherein:

[0109] The image-based speaker encoding module 710 is used to acquire the speech data of the speaker to be synthesized and extract speaker features from the speech data of the speaker to be synthesized.

[0110] The front-end processing module 720 is used to acquire text data, convert the text data into a phoneme sequence, and insert position codes into the phoneme sequence to obtain a sorted phoneme sequence.

[0111] The graph encoding module 730 is used to input text data and a sorted phoneme sequence into the graph encoder to obtain the text encoding features output by the graph encoder.

[0112] The variational adaptation module 740 is used to concatenate speaker features with text encoding features to obtain concatenated features, and input the concatenated features into the variational adapter for duration prediction to obtain predicted features;

[0113] The decoding and synthesis module 750 is used to insert positional codes into the predicted features to obtain sorted concatenated features. The sorted concatenated features are input into the decoder to obtain the Mel spectrogram to be synthesized output by the decoder, and the Mel spectrogram to be synthesized is converted into speaker synthesized audio.

[0114] In some embodiments, the graph speaker encoding module 710 extracts speaker embedding vectors from the speaker speech data to be synthesized using a speaker embedding extraction network; constructs an affinity graph based on the speaker embedding vectors and the sample embedding vectors of the training samples of the speaker embedding extraction network; wherein, in the affinity graph, the embedding vectors are used as nodes and at least one nearest neighbor embedding vector determined by cosine similarity is used as an edge; and the nodes in the affinity graph are clustered based on a graph convolutional network to generate a cluster of speakers to be synthesized.

[0115] In some embodiments, the graph speaker encoding module 710 clusters nodes in the affinity graph to generate multiple clustering proposals for synthesized speakers; the clustering detection unit based on graph convolutional networks extracts the clustering features of each clustering proposal for synthesized speakers, and identifies candidate clustering proposals for synthesized speakers from the generated multiple clustering proposals based on the clustering features; the clustering segmentation unit based on graph convolution predicts the probability value of each node in the candidate clustering proposals for synthesized speakers, and removes nodes with probability values ​​less than a preset threshold as abnormal nodes from the nodes of the candidate clustering proposals for synthesized speakers; the nodes are sorted according to the probability values ​​of each node in the candidate clustering proposals for synthesized speakers, and at least one node with a probability value greater than a preset threshold is selected to generate a clustering proposal for synthesized speakers.

[0116] In some embodiments, the graph speaker encoding module 710 obtains multiple supernodes by setting the edge weights between nodes in the affinity graph; and performs clustering operations based on each supernode, using the centroid of each supernode as the node and the relationship between the centroids as the edge, to generate multiple speaker clustering proposals to be synthesized.

[0117] In some embodiments, the graph speaker encoding module 710 is further configured to assign pseudo-labels to the cluster of speakers to be synthesized; optimize the network parameters of the speaker embedding extraction network through a noise reduction loss function to reduce the noise of the pseudo-labels generated during the clustering process; and re-input the noise-reduced pseudo-labels into the speaker embedding extraction network for iterative training.

[0118] In some embodiments, the graph encoding module 730 generates a syntax tree corresponding to the text data based on the text data; it parses the grammatical relationships between words in the syntax tree, using phonemes as nodes and the grammatical relationships between phonemes as edges to generate a syntax graph; based on a bidirectional GRU network, it performs bidirectional encoding on the grammatical relationships between two phonemes to obtain a relation encoding vector; wherein, the relation encoding vector includes a forward relation encoding vector and a backward relation encoding vector; it calculates an attention score based on the graph attention mechanism of the graph encoder and the relation encoding vector; and it encodes the sorted phoneme sequence according to the attention score to obtain the text encoding features output by the graph encoder.

[0119] In some embodiments, the graph encoding module 730 captures addressing relations based on basic grammatical content in relation encoding based on the graph attention mechanism of the graph encoder; calculates the forward relation deviation between phonemes based on the forward relation encoding vector; controls the backward relation deviation between phonemes based on the backward relation encoding vector; calculates the comprehensive deviation based on the forward relation encoding vector and the backward relation encoding vector; and obtains the attention score based on the addressing relations, forward relation deviation, backward relation deviation, and comprehensive deviation.

[0120] In some embodiments, when two phonemes belong to the same word, the graph encoding module 730 performs bidirectional encoding of the grammatical relationship between the two phonemes based on a bidirectional GRU network and using a self-circulating edge encoding algorithm; when two phonemes belong to different words, it performs bidirectional encoding of the grammatical relationship between the words to which the two phonemes belong based on a bidirectional GRU network.

[0121] For specific limitations regarding the speech synthesis device, please refer to the limitations on the speech synthesis method above, which will not be repeated here. Each module in the aforementioned speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0122] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a speech synthesis method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0123] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0124] In some embodiments, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: acquiring speech data of a speaker to be synthesized, and extracting speaker features from the speech data of the speaker to be synthesized; acquiring text data, converting the text data into a phoneme sequence, and inserting position codes into the phoneme sequence to obtain a sorted phoneme sequence; inputting the text data and the sorted phoneme sequence into a graph encoder to obtain text-encoded features output by the graph encoder; wherein the graph encoder performs syntactic relation encoding on the text data based on the syntactic graph of the text data, and performs text encoding on the sorted phoneme sequence in conjunction with a graph attention mechanism; concatenating the speaker features and the text-encoded features to obtain concatenated features, inputting the concatenated features into a variational adapter for duration prediction to obtain predicted features; inserting position codes into the predicted features to obtain sorted predicted features, inputting the sorted predicted features into a decoder to obtain a Mel spectrogram to be synthesized output by the decoder, and converting the Mel spectrogram to be synthesized into synthesized speaker audio.

[0125] In some embodiments, when the processor executes the computer program, it further implements the following steps: extracting speaker embedding vectors from the speaker speech data to be synthesized using a speaker embedding extraction network; constructing an affinity graph based on the speaker embedding vectors and the sample embedding vectors of the training samples of the speaker embedding extraction network; wherein, in the affinity graph, the embedding vectors are nodes and at least one nearest neighbor embedding vector determined by cosine similarity is an edge; and clustering the nodes in the affinity graph based on a graph convolutional network to generate a cluster of speakers to be synthesized.

[0126] In some embodiments, when the processor executes the computer program, it further implements the following steps: clustering nodes in the affinity graph to generate multiple clustering proposals for synthesized speakers; a clustering detection unit based on a graph convolutional network extracts clustering features of each clustering proposal for synthesized speakers, and identifies candidate clustering proposals for synthesized speakers from the multiple generated clustering proposals based on the clustering features; a clustering segmentation unit based on graph convolution predicts the probability values ​​of each node in the candidate clustering proposals for synthesized speakers, and removes nodes with probability values ​​less than a preset threshold as abnormal nodes from the nodes in the candidate clustering proposals for synthesized speakers; sorting each node according to the probability values ​​of each node in the candidate clustering proposals for synthesized speakers, and selecting at least one node with a probability value greater than a preset threshold to generate a clustering proposal for synthesized speakers.

[0127] In some embodiments, when the processor executes the computer program, it further implements the following steps: obtaining multiple supernodes by setting the edge weights between nodes in the affinity graph; performing clustering operations based on each supernode, using the centroid of each supernode as the node and the relationship between the centroids as the edge, to generate multiple speaker clustering proposals to be synthesized.

[0128] In some embodiments, when the processor executes the computer program, it further implements the following steps: assigning pseudo-labels to the speaker cluster to be synthesized; optimizing the network parameters of the speaker embedding extraction network through a noise reduction loss function to reduce the noise of the pseudo-labels generated during the clustering process; and re-inputting the noise-reduced pseudo-labels into the speaker embedding extraction network for iterative training.

[0129] In some embodiments, when the processor executes the computer program, it further implements the following steps: generating a syntax tree corresponding to the text data; parsing the grammatical relationships between words in the syntax tree, using phonemes as nodes and the grammatical relationships between phonemes as edges to generate a syntax graph; bidirectionally encoding the grammatical relationships between two phonemes based on a bidirectional GRU network to obtain a relation encoding vector; wherein the relation encoding vector includes a forward relation encoding vector and a backward relation encoding vector; calculating an attention score based on the graph attention mechanism of the graph encoder and the relation encoding vector; and encoding the sorted phoneme sequence according to the attention score to obtain the text encoding features output by the graph encoder.

[0130] In some embodiments, when the processor executes the computer program, it further implements the following steps: capturing addressing relations based on basic grammatical content in relation encoding using a graph attention mechanism based on a graph encoder; calculating forward relational deviations between phonemes based on forward relational encoding vectors; controlling backward relational deviations between phonemes based on backward relational encoding vectors; calculating a comprehensive deviation based on forward relational encoding vectors and backward relational encoding vectors; and obtaining an attention score based on the addressing relations, forward relational deviations, backward relational deviations, and comprehensive deviations.

[0131] In some embodiments, when the processor executes the computer program, it further implements the following steps: when two phonemes belong to the same word, bidirectionally encoding the grammatical relationship between the two phonemes based on a bidirectional GRU network and using a self-circulating edge coding algorithm; when two phonemes belong to different words, bidirectionally encoding the grammatical relationship between the words to which the two phonemes belong based on a bidirectional GRU network.

[0132] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the following steps: acquiring speech data of a speaker to be synthesized; extracting speaker features from the speech data of the speaker to be synthesized; acquiring text data; converting the text data into a phoneme sequence; inserting positional codes into the phoneme sequence to obtain a sorted phoneme sequence; inputting the text data and the sorted phoneme sequence into a graph encoder to obtain text-encoded features output by the graph encoder; wherein the graph encoder performs syntactic relation encoding on the text data based on the syntactic graph of the text data, and performs text encoding on the sorted phoneme sequence in conjunction with a graph attention mechanism; concatenating the speaker features with the text-encoded features to obtain concatenated features; inputting the concatenated features into a variational adapter for duration prediction to obtain predicted features; inserting positional codes into the predicted features to obtain sorted predicted features; inputting the sorted predicted features into a decoder to obtain a Mel spectrogram to be synthesized output by the decoder; and converting the Mel spectrogram to be synthesized into synthesized speaker audio.

[0133] In some embodiments, when the computer program is executed by a processor, it further implements the following steps: extracting speaker embedding vectors from the speaker speech data to be synthesized using a speaker embedding extraction network; constructing an affinity graph based on the speaker embedding vectors and the sample embedding vectors of the training samples of the speaker embedding extraction network; wherein, in the affinity graph, the embedding vectors are nodes and at least one nearest neighbor embedding vector determined by cosine similarity is an edge; and clustering the nodes in the affinity graph based on a graph convolutional network to generate a cluster of speakers to be synthesized.

[0134] In some embodiments, when the computer program is executed by a processor, it further implements the following steps: clustering nodes in the affinity graph to generate multiple clustering proposals for synthesized speakers; a clustering detection unit based on a graph convolutional network extracts clustering features of each clustering proposal for synthesized speakers, and identifies candidate clustering proposals for synthesized speakers from the multiple generated clustering proposals based on the clustering features; a clustering segmentation unit based on graph convolution predicts the probability values ​​of each node in the candidate clustering proposals for synthesized speakers, and removes nodes with probability values ​​less than a preset threshold as abnormal nodes from the nodes in the candidate clustering proposals for synthesized speakers; sorting each node according to the probability values ​​of each node in the candidate clustering proposals for synthesized speakers, and selecting at least one node with a probability value greater than a preset threshold to generate a clustering proposal for synthesized speakers.

[0135] In some embodiments, when the computer program is executed by the processor, it further implements the following steps: obtaining multiple supernodes by setting the edge weights between nodes in the affinity graph; performing clustering operations based on each supernode, using the centroid of each supernode as the node and the relationship between the centroids as the edge, to generate multiple speaker clustering proposals to be synthesized.

[0136] In some embodiments, when the computer program is executed by the processor, it further implements the following steps: assigning pseudo-labels to the speaker cluster to be synthesized; optimizing the network parameters of the speaker embedding extraction network through a noise reduction loss function to reduce the noise of the pseudo-labels generated during the clustering process; and re-inputting the noise-reduced pseudo-labels into the speaker embedding extraction network for iterative training.

[0137] In some embodiments, when the computer program is executed by the processor, it further performs the following steps: generating a syntax tree corresponding to the text data; parsing the grammatical relationships between words in the syntax tree, using phonemes as nodes and the grammatical relationships between phonemes as edges to generate a syntax graph; bidirectionally encoding the grammatical relationships between two phonemes based on a bidirectional GRU network to obtain a relation encoding vector; wherein the relation encoding vector includes a forward relation encoding vector and a backward relation encoding vector; calculating an attention score based on the graph attention mechanism of the graph encoder and the relation encoding vector; and encoding the sorted phoneme sequence according to the attention score to obtain the text encoding features output by the graph encoder.

[0138] In some embodiments, when the computer program is executed by a processor, it further implements the following steps: capturing addressing relations based on basic grammatical content in relation encoding using a graph attention mechanism based on a graph encoder; calculating forward relational deviations between phonemes based on forward relational encoding vectors; controlling backward relational deviations between phonemes based on backward relational encoding vectors; calculating a combined deviation based on forward relational encoding vectors and backward relational encoding vectors; and obtaining an attention score based on addressing relations, forward relational deviations, backward relational deviations, and combined deviations.

[0139] In some embodiments, when the computer program is executed by the processor, it further implements the following steps: when two phonemes belong to the same word, bidirectionally encoding the grammatical relationship between the two phonemes based on a bidirectional GRU network and using a self-circulating edge coding algorithm; when two phonemes belong to different words, bidirectionally encoding the grammatical relationship between the words to which the two phonemes belong based on a bidirectional GRU network.

[0140] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0141] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A speech synthesis method, the method comprising: Acquire the speech data of the speaker to be synthesized, and extract the speaker features from the speech data of the speaker to be synthesized; Obtain text data, convert the text data into a phoneme sequence, and insert position codes into the phoneme sequence to obtain a sorted phoneme sequence; The text data and the sorted phoneme sequence are input into a graph encoder to obtain the text encoding features output by the graph encoder; wherein, the graph encoder performs syntactic relation encoding on the text data based on the syntactic graph of the text data, and combines the syntactic relation encoding with a graph attention mechanism to perform text encoding on the sorted phoneme sequence; The speaker features are concatenated with the text encoding features to obtain concatenated features. The concatenated features are then input into a variational adapter for duration prediction to obtain predicted features. Positional encoding is inserted into the predicted features to obtain sorted predicted features. The sorted predicted features are input into the decoder to obtain the Mel spectrogram to be synthesized output by the decoder. The Mel spectrogram to be synthesized is then converted into speaker synthesized audio.

2. The method according to claim 1, characterized in that, Extracting speaker features from the speaker speech data to be synthesized includes: Speaker embedding vectors are extracted from the speech data of the speaker to be synthesized using a speaker embedding extraction network; An affinity graph is constructed based on the speaker embedding vector and the sample embedding vector of the training samples of the speaker embedding extraction network; wherein, in the affinity graph, the embedding vector is used as a node and at least one nearest neighbor embedding vector determined by cosine similarity is used as an edge; Clustering of nodes in the affinity graph is performed based on graph convolutional networks to generate a cluster of speakers to be synthesized.

3. The method according to claim 2, characterized in that, The process of clustering nodes in the affinity graph using a graph convolutional network to generate a cluster of speakers to be synthesized includes: Cluster the nodes in the affinity graph to generate multiple speaker clustering proposals to be synthesized; The clustering detection unit based on the graph convolutional network extracts the clustering features of each of the proposed speaker clustering schemes, and identifies candidate proposed speaker clustering schemes from the generated multiple proposed speaker clustering schemes based on the clustering features. The graph convolution-based clustering segmentation unit predicts the probability value of each node in the candidate speaker clustering proposal and removes nodes with probability values ​​less than a preset threshold as abnormal nodes from the nodes of the candidate speaker clustering proposal. The nodes are sorted according to their probability values ​​in the candidate speaker clustering proposals, and at least one node with a probability value greater than a preset threshold is selected to generate the speaker clustering proposal.

4. The method according to claim 3, characterized in that, The process of clustering nodes in the affinity graph to generate multiple proposed speaker clustering schemes includes: Multiple supernodes can be obtained by setting the edge weights between nodes in the affinity graph; Clustering operations are performed based on each of the supernodes, with the centroid of each supernode as the node and the relationship between the centroids as the edge, to generate multiple clustering proposals for synthesized speakers.

5. The method according to claim 2, characterized in that, The method further includes: Assign pseudo-labels to the cluster of speakers to be synthesized; The speaker embedding extraction network parameters are optimized by using a noise reduction loss function to reduce the noise of pseudo-labels generated during the clustering process. The noise-reduced pseudo-labels are then re-inputted into the speaker embedding extraction network for iterative training.

6. The method according to claim 1, characterized in that, The step of inputting the text data and the sorted phoneme sequence into a graph encoder to obtain the text encoding features output by the graph encoder includes: Generate a syntax tree corresponding to the text data based on the text data; The grammatical relationships between words in the syntax tree are parsed, and a syntax graph is generated by using phonemes as nodes and the grammatical relationships between phonemes as edges. Based on a bidirectional GRU network, the grammatical relationship between two phonemes is bidirectionally encoded to obtain a relationship encoding vector; wherein, the relationship encoding vector includes a forward relationship encoding vector and a backward relationship encoding vector; The attention score is calculated based on the graph attention mechanism of the graph encoder and the relation encoding vector; The sorted phoneme sequence is encoded based on the attention score to obtain the text encoding features output by the graph encoder.

7. The method according to claim 6, characterized in that, The attention score is calculated based on the graph attention mechanism of the graph encoder and the relation encoding vector, including: The graph attention mechanism based on the graph encoder captures the addressing relations based on basic grammar content in the relation encoding vector; Calculate the forward relation deviation between phonemes based on the forward relation encoding vector; The backward relation coding vector controls the backward relation deviation between phonemes; The overall deviation is calculated based on the forward relation encoding vector and the backward relation encoding vector; The attention score is obtained based on the addressing relationship, the forward relationship deviation, the backward relationship deviation, and the combined deviation.

8. The method according to claim 6, characterized in that, The bidirectional GRU network is used to bidirectionally encode the grammatical relationship between two phonemes, resulting in a relationship encoding vector, including: When two phonemes belong to the same word, the grammatical relationship between the two phonemes is bidirectionally encoded based on a bidirectional GRU network and using a self-circulating edge coding algorithm; When two phonemes belong to different words, the grammatical relationship between the words to which the two phonemes belong is bidirectionally encoded based on a bidirectional GRU network.

9. A speech synthesis device, characterized in that, The device includes: The image-to-speaker encoding module is used to acquire the speech data of the speaker to be synthesized and extract speaker features from the speech data of the speaker to be synthesized; The front-end processing module is used to acquire text data, convert the text data into a phoneme sequence, and insert position codes into the phoneme sequence to obtain a sorted phoneme sequence. The graph encoding module is used to input the text data and the sorted phoneme sequence into the graph encoder to obtain the text encoding features output by the graph encoder. The graph encoder performs grammatical relation encoding on the text data based on the grammar graph of the text data, and combines the grammatical relation encoding with the graph attention mechanism to perform text encoding on the sorted phoneme sequence. The variational adaptation module is used to concatenate the speaker features with the text encoding features to obtain concatenated features, and input the concatenated features into the variational adapter for duration prediction to obtain predicted features; The decoding and synthesis module is used to insert positional codes into the predicted features to obtain sorted predicted features, input the sorted predicted features into the decoder to obtain the Mel spectrogram to be synthesized output by the decoder, and convert the Mel spectrogram to be synthesized into speaker synthesized audio.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images