Speech synthesis method and device, computer equipment and storage medium

Abstract

The embodiment of the invention relates to a speech synthesis method and device, computer equipment and a storage medium. The method mainly comprises the following steps of: analyzing sentiment semantic information and an inter-word dependency relationship of text data to be synthesized by adopting a graph sentiment semantic encoder, and performing sentiment semantic enhancement to obtain sentiment semantic features; converting the to-be-synthesized text data into phoneme data; performing grammar relation coding on the text data according to the grammar graph of the to-be-synthesized text data by adopting a graph coder to obtain relation coding characteristics, and performing text coding on the phoneme data by utilizing the relation coding characteristics based on a graph attention mechanism to obtain text coding characteristics; splicing the emotion semantic features and the text coding features to obtain spliced features; performing time length prediction on the splicing features to obtain prediction data; decoding the prediction data to obtain spectrum data to be synthesized; and converting the spectrum data to be synthesized into synthesized audio. By adopting the method, the accuracy and emotion expression of speech synthesis can be improved.

Description

Technical Field

[0001] This application relates to the field of speech processing technologies, and in particular, to a speech synthesis method, apparatus, computer device, and storage medium. Background Art

[0002] With the development of speech processing technologies, the Text To Speech (TTS) technology, also known as speech synthesis technology, has emerged. As an important part of intelligent question-and-answer systems, this technology has achieved great development in the past two decades. In recent years, neural network algorithms and sequence-to-sequence models have brought the development of TTS to a certain level. Although neural network-based TTS has shown competitive performance, there is still room for improvement in speech synthesis accuracy and emotional synthesis. In traditional speaker speech synthesis or cloning methods, there are problems such as low speech synthesis accuracy and weak emotional expressiveness. Summary of the Invention

[0003] Based on this, in view of the above technical problems, it is necessary to provide a speech synthesis method, apparatus, computer device, and storage medium that can improve speech synthesis accuracy and emotional expressiveness.

[0004] In a first aspect, an embodiment of the present disclosure provides a speech synthesis method, which includes:

[0005] Analyze the emotional semantic information and word interdependence relationships of the text data to be synthesized using a graph emotional semantic encoder, and perform emotional semantic enhancement to obtain emotional semantic features;

[0006] Convert the text data to be synthesized into phoneme data;

[0007] Encode the grammatical relationships of the text data using a graph encoder based on the syntax graph of the text data to be synthesized to obtain relationship encoding features, and perform text encoding on the phoneme data using the relationship encoding features based on a graph attention mechanism to obtain text encoding features;

[0008] Concatenate the emotional semantic features and the text encoding features to obtain concatenated features;

[0009] Perform duration prediction on the concatenated features to obtain prediction data;

[0010] Decode the prediction data to obtain the spectrum data to be synthesized;

[0011] Convert the spectrum data to be synthesized into synthesized audio.

[0012] In some embodiments, analyzing the emotional semantic information and word interdependence relationships of the text data to be synthesized using a graph emotional semantic encoder, and performing emotional semantic enhancement to obtain emotional semantic features includes:

[0013] Extract the emotional semantic information of the text data to be synthesized to obtain word embedding vectors, which are used to represent the emotional semantic information between words;

[0014] Extract the text structure information of the text data to be synthesized, which is used to represent the dependency relationship between words;

[0015] Construct a dependency graph based on the word embedding vectors and the text structure information;

[0016] Perform emotional semantic enhancement on the dependency graph through a gated graph network to obtain emotional semantic features.

[0017] In some embodiments, the text structure information includes multiple word nodes and directed edges for representing dependency relationships. Constructing a dependency graph based on the word embedding vectors and the text structure information includes: using the word embedding vectors as the word nodes of the text structure information to construct a dependency graph.

[0018] In some embodiments, performing emotional semantic enhancement on the dependency graph through a gated graph network includes:

[0019] Performing emotional semantic enhancement from the direction of the directed edges of the dependency graph through a first gated graph network module;

[0020] Performing emotional semantic enhancement from the reverse direction of the direction of the directed edges of the dependency graph through a second gated graph network module.

[0021] In some embodiments, extracting the emotional semantic information of the text data to be synthesized to obtain word embedding vectors includes: extracting sub-word vectors in the text data to be synthesized; performing pooling processing on the sub-word vectors to obtain word embedding vectors.

[0022] In some embodiments, using a graph encoder to perform grammatical relationship encoding on the text data according to the syntax graph of the text data to be synthesized to obtain relationship encoding data, and performing text encoding on the relationship encoding data and phoneme data based on a graph attention mechanism to obtain text encoding features, including:

[0023] Generating a syntax tree corresponding to the text data according to the text data to be synthesized;

[0024] Parsing the grammatical relationships between words and phrases in the syntax tree, using phonemes as nodes and the grammatical relationships between phonemes as edges to generate a syntax graph;

[0025] Based on a bidirectional gated recurrent unit network, performing bidirectional encoding on the grammatical relationships between two phonemes in the syntax graph to obtain relationship encoding data; wherein, the relationship encoding data includes a forward relationship encoding vector and a backward relationship encoding vector;

[0026] Calculating attention scores based on the graph attention network of the graph encoder and the relationship encoding data;

[0027] Encode the phoneme data according to the attention scores to obtain text encoding features.

[0028] In some embodiments, calculating the attention scores based on the graph attention network of the graph encoder and the relational encoding data includes:

[0029] The graph attention network based on the graph encoder captures the addressing relationship of the basic grammar content in the relational encoding;

[0030] Calculate the forward relationship deviation between phonemes according to the forward relational encoding vector;

[0031] Control the backward relationship deviation between phonemes according to the backward relational encoding vector;

[0032] Calculate the comprehensive deviation based on the forward relational encoding vector and the backward relational encoding vector;

[0033] Obtain the attention scores according to the addressing relationship, the forward relationship deviation, the backward relationship deviation, and the comprehensive deviation.

[0034] In some embodiments, based on the bidirectional gated recurrent unit network, bidirectionally encode the syntactic relationship between two phonemes in the syntax graph to obtain relational encoding data, including:

[0035] When two phonemes belong to the same word, bidirectionally encode the syntactic relationship between the two phonemes based on the bidirectional gated recurrent unit network and using the self-loop edge encoding algorithm;

[0036] When two phonemes belong to different words, bidirectionally encode the syntactic relationship between the words to which the two phonemes belong respectively based on the bidirectional gated recurrent unit network.

[0037] In a second aspect, an embodiment of the present disclosure provides a speech synthesis device, and the device includes:

[0038] The emotional semantic extraction module is used to analyze the emotional semantic information and the inter-word dependency relationship of the text data to be synthesized by using a graph emotional semantic encoder, and perform emotional semantic enhancement to obtain emotional semantic features;

[0039] The front-end processing module is used to convert the text data to be synthesized into phoneme data;

[0040] The graph encoding module is used to perform syntactic relationship encoding on the text data according to the syntax graph of the text data to be synthesized by using a graph encoder to obtain relational encoding features, and perform text encoding on the phoneme data by using the relational encoding features based on a graph attention mechanism to obtain text encoding features;

[0041] The variational adaptation module is used to splice the emotional semantic features and the text encoding features to obtain spliced features, and perform duration prediction on the spliced features to obtain prediction data;

[0042] The audio synthesis module is used to convert the spectrum data to be synthesized into synthesized audio.

[0043] In a third aspect, embodiments of the present disclosure provide a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the corresponding steps in the voice synthesis method according to any one of the embodiments in the first aspect of the present disclosure.

[0044] In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the corresponding steps in the voice synthesis method according to any one of the embodiments in the first aspect of the present disclosure.

[0045] The above voice synthesis method, device, computer device, and storage medium extract emotional semantic features of the text data to be synthesized by using a graph emotional semantic encoder including a dependency analysis module and a relational gated graph network, splice the text encoding features processed by the graph encoder with the emotional semantic features, and further perform feature processing to synthesize audio. By introducing the graph emotional semantic encoder, the emotional semantic representation of the emotional semantic features is enhanced through dependency analysis and relational gated graph network, improving the emotionality and expressiveness of the synthesized speech. At the same time, by introducing the graph encoder, text encoding is performed by combining grammar relationship encoding based on a grammar graph with a self-attention mechanism, which can improve the accuracy of text encoding, thereby improving the accuracy of voice synthesis. Description of the Drawings

[0046] Figure 1 It is a schematic flowchart of the voice synthesis method in some embodiments;

[0047] Figure 2 It is a schematic data flow diagram of obtaining prediction data based on a variational adapter in some embodiments;

[0048] Figure 3 It is a schematic diagram of the structures of each network unit for implementing the voice synthesis method and the data flow;

[0049] Figure 4 It is a schematic flowchart of the emotional semantic feature extraction method in some embodiments

[0050] Figure 5 It is a schematic data flow diagram of extracting emotional semantic features based on a graph emotion editor in some embodiments;

[0051] Figure 6Schematic diagram of data flow of a text feature encoder based on a graph encoder in some embodiments;

[0052] Figure 7 Schematic diagram of data flow for decoding prediction data based on a decoder in some embodiments;

[0053] Figure 8 Block diagram of a speech synthesis device in some embodiments;

[0054] Figure 9 Internal structure diagram of a computer device in some embodiments. Detailed implementation manners

[0055] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The speech synthesis method provided by the present application can be applied to a computer device, which can be a server or a terminal. Among them, the terminal can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, portable wearable devices, and intelligent terminal devices such as vehicle-mounted terminals, smart speakers, and smart robots. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0057] In some embodiments, as Figure 1 shown, a speech synthesis method is provided. Taking the application of this method to an intelligent terminal device as an example, the method includes the following steps S101 to S107.

[0058] Step S101: Analyze the emotional semantic information and word dependency relationships of the text data to be synthesized by using a graph emotional semantic encoder, and perform emotional semantic enhancement to obtain emotional semantic features.

[0059] Among them, the text data to be synthesized refers to emotional text data with speech information to be synthesized. In some scenarios, such as an intelligent question-and-answer scenario, the text data to be synthesized can be the text data corresponding to the speech information to be answered generated by an intelligent question-and-answer system based on user interaction. Specifically, it can be a piece of text data with a meaning representation. The text data to be synthesized carries emotional semantic information.

[0060] The emotional semantic features refer to feature vectors that can characterize the emotions and semantics of the text data to be synthesized.

[0061] Specifically, the emotional semantic features can be obtained from the input text data to be synthesized through the graph emotional semantic encoder. By analyzing the emotional semantic information and word dependency relationships in the text data to be synthesized, and performing emotional semantic enhancement, the emotional semantic features are obtained. Exemplarily, the emotional semantic features can be obtained through the emotional semantic information extraction and word dependency relationship analysis methods in traditional methods, as well as the emotional semantic enhancement method. Of course, the emotional semantic features can also be obtained based on the graph emotional semantic encoder described in the embodiments below of this application.

[0062] Step S102: Convert the text data to be synthesized into phoneme data.

[0063] Among them, the phoneme data can be a phoneme sequence carrying position encoding information.

[0064] Specifically, the text data to be synthesized can be preprocessed through a front-end processing module. For example, it can go through text normalization, grapheme-to-phoneme conversion, and for Chinese text data, it can also go through polyphone classification, prosody prediction, and other preprocessing to be converted into a phoneme sequence.

[0065] In some embodiments, the text data to be synthesized can be first converted into a phoneme sequence, and then position encoding is further inserted into the phoneme sequence. Inserting position encoding can be understood as associating or splicing the phoneme sequence with the position encoding. For a language, the position and arrangement order of words in a sentence are very important and can affect the semantics of the sentence. Therefore, by inserting position encoding, the phonemes in the phoneme sequence can carry position information, enabling the subsequent model network to determine the phonemes to be processed based on the position information.

[0066] Step S103: Use a graph encoder to perform syntax relationship encoding on the text data according to the syntax graph of the text data to be synthesized, obtain relationship encoding features, and perform text encoding on the phoneme data based on the relationship encoding features using a graph attention mechanism to obtain text encoding features.

[0067] Among them, the graph encoder can include a relationship encoding unit and a text encoding unit. The relationship encoding unit is used to perform syntax relationship encoding on the text data based on the syntax graph of the text data to be synthesized to obtain relationship encoding features. The text encoding unit can include a graph attention network. By using the graph attention mechanism to combine the relationship feature encoding to perform text encoding on the phoneme data, text encoding features can be obtained.

[0068] A syntax graph is an extension of a syntax tree. The syntax tree is used to represent the linguistic dependencies between words. In the tree structure, the words with associative relationships in a sentence are directly connected. Generally, in order to mine the syntactic relationship between two words in a sentence, the topological structure of the syntax tree can be extended to establish a fully connected communication.

[0069] The text encoding feature refers to the text feature used to represent the syntactic relationship of the text data to be synthesized.

[0070] Specifically, the phoneme data corresponding to the text data to be synthesized after preprocessing and the text data to be synthesized can be input into the graph encoder together. In the graph encoder, the relationship encoding feature is processed by the graph attention mechanism to indicate the character relationship. The purpose of the graph encoder is to convert the input phoneme data and relationship encoding feature into a corresponding sequence of syntax-driven character embeddings (text encoding feature).

[0071] Step S104: Concatenate the emotional semantic feature and the text encoding feature to obtain a concatenated feature.

[0072] Specifically, the emotional semantic feature of the text data to be synthesized obtained through step S101 and the text encoding feature of the text data to be synthesized obtained through step S103 can be concatenated.

[0073] The purpose of concatenation is to converge the emotional semantic feature and the text encoding feature. For example, a concatenation matrix can be used to record the emotional semantic feature and the text encoding feature. Here, there is no special limitation on the concatenation method, and those skilled in the art can select a suitable concatenation method according to actual needs.

[0074] Step S105: Perform duration prediction on the concatenated feature to obtain prediction data.

[0075] In some embodiments, the duration prediction can be performed on the concatenated feature through a variational adapter to obtain prediction data.

[0076] In some alternative embodiments, the variational adapter may include a duration predictor, a length adjuster, a fundamental frequency predictor, and an energy predictor. The variational adapter can add different variation information (such as duration, fundamental frequency, energy) to the concatenated feature to obtain prediction data.

[0077] Exemplarily, the variational adapter can refer to Figure 2 as shown, Figure 2 shows a schematic diagram of the data flow for obtaining prediction data based on the variational adapter 200 in some embodiments. The arrow direction in the figure indicates the data flow.

[0078] In some embodiments, after obtaining the first prediction data by performing duration prediction on the concatenated feature, position encoding is inserted into the first prediction data to obtain prediction data.

[0079] Step S106: Decode the predicted data to obtain the spectrum data to be synthesized.

[0080] Among them, the spectrum data to be synthesized can be a spectrogram for representing audio features, such as a Mel spectrogram or a linear spectrogram, etc.

[0081] Step S107: Convert the spectrum data to be synthesized into synthesized audio.

[0082] Input the spectrum data to be synthesized into a vocoder to convert it into synthesized audio. Among them, the vocoder can be a griffin-lim vocoder (a vocoder based on speech signals, which generates an unknown phase spectrum from a known amplitude spectrum through an iterative process and reconstructs the speech waveform using the generated phase spectrum), or a neural network vocoder. Those skilled in the art can also select other vocoders for the conversion of synthesized audio according to actual needs, such as WaveNet vocoder, WaveRNN vocoder, etc.

[0083] By performing the speech synthesis method of steps S101 to S107, the graph emotion semantic encoder including a dependency analysis module and a relational gated graph network is used to extract the emotion semantic features of the text data to be synthesized, and the text encoding features processed by the graph encoder are concatenated with the emotion semantic features, and further feature processing is performed to synthesize audio. The graph emotion semantic encoder is introduced, and the emotion semantic representation of the emotion semantic features is enhanced through dependency analysis and relational gated graph network, improving the emotion degree and expression ability of the synthesized speech. At the same time, the graph encoder is introduced, and text encoding is performed by combining grammar relationship encoding based on a grammar graph with a self-attention mechanism, which can improve the accuracy of text encoding, thereby improving the accuracy of speech synthesis.

[0084] In some embodiments, the structural and data flow of each network unit of the speech synthesis method can refer to Figure 3 As shown, the arrow direction in the figure represents the data flow, specifically including the following steps from the first step to the ninth step.

[0085] First step: Input the text data to be synthesized into the graph emotion semantic encoder to extract emotion semantic features.

[0086] Second step: Input the text data to be synthesized into the front-end processing module, and after text regularization, pronunciation conversion, and prosody prediction, it is finally converted into a phoneme sequence. If the corresponding language is Chinese, polysemous word classification processing can also be performed before converting it into a phoneme sequence. Further, character embedding processing can be performed on the phoneme sequence.

[0087] Third step: Insert position encoding into the phoneme sequence output in the second step to obtain phoneme data.

[0088] Step 4: Input the text data to be synthesized and the output (phoneme data) of Step 3 into the graph encoder.

[0089] Step 5: Concatenate the features (text encoding features) output in Step 4 with the emotional semantic features extracted in Step 1, then input them into the variational adapter for duration prediction to obtain the first prediction data.

[0090] Step 6: After inserting the position encoding into the output features (after the first prediction data) of Step 5 to obtain the prediction data, input the prediction data into the decoder.

[0091] Step 7: Predict and generate the mel spectrogram to be synthesized or cloned through the decoder.

[0092] Step 9: Convert the mel spectrogram into synthesized audio through the griffin-lim vocoder or neural vocoder.

[0093] In some embodiments, as Figure 4 shown, Step S101 may include Step S401 to Step S404.

[0094] Step S401: Extract the emotional semantic information of the text data to be synthesized to obtain the word embedding vector, which is used to represent the emotional semantic information between words.

[0095] In some alternative embodiments, the graph emotional semantic encoder may include a dependency analysis module, a BERT model processing module, and an emotional semantic enhancement module. Among them, the BERT model refers to Bidirectional Encoder Representations from Transformers, a pre-training model proposed by Google in 2018, which is a bidirectional encoder representation model based on transformers.

[0096] Specifically, input the text data to be synthesized into the graph emotional semantic encoder, and output the word embedding vector through the BERT model processing module.

[0097] In some embodiments, before inputting the text data to be synthesized into the BERT model processing module, it further includes formatting the text data to be synthesized to meet the input format requirements of the BERT model.

[0098] In some embodiments, obtaining the word embedding vector may include the following steps: extract the sub-word vectors in the text data to be synthesized; perform pooling processing on the sub-word vectors to obtain the word embedding vector.

[0099] Specifically, the sub-word embedding vectors processed and output by the BERT model, whose tokens are at the sub-word level, can be converted into word-level embeddings through average pooling with a sub-word to word mapping to unify with the text structure information, that is, word embedding vectors are obtained.

[0100] Step S402: Extract the text structure information of the text data to be synthesized, where the text structure information is used to represent the dependency relationship between words.

[0101] Input the text data to be synthesized into the graph sentiment semantic encoder, and the graph sentiment semantic encoder can perform semantic analysis and word dependency relationship analysis on the text data to be synthesized.

[0102] In some alternative embodiments, input the text data to be synthesized into the dependency analysis module of the graph sentiment semantic encoder to extract the text structure information of the text to be synthesized.

[0103] The dependency analysis module is used to perform dependency parsing on the text data to be synthesized. Dependency parsing is a natural language processing technique whose purpose is to identify the dependency relationship between individual words in a sentence. In natural language processing, dependency parsing can help understand the semantic structure of a sentence and better perform tasks such as text analysis, information extraction, and speech recognition.

[0104] Specifically, the dependency analysis module identifies the dependency relationship between words in the text data to be synthesized and constructs a dependency tree. The dependency relationship between words refers to the dependency relationship of one word on another word, and this dependency relationship can be a lexical, grammatical, or semantic relationship (such as subject-predicate relationship, verb-object relationship, parallel relationship, etc.). Constructing a dependency tree means organizing all words into a tree-structured data, with each word as a node and the dependency relationship between words as tree edges.

[0105] The text structure information of the text data to be synthesized extracted by the dependency analysis module can be presented or saved in the form of a dependency tree.

[0106] In some alternative embodiments, a directed graph can be used to represent the sentences in the text data to be synthesized in the form of a dependency tree.

[0107] Among them, the structure of the directed graph is composed of vertices and edges. Taking the directed graph G composed of the set V and the set E as an example, where V is the set of vertices and E is the set of edges. The directed graph G can be defined as: G = (V, E). The node v ∈ V represents a token with sentiment semantic information, usually a word. The directed edge can be defined as: e = (vi, vj) ∈ E, which is used to represent a specific dependency relationship from the node vi (i.e., the head node) to the node vj (i.e., the dependent node).

[0108] Step S403: Based on the word embedding vectors and the text structure information, construct a dependency graph.

[0109] Among them, the text structure information is presented in the form of a dependency tree, which can include multiple word nodes and multiple directed edges for representing the dependency relationships between words. Constructing a dependency graph may include the following steps: using word embedding vectors as the word nodes of the text structure information to construct a dependency graph.

[0110] In some alternative embodiments, multiple word embedding vectors obtained through a BERT model are used as multiple word nodes of the dependency tree, and the dependency relationships between words are represented by directed edges between nodes.

[0111] Since the nodes of the dependency tree are labeled at the word level, while the vectors output by the BERT model are labeled at the sub-word level, using T to represent the sentence in the text data to be synthesized, the representations of the sentence T at these two levels are respectively:

[0112]

[0113] where w i is the i-th word, sw j is the j-th sub-word, and n and m are the numbers of words and sub-words in the sentence. Then the dependency structure and BERT embedding vectors can be obtained, which are represented by the following formulas:

[0114] [b1,b2,b j ,...,b m =BERT(SW)

[0115] E dep =DP(W)

[0116] where b j is the j-th sub-word level embedding vector output by BERT, and E dep is a set of directed edges with dependency parsing labels, which can be regarded as the dependency structure of W.

[0117] By performing average pooling on the vectors output by the BERT model with the sub-word to word mapping relationship, the sub-word level embedding vectors are converted into word level embedding, and the pooling process can be represented by the following formula:

[0118] v i =AugPool(b i1 ,b i2 ,...,b ik )

[0119] V sem =[V1,V2,V i ,...,V n

[0120] where i1 to ik are the indices of the sub-words belonging to the i-th word. V i ​is the word-level embedding vector of the i-th word. After pooling, the word embedding vector with sentiment semantic information can be represented by the node set V sem is denoted as

[0121] Considering that there can be interactions between the head nodes and their subordinate nodes in the directed graph G, by reversing the direction of each edge in E dep and retaining the dependency parsing labels of the edges to introduce another set of directed edges E rev . At the same time, the original E dep is renamed as E fwd . Then, the set E fwd and the set E rev are regarded as two independent relationship types, and the node set V sem serves as the word nodes connected by the directed edges in the set E fwd and the set E rev .

[0122] Construct positive and negative relationship graphs through the node set V sem , the set E fwd and the set E rev . The positive relationship graph and the negative relationship graph can be respectively expressed as:

[0123] G fwd =(V sem ,E fwd )

[0124] G rev =(V sem ,E rev )

[0125] G fwd and G rev together constitute the dependency graph of the text data to be synthesized. Among them, the sentiment semantic information of the words in the text data to be synthesized will flow directly or indirectly to the relevant nodes through the structures of G fwd and G rev .

[0126] Step S404: Enhance the sentiment semantics of the dependency graph through a gated graph network to obtain sentiment semantic features.

[0127] Among them, the gated graph network can be a Relational Gated Graph Network (RGGN), which is used to naturally integrate specific syntactic dependency relationship information into the semantic representation to further enhance the text information, thereby improving the expressiveness and prosody effect of the synthesized speech.

[0128] Specifically, enhancing the sentiment semantics of the dependency graph through a relational gated graph network may include the following steps: enhancing the sentiment semantics of the dependency graph composed of the positive relational graph and the directional relational graph mentioned above through the relational gated graph network.

[0129] In one iteration of the relational gated graph network, each word node first aggregates the sentiment semantic information from its adjacent incoming nodes, and the aggregation weight depends on the dependency relation category corresponding to the edge. Then, it updates its own hidden state through a Gated Recurrent Unit (GRU) network. After multiple propagation iterations, the sentiment semantic information flows to all relevant nodes along the graph topology and specific dependency relations. The iterative processing of the gated graph network can be expressed by the following formula:

[0130]

[0131]

[0132] where N(i) is the set of neighbors of node i, and W eij represents the weight parameter of the edge from node i to j during the propagation iteration determined by the dependency parsing label. t represents the t-th iteration. is the weighted sum of the hidden states of the neighbors of node i GRU(·) represents the gated aggregator, which combines the information from the neighbors to update the hidden state and the previous iteration step of node i, and passes the hidden state of the node after the last iteration step to the fully connected layer.

[0133] In some embodiments, enhancing the sentiment semantics of the dependency graph through the gated graph network may include the following steps: enhancing the sentiment semantics from the direction of the directed edges of the dependency graph through the first gated graph network module; enhancing the sentiment semantics from the reverse direction of the directed edges of the dependency graph through the second gated graph network module.

[0134] where the direction of the directed edges of the dependency graph is the positive relational graph, and the reverse direction of the directed edges is the reverse relational graph. The first gated graph network module and the second gated graph network module are respectively used to perform sentiment semantic enhancement processing on the positive relational graph and the reverse relational graph. The first gated graph network module can be expressed as: RGGN fwd , and the second gated graph network module can be expressed as: RGGN rev , where RGGN fwd and RGGN revHaving the same structure but with independent and non - shared weight parameters, this can ensure that the forward and backward information transmission paths do not interfere with each other. The emotional semantic enhancement process is carried out through the first gated graph network module and the second gated graph network module, which can be represented by the following formula:

[0135]

[0136]

[0137]

[0138]

[0139] Where are the representations of the i - th word in W learned from the forward and backward relationship graphs respectively, is and is the element - adding operation to obtain the final semantic representation R bi is the semantic representation sequence obtained by re - ordering ri based on the word - level bi - level.

[0140] After passing and updating, the corresponding node representations from the outputs of the two gated graph network modules are added together and restored to a sequence form as the output emotional semantic features after emotional semantic enhancement processing.

[0141] Exemplarily, for the graph emotional semantic encoder to extract emotional semantic features, reference can be made to Figure 5 , Figure 5 which shows the data flow of extracting emotional semantic features based on the graph emotional editor in some embodiments. Among them, the arrow direction in the figure represents the data flow.

[0142] In some embodiments, step S103 may include: generating a syntax tree corresponding to the text data according to the text data to be synthesized; parsing the syntactic relationships between words in the syntax tree, using phonemes as nodes and the syntactic relationships between phonemes as edges to generate a syntax graph; based on a bidirectional gated recurrent unit network, bidirectionally encoding the syntactic relationships between two phonemes in the syntax graph to obtain relationship encoding data; where the relationship encoding data includes a forward relationship encoding vector and a backward relationship encoding vector; calculating attention scores based on the graph attention network of the graph encoder and the relationship encoding data; encoding the phoneme data according to the attention scores to obtain text encoding features.

[0143] Specifically, the graph encoder may include a text encoding unit and a relationship encoding unit.

[0144] The relationship encoding unit converts the syntax tree of the input text data to be synthesized into a syntax graph that describes the global relationships among the involved input data.

[0145] Regarding the related concepts of the syntax tree and the syntax graph, they are similar to the dependency tree and the dependency graph involved in the previous text. Through dependency parsing, it is used to mine the syntactic relationships between two words in a sentence.

[0146] The syntax graph is an extension of the syntax tree, turning one-way connections into two-way connections by adding reverse connections. In addition, self-loop edges are introduced, and each word has a specific label. In this way, the words (or phonemes) in a sentence can be represented by nodes, and their connection relationships are represented by edges. Through two-way connections, a word can directly receive and send information to any other word, regardless of whether they are directly related.

[0147] To model the relationship between two nodes (phonemes), the relationship between node pairs is described as the shortest relationship path between them. A recurrent neural network with GRU can be used to convert the relationship sequence into a distributed representation.

[0148] For example, the shortest relationship path sp between node i and node j i→j is represented as [sp1,..., sp t ,..., sp n+1 = [e(i, k1), e(k1, k2),..., e(k n , j)], where e(·, ·) represents the edge label, and k1:n are relay nodes. A bidirectional GRU is used for path relationship encoding:

[0149]

[0150]

[0151] where represents the forward relationship encoding vector, represents the backward relationship encoding vector, GRU f represents the forward GRU network, GRU b represents the backward GRU network, and the last hidden states of the forward and backward GRU networks are concatenated to form the final relationship encoding r ij = [s n+1 ; s0]. The final relationship encoding represents the linguistic relationship between two words (or phonemes). Constructing the relationship encoding provides a global view of how to collect and distribute information for the graph encoder model in speech synthesis.

[0152] The graph encoder in the embodiments of the present application incorporates syntactic relation encoding into the self-attention mechanism to indicate the relationships between characters (or phonemes). The purpose of the graph encoder is to convert the input character embedding sequence (or phoneme sequence) and relation encoding into a corresponding syntax-driven character embedding sequence (or phoneme sequence). By incorporating the explicit relation representation between two nodes in the syntax graph into the graph attention calculation, a syntax-aware graph attention mechanism is formed, which can be simply referred to as syntax-aware graph attention. Further, the final encoded representation can be calculated by stacking multiple syntax-aware graph attention and feed-forward layer modules. In each module, the character embedding sequence is updated based on all other character embedding sequences and the corresponding relation encoding. The encoded result of the character embedding sequence of the last module can be fed into the variational adapter and decoder to generate acoustic features.

[0153] In some embodiments, based on a bidirectional GRU network, the syntactic relationship between two phonemes is bidirectionally encoded to obtain relation encoding data, including: when the two phonemes belong to the same word, the syntactic relationship between the two phonemes is bidirectionally encoded based on the bidirectional GRU network and using the self-loop edge encoding algorithm; when the two phonemes belong to different words, the syntactic relationship between the words to which the two phonemes belong respectively is bidirectionally encoded based on the bidirectional GRU network.

[0154] In this embodiment, the basic unit of the sentence in the text data to be synthesized can be a phoneme token. The relation encoding between words in NLP (Natural Language Processing) can be extended to the relation encoding between phonemes. When two phonemes belong to the same word, the syntactic relationship between the two phonemes can be bidirectionally encoded using the self-loop edge encoding algorithm; when two phonemes belong to different words, the syntactic relationship between the words to which the two phonemes belong respectively is bidirectionally encoded based on the bidirectional GRU network. By extending the relation encoding between words to the relation encoding between phonemes, the encoding accuracy can be improved.

[0155] In some embodiments, the bidirectional encoding of the syntactic relationship between two phonemes can be based on the shortest path between the two phonemes for relation encoding.

[0156] In some embodiments, calculating the attention score based on the graph attention mechanism of the graph encoder and the relation encoding vector includes: capturing the addressing relationship based on the basic syntactic content in the relation encoding by the graph attention mechanism of the graph encoder; calculating the forward relation deviation between phonemes according to the forward relation encoding vector; controlling the backward relation deviation between phonemes according to the backward relation encoding vector; calculating the comprehensive deviation based on the forward relation encoding vector and the backward relation encoding vector; and obtaining the attention score according to the addressing relationship, forward relation deviation, backward relation deviation, and comprehensive deviation.

[0157] Specifically, in order to also encode the connection direction between phonemes when calculating attention, the relationship encoding vector r ij is divided into a forward relationship encoding vector r i→j and a backward relationship encoding vector r j→i ; [r i→j ; r j→i = W rrij .

[0158] Then, a syntax-aware graph attention mechanism is used to calculate the attention scores, and the scores are based on the phoneme representations and their bidirectional relationship representations as follows:

[0159]

[0160] Among them, (a) represents the addressing relationship that captures the basic syntax content. (b) represents the forward relationship deviation between phonemes calculated according to the forward relationship encoding vector. (c) represents controlling the backward relationship deviation between phonemes according to the backward relationship encoding vector; (d) represents the comprehensive deviation (encoding the general relationship deviation) obtained based on the forward relationship encoding vector and the backward relationship encoding vector.

[0161] In some embodiments, the text data to be synthesized based on the graph encoder can use Stanza to extract the syntactic dependency parsing tree. The relationship encoding unit performs relationship encoding in two directions based on bidirectional GRU. N = 6 encoding blocks are used in the text encoding unit of the graph encoder. The multi-head attention in the graph attention unit in the graph encoder and the decoder has 4 heads. The input text data to be synthesized is first converted into a 256-dimensional character embedding sequence and then used as the input of the encoder.

[0162] Exemplarily, the graph encoder can refer to Figure 6 as shown Figure 6 shows a schematic data flow diagram of text feature encoding based on the graph encoder 600 in some embodiments, and the arrow direction in the figure represents the data flow.

[0163] In some embodiments, the predicted data can be input into the decoder for decoding, and the decoder generates a series of 80-channel mel-spectrum acoustic features as output. The mel-spectrum acoustic features with a frame extraction size of 50 ms and a frame shift of 12.5 ms are extracted and further normalized to zero mean and unit variance as the reference target. The Adam optimization training algorithm (a stochastic optimization method proposed by OpenAI) can be used, and the values of the corresponding training parameters can be β1 = 0.9, β2 = 0.98, and the same learning rate table as in the Transformer model (essentially an Encoder-Decoder architecture, a classic NLP model launched by Google in 2017) can be adopted.

[0164] Exemplarily, the decoder may refer to Figure 7 as shown Figure 7 FIG. shows a schematic diagram of the data flow for decoding based on the decoder 700 in some embodiments, and the arrow direction in the figure indicates the data flow direction.

[0165] It should be understood that although Figure 1 and Figure 4 each step in the flowcharts of Figure 1 and Figure 4 is shown sequentially according to the indication of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise clearly stated in this document, the execution of these steps has no strict order limit, and these steps may be executed in other orders. Moreover,[

[0166] In a second aspect, embodiments of the present disclosure provide a speech synthesis device, as Figure 8 shown, the speech synthesis device 800 includes: an emotional semantic extraction module 801, a front-end processing module 802, a graph encoding module 803, a variational adaptation module 804, and an audio synthesis module 805.

[0167] The emotional semantic extraction module 801 is configured to analyze the emotional semantic information and word dependency relationships of the text data to be synthesized by using a graph emotional semantic encoder, and perform emotional semantic enhancement to obtain emotional semantic features.

[0168] The front-end processing module 802 is configured to convert the text data to be synthesized into phoneme data.

[0169] The graph encoding module 803 is configured to perform syntax relationship encoding on the text data according to the syntax graph of the text data to be synthesized by using a graph encoder to obtain relationship encoding features, and perform text encoding on the phoneme data by using the relationship encoding features based on a graph attention mechanism to obtain text encoding features.

[0170] The variational adaptation module 804 is configured to splice the emotional semantic features and the text encoding features to obtain spliced features, and perform duration prediction on the spliced features to obtain prediction data.

[0171] The audio synthesis module 805 is configured to convert the spectrum data to be synthesized into synthesized audio.

[0172] In some embodiments, the emotion semantic extraction module 801 may include an emotion semantic information extraction unit, a dependency relationship extraction unit, a dependency graph construction unit, and an emotion semantic enhancement unit.

[0173] The emotion semantic information extraction unit is used to extract the emotion semantic information of the text data to be synthesized, and obtain word embedding vectors, which are used to represent the emotion semantic information between words.

[0174] The dependency relationship extraction unit is used to extract the text structure information of the text data to be synthesized, and the text structure information is used to represent the dependency relationship between words.

[0175] The dependency graph construction unit is used to construct a dependency graph based on the word embedding vectors and the text structure information.

[0176] The emotion semantic enhancement unit is used to perform emotion semantic enhancement on the dependency graph through a gated graph network to obtain emotion semantic features.

[0177] In some embodiments, the text structure information may include multiple word nodes and directed edges for representing dependency relationships. The dependency graph construction unit may include a construction subunit, and the construction subunit is used to use the word embedding vectors as the word nodes of the text structure information to construct a dependency graph.

[0178] In some embodiments, the emotion semantic enhancement unit may include a first enhancement subunit and a second enhancement subunit.

[0179] The first enhancement subunit is used to perform emotion semantic enhancement from the direction of the directed edges of the dependency graph through a first gated graph network module.

[0180] The second enhancement subunit is used to perform emotion semantic enhancement from the reverse direction of the direction of the directed edges of the dependency graph through a second gated graph network module.

[0181] In some embodiments, the emotion semantic information extraction unit may include an extraction subunit and a pooling subunit.

[0182] The extraction subunit is used to extract sub-word vectors from the text data to be synthesized.

[0183] The pooling subunit is used to perform pooling processing on the sub-word vectors to obtain word embedding vectors.

[0184] In some embodiments, the graph encoding module 803 may include a syntax graph generation unit, a bidirectional encoding unit, and a text feature encoding calculation unit.

[0185] The syntax graph generation unit is used to generate a syntax tree corresponding to the text data according to the text data to be synthesized, parse the syntax relationship between the words and phrases in the syntax tree, use phonemes as nodes, and the syntax relationship between phonemes as edges to generate a syntax graph.

[0186] The bidirectional encoding unit is used to perform bidirectional encoding on the syntactic relationship between two phonemes in the syntactic graph based on a bidirectional gated recurrent unit network, and obtain relationship encoding data. Among them, the relationship encoding data includes a forward relationship encoding vector and a backward relationship encoding vector.

[0187] The text feature encoding calculation unit is used to calculate attention scores based on the graph attention network of the graph encoder and the relationship encoding data, and encode the phoneme data according to the attention scores to obtain text encoding features.

[0188] In some embodiments, the text feature encoding calculation unit may include an addressing relationship acquisition subunit, a relationship deviation determination subunit, and an attention score calculation subunit.

[0189] The addressing relationship acquisition subunit is used to capture the addressing relationship of the basic syntactic content in the relationship encoding based on the graph attention network of the graph encoder.

[0190] The relationship deviation determination subunit is used to calculate the forward relationship deviation between phonemes according to the forward relationship encoding vector, control the backward relationship deviation between phonemes according to the backward relationship encoding vector, and calculate the comprehensive deviation based on the forward relationship encoding vector and the backward relationship encoding vector.

[0191] The attention score calculation subunit is used to obtain attention scores according to the addressing relationship, forward relationship deviation, backward relationship deviation, and comprehensive deviation.

[0192] In some embodiments, the bidirectional encoding unit can perform bidirectional encoding in different cases. When two phonemes belong to the same word, the bidirectional encoding unit is used to perform bidirectional encoding on the syntactic relationship between the two phonemes based on the bidirectional gated recurrent unit network and using the self-loop edge encoding algorithm; when two phonemes belong to different words, the bidirectional encoding unit is used to perform bidirectional encoding on the syntactic relationship between the words to which the two phonemes belong respectively based on the bidirectional gated recurrent unit network.

[0193] For the specific limitations of the speech synthesis device 800, reference may be made to the limitations on the speech synthesis method in the foregoing text, which will not be elaborated here. Each module in the foregoing speech synthesis device 800 may be implemented in whole or in part by software, hardware, and their combination. The foregoing modules may be embedded in or independent of the processor in the computer device in the form of hardware, or may be stored in the memory in the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the foregoing modules.

[0194] In a third aspect, embodiments of the present disclosure provide a computer device, which may be a terminal, and its internal structure diagram may be as Figure 9As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it realizes the corresponding steps in the voice synthesis method in any embodiment of the first aspect of the present disclosure. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0195] Those skilled in the art can understand that Figure 9 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0196] In the fourth aspect, the embodiments of the present disclosure provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it realizes the corresponding steps in the voice synthesis method in any embodiment of the first aspect of the present disclosure.

[0197] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in this application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0198] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0199] The above embodiments only represent several implementation manners of this application, and their descriptions are relatively specific and detailed. However, it should not be construed as a limitation to the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of the patent of this application should be subject to the appended claims.

Claims

1. A voice synthesis method, characterized in that, The method includes: Using a graph emotion semantic encoder to analyze the emotion semantic information and word - to - word dependency relationships of the text data to be synthesized, and performing emotion semantic enhancement to obtain emotion semantic features; Converting the text data to be synthesized into phoneme data; Using a graph encoder to perform grammar relationship encoding on the text data to be synthesized according to the grammar graph of the text data to be synthesized, obtaining relationship encoding features, and based on a graph attention mechanism, using the relationship encoding features to perform text encoding on the phoneme data to obtain text encoding features; Concatenating the emotion semantic features and the text encoding features to obtain concatenated features; Performing duration prediction on the concatenated features to obtain prediction data; Decoding the prediction data to obtain the spectrum data to be synthesized; Converting the spectrum data to be synthesized into a synthesized audio.

2. The method according to claim 1, characterized in that The step of using a graph emotion semantic encoder to analyze the emotion semantic information and word - to - word dependency relationships of the text data to be synthesized, and performing emotion semantic enhancement to obtain emotion semantic features includes: Extracting the emotion semantic information of the text data to be synthesized to obtain word embedding vectors, where the word embedding vectors are used to represent the emotion semantic information between words; Extracting the text structure information of the text data to be synthesized, where the text structure information is used to represent the dependency relationships between words; Based on the word embedding vectors and the text structure information, constructing a dependency graph; Performing emotion semantic enhancement on the dependency graph through a gated graph network to obtain the emotion semantic features.

3. The method according to claim 2, characterized in that, The text structure information includes multiple word nodes and directed edges for representing dependency relationships; the step of constructing a dependency graph based on the word embedding vectors and the text structure information includes: Using the word embedding vectors as the word nodes of the text structure information to construct the dependency graph.

4. The method according to claim 3, wherein The step of performing emotion semantic enhancement on the dependency graph through a gated graph network includes: Performing emotion semantic enhancement from the direction of the directed edges of the dependency graph through a first gated graph network module; Performing emotion semantic enhancement from the reverse direction of the direction of the directed edges of the dependency graph through a second gated graph network module.

5. The method according to claim 2, characterized in that The step of extracting the emotion semantic information of the text data to be synthesized to obtain word embedding vectors includes: Extracting sub - word vectors from the text data to be synthesized; Performing pooling processing on the sub - word vectors to obtain the word embedding vectors.

6. The method according to claim 1, characterized in that The step of using a graph encoder to perform grammar relationship encoding on the text data to be synthesized according to the grammar graph of the text data to be synthesized, obtaining relationship encoding data, and based on a graph attention mechanism, using the relationship encoding data and the phoneme data to perform text encoding to obtain text encoding features includes: Generating a grammar tree corresponding to the text data to be synthesized according to the text data to be synthesized; Parsing the grammar relationships between words in the grammar tree, using phonemes as nodes and the grammar relationships between phonemes as edges to generate the grammar graph; Based on a bidirectional gated recurrent unit network, performing bidirectional encoding on the grammar relationships between two phonemes in the grammar graph to obtain the relationship encoding data; where the relationship encoding data includes a forward relationship encoding vector and a backward relationship encoding vector; Calculate attention scores based on the graph attention network of the graph encoder and the relation-encoded data; Encode the phoneme data according to the attention scores to obtain the text encoding features.

7. The method according to claim 6, characterized in that, The calculating of the attention scores based on the graph attention network of the graph encoder and the relation-encoded data includes: The graph attention network based on the graph encoder captures the addressing relationship of the basic syntactic content in the relation encoding; Calculate the forward relation deviation between phonemes according to the forward relation encoding vector; Control the backward relation deviation between phonemes according to the backward relation encoding vector; Calculate the comprehensive deviation based on the forward relation encoding vector and the backward relation encoding vector; Obtain the attention scores according to the addressing relationship, the forward relation deviation, the backward relation deviation, and the comprehensive deviation.

8. The method according to claim 6, wherein The bi-directionally encoding of the syntactic relationship between two phonemes in the syntactic graph based on the bidirectional gated recurrent unit network to obtain relation-encoded data includes: When two phonemes belong to the same word, based on the bidirectional gated recurrent unit network and using the self-loop edge encoding algorithm to bi-directionally encode the syntactic relationship between the two phonemes; When two phonemes belong to different words, based on the bidirectional gated recurrent unit network to bi-directionally encode the syntactic relationship between the words to which the two phonemes respectively belong.

9. A voice synthesis device, characterized in that, The apparatus includes: An emotional semantic extraction module, configured to analyze the emotional semantic information and the word dependency relationship of the text data to be synthesized by using a graph emotional semantic encoder, and perform emotional semantic enhancement to obtain emotional semantic features; A front-end processing module, configured to convert the text data to be synthesized into phoneme data; A graph encoding module, configured to use a graph encoder to perform syntactic relationship encoding on the text data to be synthesized according to the syntactic graph of the text data to be synthesized to obtain relation encoding features, and perform text encoding on the phoneme data by using the relation encoding features based on a graph attention mechanism to obtain text encoding features; A variational adaptation module, configured to splice the emotional semantic features and the text encoding features to obtain spliced features, and perform duration prediction on the spliced features to obtain prediction data; An audio synthesis module, configured to convert the spectrum data to be synthesized into synthesized audio.

10. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images