Text-to-speech generation method, device and electronic device based on syntactic graph construction

Abstract

The present application relates to the field of computer technology, and discloses a text-to-speech generation method, device and electronic device based on syntactic graph construction, the method comprising obtaining a text to be processed and a target reference speech; determining text information and phoneme information corresponding to the text to be processed according to the text to be processed; constructing a network based on the text information and the target syntactic graph in the target speech generation model to generate a target syntactic graph corresponding to the text to be processed; generating a target word-level encoding corresponding to the text to be processed based on the phoneme information, boundary information, the target syntactic graph, and the target encoding network in the target speech generation model; generating a target synthesized speech based on the target word-level encoding, the target reference speech, and the target speech generation network in the target speech generation model. The target synthesized speech generated by the text-to-speech method provided in the embodiment of the present application has the rhythmic features of the text to be processed, which improves the authenticity and richness of the synthesized speech.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and electronic device for text-to-speech generation based on syntactic graph construction. Background Art

[0002] Usually, in the text-to-speech synthesis process, more emphasis is placed on replicating the speaker's voice, that is, ensuring that the synthesized speech fits the speaker's voice characteristics (such as pitch, timbre, etc.) as closely as possible. However, for synthesized speech, its presentation effect is not only determined by the speaker's voice characteristics, but also depends on the text information corresponding to the speech. For example, when the text information is a thriller novel, the synthesized speech has a faster and deeper rhythmic characteristics in the presentation effect. However, in the text-to-speech synthesis process, the related technology does not consider the rhythmic features corresponding to the text information, resulting in the rhythm of the synthesized speech not being able to fully fit the text information, thereby reducing the richness and realism of the synthesized speech. Summary of the invention

[0003] In order to solve the above problems, the embodiments of the present application provide a text-to-speech generation method, device and electronic device based on syntactic graph construction, which can consider the prosodic features corresponding to the text information in the text-to-speech synthesis process to improve the realism and richness of the synthesized speech. Specifically, the embodiments of the present application disclose the following technical solutions:

[0004] A first aspect of an embodiment of the present application provides a text-to-speech generation method based on syntactic graph construction, the method comprising: first, obtaining a text to be processed and a target reference speech; second, determining text information and phoneme information corresponding to the text to be processed based on the text to be processed; wherein the text information comprises text content and boundary information; then, constructing a network based on the text information and a target syntactic graph in a target speech generation model to generate a target syntactic graph corresponding to the text to be processed; wherein the text content comprises a plurality of characters, and the target syntactic graph is used to characterize the syntactic relationship between each of the plurality of characters; thirdly, generating a target word-level code corresponding to the text to be processed based on the phoneme information, the boundary information, the target syntactic graph, and the target coding network in the target speech generation model; finally, generating a target synthesized speech based on the target word-level code, the target reference speech, and the target speech generation network in the target speech generation model; wherein the target synthesized speech comprises the prosodic features of the text to be processed.

[0005] In some embodiments, a target syntactic graph construction network is constructed based on text information and a target speech generation model to generate a target syntactic graph corresponding to the text to be processed, including: generating an initial dependency graph corresponding to the text information based on text content, word boundaries in boundary information, and a dependency parser in the syntactic graph construction network; wherein the initial dependency graph includes multiple word nodes and unidirectional connection relationships between each word node in the multiple word nodes; generating a target syntactic graph corresponding to the initial dependency graph based on the initial dependency graph and a syntactic graph builder in the syntactic graph construction network.

[0006] In some embodiments, a syntactic graph builder in a network is constructed based on an initial dependency graph and a syntactic graph to generate a target syntactic graph corresponding to the initial dependency graph, including: based on the syntactic graph builder and the word boundaries in the boundary information, each word node in the initial dependency graph is divided into at least one byte node, and the first byte node in each word node is determined; a first dependency connection relationship is established between multiple first byte nodes, a start node, and an end node; according to the intra-word order of at least one byte node in each word node, a second dependency connection relationship is established between at least one byte node in each word node; according to the first dependency relationship and the second dependency connection relationship, a target syntactic graph is obtained; wherein the first dependency relationship and / or the second dependency relationship is used to characterize the bidirectional connection relationship between each byte node.

[0007] In some embodiments, based on phoneme information, boundary information, a target syntactic graph, and a target encoding network in a target speech generation model, a target word-level code corresponding to the text to be processed is generated, including: based on the phoneme information, word boundaries in the boundary information, and a word-level average pooling layer in the target encoding network, an initial word-level code corresponding to the text to be processed is generated; based on the initial word-level code, the target syntactic graph, and multiple gated graph convolutional layers in the target encoding network, a target word-level code is generated.

[0008] In some embodiments, based on the phoneme information, the word boundaries in the boundary information, and the word-level average pooling layer in the target encoding network, an initial word-level encoding corresponding to the text to be processed is generated, including: based on the phoneme information and the word boundaries, determining at least one phoneme feature corresponding to each byte in the text to be processed; performing an average pooling operation on at least one phoneme feature corresponding to each byte to determine the comprehensive phoneme feature corresponding to each byte; and determining the initial word-level encoding corresponding to the text to be processed according to the comprehensive phoneme feature corresponding to each byte.

[0009] In some embodiments, a target word-level code is generated based on an initial word-level code, a target syntactic graph, and multiple gated graph convolutional layers in a target coding network, including: generating a first word-level code based on the initial word-level code, the target syntactic graph, and a first gated graph convolutional layer among multiple gated graph convolutional layers; generating a second word-level code based on the first word-level code, the target syntactic graph, and a second gated graph convolutional layer among multiple gated graph convolutional layers; determining a target word-level code based on the initial word-level code, the first word-level code, and the second word-level code.

[0010] In some embodiments, a target synthesized speech is generated based on a target word-level encoding, a target reference speech, and a target speech generation network in a target speech generation model, including: generating a target style vector based on the target word-level encoding, the target reference speech, and a style vector unit in the target speech generation network; wherein the target style vector is used to characterize the prosodic information corresponding to the target reference speech and the prosodic information corresponding to the text to be processed; and generating a target synthesized speech based on the target style vector, the text to be processed, and a generation unit in the target speech generation network.

[0011] In some embodiments, the method also includes: obtaining sample text data and sample reference speech data; generating predicted synthesized speech based on the sample text data, the sample reference speech data, and the speech generation model to be trained; wherein the speech generation model to be trained includes a syntactic graph construction network to be trained and a coding network to be trained; obtaining sample synthesized speech data; using the predicted synthesized speech as the initial training output information of the speech generation model to be trained and the sample synthesized speech data as supervision information, iterating the speech generation model to be trained to obtain a target speech generation model; wherein the target speech generation model includes a target syntactic graph construction network and a target coding network.

[0012] The second aspect of the embodiment of the present application provides a text-to-speech generation device based on a syntactic graph, comprising: an acquisition module, a determination module, a first generation module, a second generation module, and a third generation module. The acquisition module is configured to acquire a text to be processed and a target reference speech. The determination module is configured to determine the text information and phoneme information corresponding to the text to be processed according to the text to be processed; wherein the text information includes text content and boundary information. The first generation module is configured to construct a network based on the text information and the target syntactic graph in the target speech generation model to generate a target syntactic graph corresponding to the text to be processed; wherein the text content includes multiple characters, and the target syntactic graph is used to characterize the syntactic relationship between each of the multiple characters. The second generation module is configured to generate a target word-level encoding corresponding to the text to be processed based on the phoneme information, boundary information, the target syntactic graph, and the target encoding network in the target speech generation model. The third generation module is configured to generate a target synthesized speech based on the target word-level encoding, the target reference speech, and the target speech generation network in the target speech generation model; wherein the target synthesized speech includes the prosodic features of the text to be processed.

[0013] A third aspect of an embodiment of the present application provides an electronic device, comprising: a processor and a memory, the memory being used to store computer-executable instructions; the processor being used to read the instructions from the memory and execute the instructions to implement the text-to-speech generation method based on syntactic graph construction as described in the first aspect above.

[0014] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores computer program instructions. When a computer reads the instructions, the text-to-speech generation method based on syntactic graph construction described in the first aspect is executed.

[0015] A fifth aspect of an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the text-to-speech generation method based on syntactic graph construction described in the first aspect above.

[0016] A sixth aspect of the embodiments of the present application provides a computer program, which, when executed by a processor, can implement the text-to-speech generation method based on syntactic graph construction described in the first aspect above.

[0017] The text-to-speech generation method based on syntactic graph construction provided in the embodiment of the present application generates a target synthesized speech corresponding to the text to be processed and the target reference speech through a pre-trained target speech generation model. First, a network is constructed based on the text information corresponding to the text to be processed and the target syntactic graph in the target speech generation model to generate the target syntactic graph corresponding to the text to be processed; then, based on the phoneme information and boundary information corresponding to the text to be processed, the target syntactic graph, and the target encoding network in the target speech generation model, the target word-level encoding corresponding to the text to be processed is generated; finally, based on the target word-level encoding, the target reference speech, and the target speech generation network in the target speech generation model, the target synthesized speech is generated. In the text-to-speech generation method based on syntactic graph construction provided in the embodiment of the present application, since the target syntactic graph generated by the target syntactic graph construction network can represent the syntactic relationship between each character in a plurality of characters, it is more in line with the syntactic information of Chinese syntax and can provide prosodic features that are more in line with the text information. Therefore, the target word-level encoding generated according to the target syntactic graph can have the prosodic features corresponding to the text to be processed, so that when the target synthetic speech is generated based on the target word-level encoding and the target reference speech, the generated target synthetic speech also has the prosodic features of the text to be processed. Therefore, the text-to-speech generation method based on syntactic graph construction provided in the embodiment of the present application can extract the prosodic features of the text information in the text-to-speech synthesis process to improve the realism and richness of the synthetic speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of a text-to-speech generation method based on a syntactic graph provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a target speech generation model provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of another text-to-speech generation method based on syntactic graph construction provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of an initial dependency graph and a target syntax graph provided in an embodiment of the present application;

[0023] Figure 5A schematic diagram of another text-to-speech generation method based on syntactic graph construction provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of another text-to-speech generation method based on syntactic graph construction provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of another text-to-speech generation method based on syntactic graph construction provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of another target speech generation model provided in an embodiment of the present application;

[0027] Fig. 9 A schematic diagram of another text-to-speech generation method based on syntactic graph construction provided in an embodiment of the present application;

[0028] Fig.10 A schematic diagram of another text-to-speech generation method based on syntactic graph construction provided in an embodiment of the present application;

[0029] Fig.11 A schematic diagram of an X-vector feature vector extractor provided in an embodiment of the present application;

[0030] Fig.12 A schematic diagram of another text-to-speech generation method based on syntactic graph construction provided in an embodiment of the present application;

[0031] Fig.13 A schematic diagram of a target speech generation network provided in an embodiment of the present application;

[0032] Fig.14 A schematic diagram of a text-to-speech generation device based on a syntactic graph provided in an embodiment of the present application;

[0033] Fig.15 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention and to make the above-mentioned purposes, features and advantages of the embodiments of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0035] Text to Speech (TTS) is the process of converting written text data into audible voice data for output. Usually, in the process of implementing text to speech, more emphasis is placed on replicating the voice of the reference speaker, that is, making the synthesized speech as close as possible to the voice characteristics of the reference speaker, such as the pitch, timbre and rhythm of the reference speaker.

[0036] However, the quality of the presentation of a synthesized speech depends not only on the voice characteristics of the reference speaker, but also on the text information corresponding to the speech. For example, when the text information is a thriller, the synthesized speech is more likely to have fast and deep rhythmic features in the presentation effect; and when the text information is nonfiction, the synthesized speech is more likely to have slow and steady rhythmic features in the presentation effect. However, the method of synthesizing speech in the related art cannot generate speech that fits the text information, making the synthesized speech not realistic and rich enough, and unable to provide users with a better experience.

[0037] In order to solve the above problems, an embodiment of the present application provides a text-to-speech generation method based on a syntactic graph, which can extract the prosodic features of text information during the process of synthesizing speech, and generate synthesized speech based on the prosodic features of the text information, thereby ensuring that the synthesized speech has the prosodic features of the text information, improving the authenticity and richness of the synthesized speech, and providing users with a better user experience.

[0038] The following is combined with Figures 1 to 13 The text-to-speech generation method based on syntactic graph construction provided in an embodiment of the present application is described.

[0039] Figure 1 A schematic diagram of a text-to-speech generation method based on a syntactic graph provided in an embodiment of the present application. Figure 1 As shown, the text-to-speech method includes steps 110 to 150 as shown below.

[0040] Step 110: Obtain the text to be processed and the target reference speech.

[0041] For example, in the process of text-to-speech generation, the information of the text to be processed needs to be expressed with the sound information of the target reference speech to obtain the target synthesized speech that is finally generated. Therefore, before generating the target synthesized speech, it is necessary to obtain the text to be processed and the target reference speech.

[0042] In some examples, the text to be processed is used to provide text information to the target synthesized speech. The text to be processed can be a paragraph of text, such as a thriller or a nonfiction piece; or, the text to be processed can be a sentence or a phrase, such as "Monkeys like to eat bananas". The present application embodiment does not limit the text to be processed.

[0043] In some examples, the target reference speech is used to provide sound information to the target synthesized speech, and the sound information includes the prosody information of the target speaker's speech, such as timbre and pitch. The target reference speech can be the target speaker's speech, such as the target reference speech can be a segment of the target speaker's speech of a preset duration. The target speaker can be a real person or a virtual person, which is not limited by the embodiments of the present application.

[0044] Step 120: Determine text information and phoneme information corresponding to the text to be processed according to the text to be processed.

[0045] Exemplarily, the text to be processed can be parsed to obtain text information and phoneme information corresponding to the text to be processed, and the phoneme information corresponds to the text information. Among them, the text information is the written form of the language, which can reflect the direct meaning, grammatical structure, contextual association and other information of the language. The phoneme information is the actual pronunciation of the phoneme sequence formed after further conversion of the text information, which can reflect the pronunciation characteristics and acoustic performance of the language.

[0046] In some examples, after obtaining text information based on the text to be processed, each word in the text information can be converted into its corresponding phoneme sequence based on the text information and the phoneme dictionary, so as to obtain the phoneme information corresponding to the text information. The phoneme dictionary can be a pre-set dictionary library, such as a phoneme dictionary that can be a resource library built according to the target language; or the phoneme dictionary can also be an existing phoneme dictionary library in the relevant technology, which is not limited in the embodiments of the present application.

[0047] Exemplarily, the text information may include text content and boundary information, wherein the boundary information includes word boundaries and character boundaries.

[0048] In some examples, the text content is the plain text corresponding to the text information. The text content is used to express the meaning of the text. The text content may include multiple characters, for example, when the text content includes words (such as characters or letters), numbers, and punctuation marks. The boundary information is the separation position between words or characters in the text content. For example, the boundary information may also be the separation between two adjacent words or two adjacent characters. Among them, the separation between words can be called word boundaries, and the separation between characters can be called character boundaries.

[0049] For example, when the text to be processed is "Monkeys like to eat bananas", the word boundaries are the separation between "monkey" and "like", "like" and "eat", and "eat" and "banana"; the character boundaries are the separation between "monkey" and "son", "son" and "like", "like" and "happy", "happy" and "eat", "eat" and "fragrant", and "fragrant" and "banana".

[0050] The embodiment of the present application can characterize the essential characteristics of the text to be processed from two levels by converting the text to be processed into text information and phoneme information. The text information focuses on characterizing the semantic structure and vocabulary meaning corresponding to the text to be processed, and the phoneme information focuses on the pronunciation details and voice characteristics corresponding to the text to be processed, thereby providing necessary input information for the subsequent text-to-speech generation process.

[0051] Figure 2 A schematic diagram of a target speech generation model provided in an embodiment of the present application. Figure 2 As shown, the target speech generation model 1 includes a syntax graph construction sub-model 100 and a generation sub-model 200. It should be noted that the text-to-speech generation method based on syntax graph construction provided in the embodiment of the present application can be implemented based on the target speech generation model 1.

[0052] Exemplarily, the syntax graph construction sub-model 100 includes a target syntax graph construction network 10 and a target encoding network 20. The target syntax graph construction network 10 may include a dependency parser 11 and a syntax graph builder 12; the target encoding network 20 may include a word-level average pooling layer 21 and a plurality of gated graph convolution layers 22.

[0053] Exemplarily, the generation submodel 200 includes the target speech generation network 30. It should be noted that the generation submodel 200 can be a generator in the speech generation model in the related art, or can be an improved generator in the embodiment of the present application. The embodiment of the present application does not limit this.

[0054] Combine the following Figure 2 The target speech generation model 1 in the embodiment of the present application further illustrates the text-to-speech generation method based on syntactic graph construction.

[0055] Step 130, constructing a network based on the text information and the target syntactic graph in the target speech generation model to generate a target syntactic graph corresponding to the text to be processed.

[0056] Exemplarily, the text content includes a plurality of characters, and the target syntax graph is used to represent the syntax relationship between the plurality of characters. Wherein, when the text content is a Chinese sentence, the characters are used to represent the Chinese characters in the Chinese sentence.

[0057] In some examples, a syntactic graph is a graphical representation of text information, and the syntactic graph can characterize the syntactic relationships between the words in the text information. Through the syntactic relationships between the words, the prosodic features of each word can be obtained, including pronunciation features, intonation features, etc.

[0058] The target syntactic graph generated by the target syntactic construction network in the embodiments of the present application includes the syntactic relationships between the characters. Compared with the syntactic graph in the related art that can only characterize the syntactic relationships between the words, it is more in line with the syntactic information of Chinese syntax and can provide more prosodic features that conform to the text information.

[0059] For example, for a Chinese sentence, the prosody corresponding to the same character in different sentences may be different, that is, the same character may have different pronunciations in different sentences. For example, the pronunciation of the character "鲜" in "新鲜" and "鲜为人知" is different, that is, the prosody is different.

[0060] Therefore, when determining the prosody of a Chinese character, it is necessary to combine the sentence in which it is located, that is, the dependency relationship between the Chinese character and other characters in the sentence. The target syntactic graph provided by the embodiments of the present application can characterize the dependency relationship between the characters. Therefore, more accurate and rich prosodic features of the text to be processed can be further obtained through the target syntactic graph.

[0061] Figure 3 It is a schematic diagram of another text-to-speech generation method based on syntactic graph construction provided by the embodiments of the present application. As Figure 3 shown, the above step 130 includes steps 1310 to 1320 as shown below.

[0062] Step 1310, based on the text content, the word boundaries in the boundary information, and the dependency parser in the syntactic graph construction network, generate an initial dependency graph corresponding to the text information.

[0063] Continue to refer to Figure 2 , input the text content and the boundary information, such as the word boundaries in the text content and the boundary information, into the dependency parser 11 in the target syntactic graph construction network 10, and after being processed by the dependency parser 11, an initial dependency graph corresponding to the text information is generated.

[0064] In some examples, the initial dependency graph is a directed graph based on a unidirectional structure. The initial dependency graph includes multiple word nodes and the unidirectional connection relationships between the word nodes in the multiple word nodes.

[0065] Figure 4 It is a schematic diagram of an initial dependency graph and a target syntactic graph provided by the embodiments of the present application. Among them, Figure 4(a) in FIG. 1 is a schematic diagram of an initial dependency graph. For example, when the text information is “Monkeys like to eat bananas”, after passing through the dependency parser 11, the generated initial dependency graph is Figure 4 As shown in (a) in .

[0066] like Figure 4 As shown in (a) in the figure, the initial dependency graph includes four word nodes, "monkey", "like", "eat" and "banana", among which "like" is the parent node, "monkey", "eat" and "banana" are three child nodes, and a unidirectional connection relationship is established between two adjacent word nodes. Since "like" is the parent node, the unidirectional connection relationship between "monkey" and "like" is "like" pointing to "monkey", the unidirectional connection relationship between "like" and "eat" is "like" pointing to "eat", and the unidirectional connection relationship between "eat" and "banana" is "eat" pointing to "banana".

[0067] In some examples, the initial dependency graph can characterize the prosodic features corresponding to the text information. However, the prosodic information determined by the initial dependency graph is not accurate enough. On the one hand, since the initial dependency graph has a unidirectional connection relationship between the word nodes, that is, the word nodes in the sentence cannot obtain any information from other word nodes, that is, from other non-adjacent word nodes during the graph aggregation process, resulting in the prosodic features of the acquired text information being inaccurate. On the other hand, since the nodes of the initial dependency graph are word nodes, the pronunciation rules of each character in Chinese words vary depending on the sentence or word. Therefore, the initial dependency graph is used to encode the text information. It is impossible to present the pronunciation features of each character in the text information, which may affect the effect of subsequent synthesized speech. Therefore, when it comes to a more complex sentence, its corresponding rhythm, such as pronunciation, pauses, stress, etc., is also more complex, and directly using the initial dependency graph cannot better reflect the prosodic features in the sentence.

[0068] In order to more accurately obtain the prosodic features corresponding to the text information, the embodiment of the present application introduces a syntactic graph builder 12 in the target syntactic graph construction network 10, and further processes the initial dependency graph through the syntactic graph builder 12 to obtain richer prosodic features in the text information.

[0069] Step 1320, constructing a syntactic graph builder in the network based on the initial dependency graph and the syntactic graph, and generating a target syntactic graph corresponding to the initial dependency graph.

[0070] Exemplarily, the syntax graph builder 12 is used to convert the initial dependency graph into a target syntax graph, wherein the target syntax graph is a directed graph based on a bidirectional structure, and the target syntax graph includes a plurality of sub-nodes and bidirectional connection relationships between each of the plurality of sub-nodes.

[0071] Continue to refer to Figure 2 After obtaining the initial dependency graph output by the dependency parser 11, the initial dependency graph can be input into the syntax graph builder 12 to obtain the target syntax graph.

[0072] In some examples, the target syntax graph is based on the initial dependency graph, and bidirectional edges are added between word nodes, so that the information flow in the target syntax graph becomes a bidirectional information flow. That is, for any pair of parent nodes and child nodes, the target syntax graph includes both the forward edge from the parent node to the child node and the reverse edge from the child node to the parent node.

[0073] Figure 5 A schematic diagram of another text-to-speech generation method based on syntactic graph construction provided in an embodiment of the present application. Figure 5 As shown, the above step 1320 includes steps 1321 to 1324 as shown below.

[0074] Step 1321, based on the syntax graph builder and the word boundaries in the boundary information, divide each word node in the initial dependency graph into at least one sub-node, and determine the first sub-node in each word node.

[0075] Exemplarily, the syntax graph builder 12 may further split each word node in the initial dependency graph into at least one sub-node based on word boundaries, and establish dependency relationships between the sub-nodes.

[0076] For example, for a Chinese sentence (i.e., text content), the same character may have different pronunciation rhythms or pronunciations in different words or sentences, which determines that the phonemes of the character may be different. In order to enable the target syntactic graph to more accurately represent the pronunciation rules of Chinese, the syntactic graph builder 12 provided in the embodiment of the present application can expand each word in the initial dependency graph as a node to each word as a node, that is, after splitting each word node into byte nodes, establish a dependency relationship between each byte node, so as to obtain more accurate pronunciation and rhythm information.

[0077] Continue to refer to Figure 4 , Figure 4 (b) in is a schematic diagram of a target syntax graph. For example, Figure 4 The initial dependency graph of (a) in FIG. 1 can be generated as follows after being processed by the syntax graph builder 12: Figure 4 The target syntax graph shown in (b) in .

[0078] like Figure 4As shown in (b) of [the figure], in the target syntactic graph, the word node "monkey" is split into two character nodes "猴" and "子", the word node "喜欢" is split into two character nodes "喜" and "欢", the word node "喜蕉" is split into two character nodes "香" and "蕉", and the character node corresponding to the word node "吃" is also "吃".

[0079] In some examples, after dividing each word node into at least one character node, the first character node in each word node can be determined according to the in-word order of the character nodes in the word node to which they belong. For example, in the word node "猴子", the in-word order is "猴" and "子", so the first character node in this word node is "猴". By analogy, the first character nodes corresponding to each word node can be determined to be "猴", "喜", "吃", "香" in sequence.

[0080] Step 1322, establish the first dependency connection relationship among multiple first character nodes, start nodes, and end nodes.

[0081] Exemplarily, the target syntactic graph also includes a start node (which can also be called BOS) and an end node (which can also be called EOS). Among them, the start node is connected to the first character node in the first word node in the text content, and the end node is connected to the first character node in the last word node in the text content.

[0082] In some examples, the first dependency connection relationship can be established among the first character nodes, start nodes, and end nodes corresponding to multiple word nodes respectively. Among them, the first dependency connection relationship can be a bidirectional connection relationship. For example, according to the adjacent relationship of multiple word nodes in the initial dependency graph, the first dependency relationship among the first character nodes in each adjacent node can be established. It should be noted that the target syntactic graph is established based on the initial dependency graph, so the order of each word node is the same as the order of each word node in the initial dependency graph.

[0083] As Figure 4 shown in (b) of [the figure], in the target syntactic graph, solid arrows are used to represent the first dependency connection relationship, and there are bidirectional connections (i.e., the first dependency connection relationship) between "BOS" and "猴", "猴" and "喜", "喜" and "吃", "吃" and "香", and "香" and "EOS".

[0084] Step 1323, establish the second dependency connection relationship among at least one character node in each word node according to the in-word order of at least one character node in each word node.

[0085] In some examples, after dividing each word node into at least one character node, a second dependency relationship can be established between at least one character node in each word node according to the in-word order of the character nodes in the corresponding word node. Among them, the second dependency connection relationship can be a bidirectional connection relationship.

[0086] As Figure 4 shown in (b) of , the dashed arrows are used to represent the second dependency relationship. There is a bidirectional connection (i.e., the second dependency connection relationship) between the character node "hou" and the character node "zi" in the word node "monkey". Similarly, there are bidirectional connections (i.e., the second dependency connection relationships) between "xi" and "huan", and between "xiang" and "jiao".

[0087] Step 1324, obtain a target syntactic graph according to the first dependency relationship and the second dependency connection relationship.

[0088] Among them, the first dependency relationship and / or the second dependency relationship are used to represent the bidirectional connection relationship between each character node.

[0089] In some examples, after establishing the first dependency connection relationship and the second dependency connection relationship between each character node, a target syntactic graph as shown in (b) of can be obtained. Figure 4 can be obtained.

[0090] In the embodiment of the present application, the initial dependency graph can be converted into a target syntactic graph through the syntactic graph builder 12. Since all character nodes in the target syntactic graph are bidirectionally connected, more accurate pronunciation and prosody features in the text information can be obtained, which also provides a better input for generating the target character-level encoding subsequently. That is to say, the improvement of the syntactic graph extraction process in the embodiment of the present application obtains syntactic information that is more in line with Chinese syntax, and further makes the prosody features more in line with the prosody features of the input text in the subsequent prosody features.

[0091] Step 140, generate a target character-level encoding corresponding to the text to be processed based on the phoneme information, boundary information, target syntactic graph, and the target encoding network in the target speech generation model.

[0092] Referring to Figure 2 , after obtaining the target syntactic graph, the phoneme information, boundary information, and target syntactic graph corresponding to the text to be processed can be input into the target encoding network 20, and the target encoding network 20 outputs the target character-level encoding.

[0093] Figure 6 is a schematic diagram of another text-to-speech generation method based on syntactic graph construction provided by the embodiment of the present application. As Figure 6 shown, the above step 140 includes steps 1410 to 1420 as shown below.

[0094] Step 1410, based on the phoneme information, the word boundaries in the boundary information, and the word-level average pooling layer in the target encoding network, generate an initial word-level encoding corresponding to the text to be processed.

[0095] In some examples, the boundary information includes word boundaries. Figure 2 As shown, the phoneme information and word boundaries can be input into the word-level average pooling layer 21 in the target encoding network 20 to generate an initial word-level encoding.

[0096] In some embodiments, step 1410 includes: determining at least one phoneme feature corresponding to each byte in the text to be processed based on phoneme information and word boundaries; performing an average pooling operation on at least one phoneme feature corresponding to each byte to determine a comprehensive phoneme feature corresponding to each byte; and determining an initial word-level encoding corresponding to the text to be processed based on the comprehensive phoneme feature corresponding to each byte.

[0097] In some examples, after obtaining the phoneme information corresponding to the text to be processed, the phoneme information needs to be further processed. The phoneme information corresponding to the text to be processed may be a phoneme coding sequence. Since each byte (such as a Chinese character in a Chinese sentence) can usually be composed of multiple phonemes, the coding sequence corresponding to each byte is determined in the phoneme coding sequence corresponding to the text to be processed.

[0098] Exemplarily, the word boundaries can be used to determine which adjacent phonemes in the phoneme encoding sequence corresponding to the text to be processed are the phonemes corresponding to a byte point. After determining the phoneme corresponding to each byte point, an average pooling operation can be performed on at least one phoneme corresponding to the same byte point, so as to obtain the comprehensive phoneme features of each byte point. After obtaining the comprehensive phoneme features of each byte point, each byte point is encoded to determine the initial word-level encoding corresponding to the text to be processed. That is, in the embodiment of the present application, encoding is performed in units of characters rather than words.

[0099] The embodiment of the present application adopts character-level average pooling to generate character-level encoding in units of characters. Compared with the word-level encoding in the related art, more accurate rhythmic information in the text to be processed can be obtained.

[0100] Step 1420, generating a target word-level encoding based on the initial word-level encoding, the target syntactic graph, and multiple gated graph convolutional layers in the target encoding network.

[0101] In some examples, such as Figure 2As shown, after obtaining the initial word-level code output by the word-level average pooling layer 21, the initial word-level code and the target syntactic graph can be input into a plurality of gated graph convolution layers 22 in the target encoding network 20, and the target word-level code is determined according to the output of each gated graph convolution layer 22. The target word-level code can represent the corresponding prosodic features in the text to be processed.

[0102] Exemplarily, the target coding network 20 may include multiple gated graph convolution layers 22. For example, the target coding network 20 may include two gated graph convolution layers, such as a first gated graph convolution layer and a second gated graph convolution layer. Alternatively, the target coding network 20 may also include more gated graph convolution layers, and the embodiment of the present application does not limit the number of gated graph convolution layers 22. The embodiment of the present application takes the example that the target coding network 20 may include a first gated graph convolution layer and a second gated graph convolution layer as an example for exemplary description.

[0103] In some examples, the target syntax graph input to each gated graph convolution layer 22 may be a target syntax graph obtained by processing the output result of the syntax graph builder 12. In other words, the output result of the syntax graph builder 12 may be converted into a data type that can be processed by the gated graph convolution layer 22 and input to each gated graph convolution layer 22.

[0104] For example, the target syntactic graph input to the gated graph convolution layer 22 may include a node embedding feature vector and a graph structure feature vector corresponding to the target syntactic graph. The node embedding feature vector is used to characterize the features of each byte node in the target syntactic graph, and the graph structure feature vector is used to characterize the relationship between each byte node in the target syntactic graph.

[0105] In some examples, the gated graph convolution layer 22 is used to extract the syntactic information of each byte node and the dependency relationship between each byte node in the target syntax graph. For example, each gated graph convolution layer 22 may include 5 iterations to extract the dependency relationship between each byte node in the target syntax graph.

[0106] The embodiment of the present application can further extract the phoneme features of each byte in the text to be processed according to the initial word-level encoding and the target syntactic graph through the gated graph convolution layer 22, so as to obtain richer and more accurate prosodic information of each byte in the text to be processed.

[0107] Figure 7 A schematic diagram of another text-to-speech generation method based on syntactic graph construction provided in an embodiment of the present application. Figure 7 As shown, the above step 1420 includes steps 1421 to 1423 as shown below.

[0108] Step 1421, generating a first word-level code based on the initial word-level code, the target syntactic graph, and a first gated graph convolutional layer among a plurality of gated graph convolutional layers.

[0109] Step 1422: Generate a second word-level code based on the first word-level code, the target syntactic graph, and a second gated graph convolutional layer among the plurality of gated graph convolutional layers.

[0110] Step 1423, determining a target word-level code based on the initial word-level code, the first word-level code and the second word-level code.

[0111] In some examples, first, the initial word-level code output by the word-level average pooling layer 21 and the target syntactic graph output by the target syntactic graph construction network 10 are input to the first gated convolution layer, and the first word-level code is output after being processed by the first gated convolution layer; then, the first word-level code output by the first gated convolution layer and the target syntactic graph output by the target syntactic graph construction network 10 are input to the second gated graph convolution layer, and the second word-level code is output after being processed by the second gated convolution layer. Finally, the initial word-level code output by the word-level average pooling layer 21, the first word-level code output by the first gated convolution layer, and the second word-level code output by the second gated convolution layer are summed to obtain the target word-level code.

[0112] It should be noted that a larger number of gated graph convolutional layers 22 may be set in the target coding network 20 for superposition, and the output of each gated graph convolutional layer may be summarized to output the target word-level code. The processing method of a larger number of gated graph convolutional layers is similar to the above steps 1421 to 1423, and will not be repeated here.

[0113] The text-to-speech generation method based on syntax graph construction provided in the embodiment of the present application obtains the target syntax graph corresponding to the text to be processed through the target syntax graph construction network in the target speech generation model, and then obtains the target word-level code corresponding to the text to be processed according to the target syntax graph and the target coding network 20. Since the target syntax graph represents the bidirectional dependency between each byte in the text information, the syntactic information that is more in line with the Chinese syntax can provide a rhythmic feature that is more in line with the text information, and the target word-level code obtained based on the target syntax graph can better reflect the rhythmic features of each byte in the text to be processed. Therefore, when the target word-level code is used as the input of the subsequent generation sub-model 200 for speech synthesis, the generated target synthesized speech can have the rhythmic features of the text to be processed, which improves the realism and richness of the target synthesized speech.

[0114] Step 150, generating a target synthesized speech based on the target word-level encoding, the target reference speech, and the target speech generation network in the target speech generation model.

[0115] Exemplarily, the target synthesized speech includes prosodic features of the text to be processed.

[0116] In some examples, such as Figure 2 As shown, after the target word-level code is obtained by constructing the sub-model 100 through the syntactic graph, the target word-level code and the target reference speech can be input into the target speech generation network 30 in the generation sub-model 200 to generate the target synthesized speech corresponding to the text to be processed and the target reference speech.

[0117] It should be noted that the target speech generation network 30 can adopt a generator used for speech synthesis in the relevant technology, or it can be a generator improved by the embodiment of the present application. The input of the generator in the embodiment of the present application is the target word-level encoding. Therefore, whether it is a generator in the relevant technology or a generator improved by the embodiment of the present application, it can be achieved that the final synthesized speech has the rhythmic characteristics of the text to be processed, and the authenticity and richness of the final synthesized speech are improved. For example, the embodiment of the present application takes the improved generator (i.e., the target speech generation network 30) as an example to exemplify the process of generating the target synthesized speech.

[0118] In some embodiments, the target speech generation network 30 includes a style vector unit and a generation unit.

[0119] Combine the following Figures 8 to 13 The target speech generation network 30 provided in the embodiment of the present application is further described.

[0120] Figure 8 Schematic diagram of another target speech generation model provided in an embodiment of the present application. Figure 8 As shown, the target speech generation network 30 includes a style vector unit 31 and a generation unit 32. The style vector unit 31 is used to further extract the prosodic information corresponding to the text to be processed and the prosodic information corresponding to the target reference speech; the generation unit 32 is used to generate the final target synthesized speech according to the prosodic information (such as the first prosodic information and the second prosodic information) extracted by the style vector unit.

[0121] Fig. 9 A schematic diagram of another text-to-speech generation method based on syntactic graph construction provided in an embodiment of the present application. Fig. 9 As shown, the above step 150 includes steps 1510 to 1520 as shown below.

[0122] Step 1510, generating a target style vector based on the target word-level encoding, the target reference speech, and the style vector unit in the target speech generation network.

[0123] Exemplarily, the target style vector may be used to represent first prosody information corresponding to the text to be processed and second prosody information corresponding to the target reference speech.

[0124] In some embodiments, the style vector unit 31 includes a style diffusion sampler and a feature extractor.

[0125] Fig.10 A schematic diagram of another text-to-speech generation method based on syntactic graph construction provided in an embodiment of the present application. Fig.10 As shown, the above step 1510 includes steps 1511 to 1513 as shown below.

[0126] Step 1511: Generate a first prosody vector corresponding to the text to be processed based on the target word-level encoding and the style diffusion sampler in the style vector unit.

[0127] The first prosody vector is used to represent the first prosody information. That is, the style diffusion sampler in the style vector unit 31 can extract the prosody feature (ie, the first prosody feature) corresponding to the text to be processed based on the target word-level encoding.

[0128] It should be noted that the style diffusion sampler in the embodiment of the present application can be a pre-trained style diffusion sampler. The style diffusion sampler gradually converts the sample data (such as the target word-level encoding) into a Gaussian distribution, thereby learning the relationship between the sample data and the noisy data, and thus recovering the sample data from the Gaussian distribution. The style diffusion sampler can improve the denoising ability of the target speech generation model, making the target speech generation model more resistant to noise.

[0129] In some examples, during the training of the style diffusion sampler, first, a sample word-level code is obtained (wherein the sample word-level code can be obtained through steps 110 to 140 above); then, Gaussian noise is gradually added to the sample word-level code in an iterative manner until the sample vector corresponding to the sample word-level code becomes a pure Gaussian noise vector.

[0130] For example, the sample word-level encoding is y0, the number of iterations is t, and the sample vector after adding noise is y t , then y t =k t *y t-1 That is, in every y t-1 Based on this, add a tiny Gaussian noise k t , until y t becomes a pure Gaussian noise vector. Where k indicates the variance, k t Indicates the variance value corresponding to different cycle times t, k t It can be calculated in advance based on the standard Gaussian distribution, that is, for any t, the determined k can be directly calculated t Used for model training.

[0131] On the basis of the above, randomly select t and determine the corresponding k through the above calculation process t With y t , and use the above parameters as the training basis for the model. Among them, y0 and t are training inputs, y t The training output is used to train the style diffusion sampler (also called diffusion model), which can make the style diffusion sampler learn the relationship between sample data and noisy data. It should be noted that during the training process, due to the large change in the model loss in the early stage of training, the random selection of t can effectively avoid the model from over-concentrating on the information of a certain time period in the early stage of training, thereby ignoring the information of other time periods. Based on this, the random distribution of t should be as uniform as possible.

[0132] After completing the training based on sufficient sample data, by inversely processing the trained parameters, the style diffusion sampler can restore the data from the Gaussian distribution through a reverse process, thereby obtaining the rhythm vector (i.e., the first rhythm vector) corresponding to the text to be processed.

[0133] In the embodiment of the present application, by inputting the target word-level encoding output by the syntactic graph construction sub-model 100 into the trained style diffusion sampler in the style vector unit 31, the output prosodic vector (i.e., the first prosodic vector) corresponding to the text to be processed can further improve the accuracy of the prosodic information corresponding to the text to be processed, thereby providing a better input for the subsequent generation of the target synthesized speech.

[0134] Step 1512: Generate a second prosody vector corresponding to the target reference speech based on the target reference speech and the feature extractor in the style vector unit.

[0135] The second prosody vector is used to represent the second prosody information. That is, the feature extractor in the style vector unit 31 can extract the prosody feature (ie, the second prosody feature) corresponding to the target reference speech.

[0136] In some embodiments, the feature extractor may include an X-vector feature extractor, wherein the X-vector feature extractor may include multiple time-delay network layers and multiple fully connected network layers.

[0137] In some embodiments, the above step 1512 includes: generating a reference audio feature corresponding to the target reference speech based on the target reference speech; and passing the reference audio feature through multiple time-delay network layers and multiple fully connected network layers in sequence to generate an X vector of a preset dimension.

[0138] Exemplarily, the reference audio feature may include a Mel Frequency Cepstral Coefficient (MFCCs) feature, and the second prosody vector may include an X vector.

[0139] In some examples, the X-vector feature vector extractor uses a deep neural network (DNN) to map the target reference speech to an embedding of a preset fixed dimension, namely, an X-vector. Among them, the X-vector can better represent the prosodic information (i.e., the second prosodic information) corresponding to the target reference speech, and can represent the timbre information, high pitch information, and energy information of the target reference speech.

[0140] Fig.11 Schematic diagram of an X-vector feature vector extractor provided in an embodiment of the present application. Fig.11 As shown, the X-vector feature vector extractor includes multiple time-delay network layers and multiple fully connected network layers.

[0141] It should be noted that the embodiment of the present application does not limit the number of delay network layers and fully connected network layers in the X-vector feature vector extractor. Fig.11 In the figure, an X-vector feature vector extractor including 5 delay network layers (i.e., delay network layer 1 to delay network layer 5) and 3 fully connected network layers (i.e., fc 1 to fc 3) is taken as an example for exemplary description.

[0142] like Fig.11 As shown in Figure 1, the parameters of each delay network layer can be expressed as F(N, D, K). The parameters F of each delay network layer are activated by the context of the previous layer with a width of K and an extended length of D, and N is the dimension of the output vector.

[0143] For example, when the target reference speech features are 24-dimensional Mel-frequency cepstrum features (hereinafter referred to as MFCC features) and the frame length is 25ms, the feature map output by the delay network layer 1 undergoes 1D convolution, input channel × convolution kernel size × output channel = 24 × 5 × 512; the feature maps output by the delay network layer 2 and the delay network layer 3 undergo 1D convolution, input channel × convolution kernel size × output channel = 512 × 3 × 512; the feature map output by the delay network layer 4 undergoes 1D convolution, input channel × convolution kernel size × output channel = 512 × 1 × 512; the feature map output by the delay network layer 5 undergoes 1D convolution, input channel × convolution kernel size × output channel = 512 × 1 × 1500. After the output of the delay network layer 5, the output feature vector is further statistically pooled through fc1, fc2, and fc3.

[0144] For example, statistical pooling processing can be to calculate the mean and standard deviation in the dimension of sequence length T, and then connect the mean and standard deviation in series. Therefore, after statistical pooling, the dimension of sequence length T will disappear, and 1500 means and 1500 standard deviations will be obtained, which can be connected in series to obtain a vector of length 3000. Among them, fc1, fc2 and fc3 can all be standard fully connected network layers, and fc3 can be represented by the feature vector of the target reference speech, which contains the rhythm information and phoneme information corresponding to the target reference speech.

[0145] The embodiment of the present application inputs the target reference speech into a feature extractor (such as an X-vector feature extractor) in the style vector unit 31 to extract the prosody vector (i.e., the second prosody vector) corresponding to the target reference speech, thereby providing a better input for the subsequent generation of the target synthesized speech.

[0146] Step 1513: Generate a target style vector based on the first prosody vector and the second prosody vector.

[0147] After the first prosody vector is acquired in step 1511 and the second prosody vector is acquired in step 1512 , a target style vector may be generated based on the first prosody vector and the second prosody vector.

[0148] In some embodiments, the above step 1513 includes: performing a sum operation on the first prosody vector and the second prosody vector to obtain an initial style vector; and performing a temporal average pooling process on the initial style vector to determine a target style vector.

[0149] In some examples, the temporal average pooling process may determine an optimal style vector from multiple initial style vectors as the target style vector.

[0150] In some examples, Temporal Average Pooling can be an operation performed on time series data, the main purpose of which is to reduce the complexity of the time dimension (such as T) while retaining important information on each feature channel (such as F). For example, Temporal Average Pooling can average all values ​​on each time dimension (T) and turn each feature channel into a single value.

[0151] For example, the temporal average pooling process can transform a feature map of dimension (batchsize, F, T) into a feature vector of dimension (batchsize, F). In the temporal average pooling process, the dimension of T disappears. Therefore, temporal average pooling can be regarded as extracting the feature that best represents the feature map information from the features of a series of frames, and transforming the variable-length frame sequence corresponding to the most representative feature into a fixed-length feature vector (i.e., the target style vector). For example, the feature vectors of multiple frames can be averaged to determine the target style vector.

[0152] Step 1520, generating a target synthesized speech based on the target style vector, the text to be processed, and the generation unit in the target speech generation network.

[0153] Exemplarily, the target synthesized speech has prosodic features corresponding to the text to be processed (ie, first prosodic features) and prosodic features corresponding to the reference speech (ie, second prosodic features).

[0154] In some embodiments, the generation unit 32 includes a phoneme encoder, a variance adapter, a mel-spectrogram encoder, and a vocoder.

[0155] Fig.12 A schematic diagram of another text-to-speech generation method based on a syntactic graph provided in an embodiment of the present application. Fig.12 The process of generating the target synthesized speech by the generating unit 32 is described. Fig.12 As shown, the above step 1520 includes steps 1521 to 1523 as shown below.

[0156] Step 1521, based on the text to be processed, the target style vector and the phoneme encoder, generate a phoneme state sequence corresponding to the text to be processed.

[0157] In some embodiments, the phoneme encoder comprises at least one style adaptive normalization layer.

[0158] Exemplarily, the phoneme encoder further includes a first convolutional layer, a first fully connected layer, a plurality of first conversion modules and a second fully connected layer.

[0159] In some embodiments, the above step 1521 includes: based on the text to be processed, determining the initial phoneme vector and the phoneme position code corresponding to the text to be processed; generating a first intermediate sequence based on the initial phoneme vector and the processing of the first convolutional layer and the first fully connected layer in the phoneme encoder; generating a phoneme state sequence based on the first intermediate sequence, the phoneme position code, the target style vector, and multiple first conversion modules and the second fully connected layer in the phoneme encoder.

[0160] In some examples, the phoneme encoder can convert the input text to be processed into a phoneme sequence, and further convert the phoneme sequence into a phoneme encoding result and output it, where the phoneme encoding result is represented by a phoneme hidden state sequence.

[0161] Exemplarily, the phoneme position code is used to represent the position information of each phoneme corresponding to the text to be processed in the text to be processed. The phoneme position code can be determined in different ways.

[0162] In some examples, the phoneme position coding can be determined by absolute position coding. For example, each phoneme in the text to be processed can be assigned an index (such as an integer index), such as the index of the first phoneme is 1, the index of the second phoneme is 2, and so on, and the position coding of each phoneme can be determined by the index information of each phoneme.

[0163] In other examples, the phoneme position coding can be determined by relative position coding. For example, the relative order between the phonemes can be used, such as encoding based on the distance between the phonemes. For example, the first phoneme coding can be [1,0,0,...], the second phoneme coding can be [0,1,0,...], and so on, and the position coding of each phoneme corresponding to the text to be processed can be determined. Alternatively, it is also possible to encode using a function, such as the sin / cos position embedding that can be used to determine the position coding of each phoneme.

[0164] In some other examples, the phoneme position coding may be determined by a coding method combining absolute position and relative position. For example, in some cases, the absolute position of a phoneme and its position relative to other phonemes may be considered at the same time, that is, the position coding of each phoneme may be determined by combining the two coding methods.

[0165] It should be noted that in the embodiment of the present application, the position coding of each phoneme in the text to be processed can be determined by any of the three methods mentioned above, or the position coding of each phoneme can be determined by other methods. The embodiment of the present application does not limit the method of phoneme position coding.

[0166] Exemplarily, the first conversion module may include a first attention mechanism layer, at least one style adaptive normalization layer and a second convolution layer. The number of first conversion modules in the phoneme encoder may be multiple, for example, 4, or more than 4, which is not limited in the embodiment of the present application.

[0167] Step 1522: Based on the phoneme state sequence and the variance adapter, preset variance information is added to the phoneme state sequence to obtain an adjusted phoneme state sequence.

[0168] Exemplarily, the preset variance information includes at least one of duration information, tone information and energy information.

[0169] In some examples, after a phoneme state sequence (also referred to as a phoneme hidden state sequence) is obtained through a phoneme encoder, the phoneme state sequence is input into a variance adapter. The variance adapter is used to add additional variance information to the phoneme state sequence, thereby obtaining an adjusted phoneme state sequence. The variance adapter can add information such as duration, pitch, and energy to the phoneme state sequence.

[0170] For example, the variance adapter can predict the duration of the phoneme in each duration through a duration predictor to adjust the length of the speech frame in the phoneme state sequence, and predict and adjust the corresponding pitch value and energy value through a pitch predictor and an energy predictor respectively.

[0171] Step 1523: based on the adjusted phoneme state sequence, the target style vector and the mel-spectrogram encoder, convert the adjusted phoneme state sequence into a mel-spectrogram sequence.

[0172] In some embodiments, the mel-spectrogram encoder includes at least one style adaptive normalization layer.

[0173] Exemplarily, the mel spectrum encoder further includes a third fully connected layer, a plurality of second conversion modules and a fourth fully connected layer.

[0174] In some embodiments, the above step 1523 includes: generating a second intermediate sequence based on the adjusted phoneme state sequence and the third fully connected layer in the mel-spectrogram encoder; generating a mel-spectrogram sequence based on the second intermediate sequence, the target style vector, and multiple second conversion modules and the fourth fully connected layer in the mel-spectrogram encoder.

[0175] Exemplarily, the second conversion module includes a second attention mechanism layer, at least one style adaptive normalization layer and a third convolution layer. The number of second conversion modules in the mel spectrum encoder can be multiple, for example, 4, or more than 4, which is not limited in the embodiment of the present application.

[0176] It should be noted that the structure of the second conversion module in the mel spectrum encoder and the first conversion module in the phoneme encoder may be the same, and the number of the second conversion modules and the number of the first conversion modules may also be the same.

[0177] Exemplarily, at least one style adaptive normalization layer is provided in each of the mel spectrum encoder and the phoneme encoder, and the target style vector generated by the style vector unit 31 can be input into each style adaptive normalization layer to further process the target style vector.

[0178] The following is an explanation of the processing process of the target style vector by the style adaptive normalization layer. It should be noted that the processing process of each style adaptive normalization layer in the mel spectrum encoder and the phoneme encoder is similar, and the style adaptive normalization layer mentioned in the following embodiments can be any style adaptive normalization layer in the mel spectrum encoder, and can also be any style adaptive normalization layer in the phoneme encoder.

[0179] In some examples, the style adaptive normalization layer can be used to add the target style vector generated by the style vector unit 32 to the process of speech synthesis. For example, the style adaptive normalization layer can adjust the synthesized speech based on the target style vector after receiving the target style vector. For example, assuming that the target style feature vector , where H is the dimension of the style feature vector, and the normalized vector output after the style adaptive normalization layer is .

[0180] In some examples, the normalized vector y may be determined by formula (1) and formula (2) as described below.

[0181] Formula (1);

[0182] Formula (2);

[0183] in, is the mean of the normal distribution, is the standard deviation of the normal distribution.

[0184] On this basis, the gain and bias of the target style vector can be further determined by formula (3) as shown below.

[0185] Formula (3);

[0186] Among them, g(w) and b(w) are the gain and bias parameters of the style adaptive normalization layer (also known as the affine layer and SALN layer), and the SALN layer is a single fully connected layer.

[0187] In some examples, unlike the fixed gain and bias in the traditional normalization layer, g(w) and b(w) can adaptively perform scaling and shifting of features in the target speech generation network 30 according to the target style vector itself. The embodiment of the present application adopts a style adaptive normalization layer (SALN) to replace the original normalization layer, and constructs an affine layer that converts the target style vector into the corresponding bias and gain according to the above method.

[0188] Step 1524, generating a target synthesized speech based on the mel-spectrogram sequence and the vocoder.

[0189] For example, after obtaining the mel-spectrogram sequence output by the mel-spectrogram encoder, the mel-spectrogram sequence can be input into a vocoder to determine the target synthesized speech that is finally generated.

[0190] In the process of generating a target synthesized speech through a target speech generation network, the text-to-speech generation method based on syntactic graph provided in the embodiment of the present application uses a style vector unit to extract the rhythm of the target word-level encoding and the target reference speech, and then calculates the target style vector; on the other hand, for the target style vector, the embodiment of the present application uses a style adaptive method to embed it into the generation process of the generation unit (such as a style adaptive normalization layer), so that the generated effect is more in line with the target reference speech and the target synthesized speech of the text to be processed.

[0191] Fig.13 A schematic diagram of a target speech generation network provided in an embodiment of the present application. Fig.13 As shown, the target speech generation network 30 includes a style vector unit 31 ( Fig.13 The left part of the ) and the generation unit 32 ( Fig.13 The right part in FIG. 1 ). The style vector unit 31 includes a style diffusion sampler, a prosodic text encoder and a V-vector feature vector extractor. The generation unit 32 includes a phoneme encoder, a variance adapter, a mel spectrum encoder and a vocoder.

[0192] Combine the following Fig.13 , the process of generating the target synthesized speech by the target speech generation network 30 is explained.

[0193] After obtaining the text to be processed, convert the text to be processed into a phoneme vector (such as 256 dimensions). Input the phoneme vector into the style vector unit 31. The prosodic text encoder and style diffusion sampler in the style vector unit 31 extract the first prosodic vector corresponding to the text to be processed based on the phoneme vector. Input the target reference speech into the style vector unit 31 to obtain the MCFF features of the target reference speech, and input the MCFF features of the target reference speech into the V-vector feature vector extractor in the style vector unit 31 to extract the second prosodic vector corresponding to the target reference speech. After summing the first prosodic vector and the second prosodic vector, a style vector is obtained. The style vector is subjected to temporal average pooling to obtain the target style vector.

[0194] The phoneme vector is input into the phoneme encoder in the generation unit 32. Specifically, the phoneme vector can be input into two one-dimensional convolutional layers with residual connections, and then into a fully connected layer. The number of filters of the one-dimensional convolutional layer in the phoneme encoder is 256, and the convolution kernel size is 3. After that, the phoneme vector processed by the fully connected layer is added to the phoneme position code (the phoneme position code is used for the position of the phoneme in the sentence), and the result (i.e., the first intermediate sequence) and the target style vector are input into 4 feedforward transmitters (Feedforward Transmitter using Fast Fourier Transform, FFT) blocks for calculation to obtain the phoneme encoding result, which is then output by the fully connected layer (i.e., the phoneme state sequence).

[0195] The phoneme encoding result output by the phoneme encoder is input into the variance adapter, which is used to extract three kinds of information from the phoneme encoding result as output. The three kinds of information include: phoneme duration, pitch (used to express emotion and rhythm information), and energy (used to express volume information).

[0196] After the variance adapter extracts the above information, the extracted information (such as the adjusted phoneme state sequence) is input into the Mel spectrum encoder. Specifically, the output result of the variance adapter can be input into two fully connected layers (such as a [128, 256] fully connected layer) for processing, and the processed result (i.e., the second intermediate sequence) is added to the phoneme position code, and the added result and the target style vector are input into four FFT blocks for calculation. Finally, a fully connected layer is used to adapt the dimension of the Mel spectrum and output the Mel spectrum (i.e., a Mel spectrum sequence).

[0197] like Fig.13 As shown, in the above process, each FFT block is composed of a multi-head attention mechanism, a style adaptive normalization, a one-dimensional convolution, and another style adaptive normalization in sequence. The target style vector output by the style vector unit 31 is embedded in each style adaptive normalization for processing. Among them, in each FFT block, the hidden layer size of the one-dimensional convolution can be 256, the number of attention heads can be 2, the convolution kernel size can be 9, and the number of filters can be 1024. All activation functions can be Mish activation functions, and the parameter size of the random inactivation layer can be set to 0.1.

[0198] Finally, the mel spectrum output by the mel spectrum encoder is output through the vocoder as the final synthesized speech, that is, the target synthesized speech.

[0199] The text-to-speech generation method based on syntactic graph construction provided in the embodiment of the present application generates a target synthesized speech corresponding to the text to be processed and the target reference speech through a pre-trained target speech generation model. First, a network is constructed based on the text information corresponding to the text to be processed and the target syntactic graph in the target speech generation model to generate the target syntactic graph corresponding to the text to be processed; then, based on the phoneme information and boundary information corresponding to the text to be processed, the target syntactic graph, and the target encoding network in the target speech generation model, the target word-level encoding corresponding to the text to be processed is generated; finally, based on the target word-level encoding, the target reference speech, and the target speech generation network in the target speech generation model, the target synthesized speech is generated. In the text-to-speech generation method based on syntactic graph construction provided in the embodiment of the present application, since the target syntactic graph generated by the target syntactic graph construction network can represent the syntactic relationship between each character in a plurality of characters, it is more in line with the syntactic information of Chinese syntax and can provide prosodic features that are more in line with the text information. Therefore, the target word-level encoding generated according to the target syntactic graph can have the prosodic features corresponding to the text to be processed, so that when the target synthetic speech is generated based on the target word-level encoding and the target reference speech, the generated target synthetic speech also has the prosodic features of the text to be processed. Therefore, the text-to-speech generation method based on syntactic graph construction provided in the embodiment of the present application can extract the prosodic features of the text information in the text-to-speech synthesis process to improve the realism and richness of the synthetic speech.

[0200] In some embodiments, the text-to-speech generation method based on syntactic graph construction also includes: obtaining sample text data and sample reference speech data; generating predicted synthesized speech based on the sample text data, the sample reference speech data, and the speech generation model to be trained; wherein the speech generation model to be trained includes a syntactic graph construction network to be trained and an encoding network to be trained; obtaining sample synthesized speech data; using the predicted synthesized speech as the initial training output information of the speech generation model to be trained and the sample synthesized speech data as supervision information, iterating the speech generation model to be trained to obtain a target speech generation model; wherein the target speech generation model includes a target syntactic graph construction network and a target encoding network.

[0201] Exemplarily, the target speech generation model 1 involved in the above embodiment is a pre-trained model. Among them, the syntactic graph construction submodel 100 and the generation submodel 200 in the target speech generation model 1 are both trained models. It should be noted that the syntactic graph construction submodel 100 and the generation submodel 200 can be trained as a whole or separately, and the embodiment of the present application does not limit this.

[0202] In some examples, the speech generation model to be trained may include a syntactic graph construction network to be trained and a coding network to be trained, and training the speech generation model to be trained includes training the syntactic graph construction network to be trained and the coding network to be trained, thereby obtaining a target syntactic graph construction network and a target coding network.

[0203] In some examples, the speech generation model to be trained also includes a speech generation sub-network to be trained, and training the training speech generation model includes training the speech generation sub-network to be trained, thereby obtaining a target speech generation network.

[0204] Exemplarily, during the model pre-training process, the audio corresponding to at least one reference speaker may be used as sample reference speech data for model training. For example, the number of reference speakers may be greater than a preset number of people, such as greater than 100. The reference speaker corresponds to at least 100 sample voices, and the audio length of each sample voice is greater than a preset audio length, such as greater than 3 seconds.

[0205] In some examples, after obtaining sample text data and sample reference speech data, a predicted synthesized speech can be generated based on the sample text data, the sample reference speech data, and the speech generation model to be trained. Then, the predicted synthesized speech is used as the initial training output information of the speech generation model to be trained, and the sample synthesized speech data is used as the supervision information to iterate the speech generation model to be trained to obtain the target speech generation model.

[0206] It should be noted that the text-to-speech generation method based on syntactic graph construction provided in the embodiment of the present application can not only make the rhythmic features of the acquired target synthesized speech more ideal, but also synthesize synthesized speech with different voice styles given other reference speech.

[0207] Fig.14 A schematic diagram of a text-to-speech generation device based on a syntactic graph provided in an embodiment of the present application. Fig.14 As shown, the text-to-speech generation device 1400 constructed based on the syntax graph includes: an acquisition module 1410, a determination module 1420, a first generation module 1430, a second generation module 1440 and a third generation module 1450. Among them:

[0208] The acquisition module 1410 is configured to: acquire the text to be processed and the target reference speech.

[0209] The determination module 1420 is configured to: determine the text information and phoneme information corresponding to the text to be processed according to the text to be processed; wherein the text information includes text content and boundary information.

[0210] The first generation module 1430 is configured to: construct a network based on the text information and the target syntax graph in the target speech generation model to generate a target syntax graph corresponding to the text to be processed; wherein the text content includes multiple characters, and the target syntax graph is used to represent the syntactic relationship between each of the multiple characters.

[0211] The second generation module 1440 is configured to generate a target word-level code corresponding to the text to be processed based on the phoneme information, the boundary information, the target syntactic graph, and the target coding network in the target speech generation model.

[0212] The third generation module 1450 is configured to generate a target synthesized speech based on the target word-level encoding, the target reference speech, and the target speech generation network in the target speech generation model; wherein the target synthesized speech includes the prosodic features of the text to be processed.

[0213] In some embodiments, the first generation module 1430 is configured to: generate an initial dependency graph corresponding to the text information based on the text content, word boundaries in the boundary information, and a dependency parser in the syntactic graph construction network; wherein the initial dependency graph includes multiple word nodes and a unidirectional connection relationship between each word node in the multiple word nodes; generate a target syntactic graph corresponding to the initial dependency graph based on the initial dependency graph and the syntactic graph builder in the syntactic graph construction network.

[0214] In some embodiments, the first generation module 1430 is configured to: divide each word node in the initial dependency graph into at least one byte node based on the syntactic graph builder and the word boundaries in the boundary information, and determine the first byte node in each word node; establish a first dependency connection relationship between multiple first byte nodes, start nodes and end nodes; establish a second dependency connection relationship between at least one byte node in each word node according to the intra-word order of at least one byte node in each word node; obtain a target syntactic graph according to the first dependency relationship and the second dependency connection relationship; wherein the first dependency relationship and / or the second dependency relationship is used to characterize the bidirectional connection relationship between each byte node.

[0215] In some embodiments, the second generation module 1440 is configured to: generate an initial word-level encoding corresponding to the text to be processed based on phoneme information, word boundaries in boundary information, and a word-level average pooling layer in a target encoding network; generate a target word-level encoding based on the initial word-level encoding, a target syntactic graph, and multiple gated graph convolutional layers in a target encoding network.

[0216] In some embodiments, the second generation module 1440 is configured to: determine at least one phoneme feature corresponding to each byte in the text to be processed based on phoneme information and word boundaries; perform an average pooling operation on at least one phoneme feature corresponding to each byte to determine the comprehensive phoneme feature corresponding to each byte; and determine the initial word-level encoding corresponding to the text to be processed based on the comprehensive phoneme feature corresponding to each byte.

[0217] In some embodiments, the second generation module 1440 is configured to: generate a first word-level code based on the initial word-level code, the target syntactic graph, and a first gated graph convolution layer among multiple gated graph convolution layers; generate a second word-level code based on the first word-level code, the target syntactic graph, and a second gated graph convolution layer among multiple gated graph convolution layers; determine the target word-level code based on the initial word-level code, the first word-level code, and the second word-level code.

[0218] In some embodiments, the third generation module 1450 is configured to: generate a target style vector based on the target word-level encoding, the target reference speech, and the style vector unit in the target speech generation network; wherein the target style vector is used to characterize the prosodic information corresponding to the target reference speech and the prosodic information corresponding to the text to be processed; generate a target synthetic speech based on the target style vector, the text to be processed, and the generation unit in the target speech generation network.

[0219] In some embodiments, the text-to-speech generation device 1400 based on syntactic graph construction also includes a training module. Among them, the acquisition module 1410 is configured to: acquire sample text data and sample reference speech data, and acquire sample synthesized speech data. The training module is configured to: generate predicted synthesized speech based on sample text data, sample reference speech data, and the speech generation model to be trained; use the predicted synthesized speech as the initial training output information of the speech generation model to be trained, and the sample synthesized speech data as the supervision information, iterate the speech generation model to be trained, and obtain the target speech generation model; wherein the speech generation model to be trained includes the syntactic graph construction network to be trained and the encoding network to be trained, and the target speech generation model includes the target syntactic graph construction network and the target encoding network.

[0220] Fig.15 A schematic diagram of an electronic device provided in an embodiment of the present application. In some embodiments, the electronic device includes one or more processors and a memory. The memory is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the text-to-speech generation method based on the syntactic graph in the above-mentioned embodiment.

[0221] like Fig.15As shown, the electronic device 1000 includes: a processor (processor) 1001 and a memory 1002. Exemplarily, the electronic device 1000 may also include: a communication interface (Communications Interface) 1003 and a communication bus 1004. The processor 1001, the memory 1002 and the communication interface 1003 communicate with each other through the communication bus 1004. The communication interface 1003 is used to communicate with other devices such as a client or other server network elements.

[0222] In some embodiments, the processor 1001 is used to execute the program 1005, and specifically, the relevant steps in the above-mentioned text-to-speech generation method embodiment based on syntax graph construction can be executed. Specifically, the program 1005 can include program code, which includes computer executable instructions.

[0223] Exemplarily, the processor 1001 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The electronic device 1000 may include one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0224] In some embodiments, the memory 1002 is used to store the program 1005. The memory 1002 may include a high-speed RAM memory, and may also include a non-volatile memory (NVM), such as at least one disk memory. The program 1005 can be specifically called by the processor 1001 to enable the electronic device 1000 to perform the operation of the text-to-speech generation method based on the syntactic graph construction.

[0225] An embodiment of the present application provides a computer-readable storage medium storing at least one executable instruction. When the executable instruction is executed on the electronic device 1000, the electronic device 1000 executes the text-to-speech generation method based on syntactic graph construction in the above embodiment.

[0226] The executable instructions may be specifically used to enable the electronic device 1000 to perform operations of a text-to-speech generation method based on syntactic graph construction.

[0227] For example, the computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0228] In some embodiments, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the text-to-speech generation method based on syntactic graph construction described in any of the above embodiments.

[0229] In some embodiments, the embodiments of the present application further provide a computer program, which, when executed by a processor, can implement the text-to-speech generation method based on syntactic graph construction described in any of the above embodiments.

[0230] The beneficial effects that can be achieved by the text-to-speech generation device, electronic device, computer-readable storage medium, computer program product and computer program based on syntactic graph provided in the embodiments of the present application can refer to the beneficial effects of the text-to-speech generation method based on syntactic graph provided above, and will not be repeated here.

[0231] It should be noted that, in the application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0232] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0233] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0234] For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by an instruction execution system, apparatus, or device or in conjunction with such instruction execution system, apparatus, or device.

[0235] More specific examples (a non-exhaustive list) of computer readable media include the following: an electrical connection having one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM).

[0236] In addition, the computer readable medium can even be a paper or other suitable medium on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, and then editing, interpreting or processing in other suitable ways as necessary, and then storing it in the computer memory. It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof.

[0237] In the above embodiments, multiple steps or methods may be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it may be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0238] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A text-to-speech generation method based on syntactic graph construction, characterized in that: The method comprises: Obtain the text to be processed and the target reference speech; According to the text to be processed, determining text information and phoneme information corresponding to the text to be processed; wherein the text information includes text content and boundary information; Building a network based on the text information and the target syntax graph in the target speech generation model to generate a target syntax graph corresponding to the text to be processed; wherein the text content includes a plurality of characters, and the target syntax graph is used to represent the syntactic relationship between each of the plurality of characters; Based on the phoneme information and the word boundaries in the boundary information, at least one phoneme feature corresponding to each byte in the text to be processed is determined, and an average pooling operation is performed on at least one phoneme feature corresponding to each byte through a word-level average pooling layer in a target encoding network to determine a comprehensive phoneme feature corresponding to each byte, and according to the comprehensive phoneme feature corresponding to each byte, an initial word-level encoding corresponding to the text to be processed is determined; wherein the target speech generation model includes the target encoding network; Generate a target word-level code corresponding to the text to be processed based on the initial word-level code, the target syntactic graph, and a plurality of gated graph convolutional layers in the target coding network; Based on the target word-level encoding, the target reference speech, and the target speech generation network in the target speech generation model, a target synthesized speech is generated; wherein the target synthesized speech includes the prosodic features of the text to be processed.

2. The method according to claim 1, characterized in that: The step of constructing a network based on the text information and the target syntactic graph in the target speech generation model to generate a target syntactic graph corresponding to the text to be processed includes: Based on the text content, the word boundaries in the boundary information, and the dependency parser in the target syntactic graph construction network, an initial dependency graph corresponding to the text information is generated; wherein the initial dependency graph includes a plurality of word nodes and a unidirectional dependency connection relationship between each word node in the plurality of word nodes; A syntax graph builder in a network is constructed based on the initial dependency graph and the target syntax graph to generate the target syntax graph corresponding to the initial dependency graph.

3. The method according to claim 2, characterized in that The constructing a syntactic graph builder in the network based on the initial dependency graph and the target syntactic graph to generate the target syntactic graph corresponding to the initial dependency graph includes: Based on the syntax graph builder and the word boundaries in the boundary information, each of the word nodes in the initial dependency graph is divided into at least one sub-node, and a first sub-node in each of the word nodes is determined; Establishing a first dependent connection relationship between the first byte node, the start node and the end node; Establishing a second dependency connection relationship between at least one byte node in each of the word nodes according to the intra-word order of at least one byte node in each of the word nodes; The target syntax graph is obtained according to the first dependency connection relationship and the second dependency connection relationship; wherein the first dependency connection relationship and / or the second dependency connection relationship are used to characterize the bidirectional connection relationship between each of the byte nodes.

4. The method according to claim 1, characterized in that The step of generating a target word-level code corresponding to the text to be processed based on the initial word-level code, the target syntactic graph, and a plurality of gated graph convolutional layers in the target coding network comprises: Generate a first word-level code based on the initial word-level code, the target syntactic graph, and a first gated graph convolutional layer among the plurality of gated graph convolutional layers; Generate a second word-level code based on the first word-level code, the target syntactic graph, and a second gated graph convolutional layer among the plurality of gated graph convolutional layers; The target word-level code is determined based on the initial word-level code, the first word-level code and the second word-level code.

5. The method according to any one of claims 1 to 4, characterized in that The generating target synthesized speech based on the target word-level encoding, the target reference speech, and the target speech generation network in the target speech generation model comprises: Generate a target style vector based on the target word-level encoding, the target reference speech, and a style vector unit in the target speech generation network; wherein the target style vector is used to characterize the prosodic information corresponding to the target reference speech and the prosodic information corresponding to the text to be processed; The target synthesized speech is generated based on the target style vector, the text to be processed, and a generation unit in the target speech generation network.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Obtaining sample text data and sample reference voice data; Generate predicted synthesized speech based on the sample text data, the sample reference speech data, and the speech generation model to be trained; wherein the speech generation model to be trained includes a syntactic graph construction network to be trained and a coding network to be trained; Obtain sample synthesized speech data; The predicted synthesized speech is used as the initial training output information of the speech generation model to be trained, and the sample synthesized speech data is used as the supervision information, and the speech generation model to be trained is iterated to obtain the target speech generation model; wherein the target speech generation model includes the target syntactic graph construction network and the target encoding network.

7. A text-to-speech generation device based on a syntactic graph, characterized in that: include: The acquisition module is configured to: acquire the text to be processed and the target reference speech; A determination module is configured to: determine text information and phoneme information corresponding to the text to be processed according to the text to be processed; wherein the text information includes text content and boundary information; A first generation module is configured to: construct a network based on the text information and a target syntax graph in a target speech generation model to generate a target syntax graph corresponding to the text to be processed; wherein the text content includes a plurality of characters, and the target syntax graph is used to represent the syntactic relationship between each of the plurality of characters; The second generation module is configured to: determine at least one phoneme feature corresponding to each byte in the text to be processed based on the phoneme information and the word boundary in the boundary information, perform an average pooling operation on at least one phoneme feature corresponding to each byte through the word-level average pooling layer in the target encoding network, determine the comprehensive phoneme feature corresponding to each byte, and determine the initial word-level encoding corresponding to the text to be processed according to the comprehensive phoneme feature corresponding to each byte; wherein the target speech generation model includes the target encoding network; based on the initial word-level encoding, the target syntactic graph, and a plurality of gated graph convolutional layers in the target encoding network, generate a target word-level encoding corresponding to the text to be processed; The third generation module is configured to generate a target synthesized speech based on the target word-level encoding, the target reference speech, and the target speech generation network in the target speech generation model; wherein the target synthesized speech includes the prosodic features of the text to be processed.

8. An electronic device, characterized in that: include: one or more processors; and A memory configured to: store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the text-to-speech generation method based on syntactic graph construction according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the text-to-speech generation method based on syntactic graph construction according to any one of claims 1-6 is implemented.

Images