Text processing method and apparatus, and electronic device

Abstract

The application relates to the computer technical field and discloses a text processing method, a text processing device and electronic equipment. The method comprises the following steps: obtaining a text to be processed and a target reference voice; determining text information and phoneme information corresponding to the text to be processed according to the text to be processed; constructing a network based on a target syntax graph in a text information and target voice generation model to generate a target syntax graph corresponding to the text to be processed; generating target word-level coding corresponding to the text to be processed based on the phoneme information, boundary information, the target syntax graph and a target coding network in the target voice generation model; and generating a target synthesized voice based on the target word-level coding, the target reference voice and a target voice generation network in the target voice generation model. The text processing method provided in the application has the prosodic features of the text to be processed, and the authenticity and richness of the synthesized voice are improved.

Description

[0001] This application is a divisional application, the original application number is 2024110597135, the original application date is August 5, 2024, and the entire contents of the original application are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of computers, and in particular to a text processing method and device and electronic equipment. BACKGROUND

[0003] Generally, in the text-to-speech synthesis process, more emphasis is placed on the replication of the speaker's voice, that is, to ensure that the synthesized speech is as close as possible to the speaker's voice characteristics (such as pitch, timbre, etc.). However, for synthesized speech, its presentation effect is not only determined by the speaker's voice characteristics, but also depends on the text information corresponding to the speech. For example, when the text information is a horror novel, the synthesized speech has a fast and deep rhythm characteristic in the presentation effect. However, the related art does not consider the rhythm characteristics corresponding to the text information in the text-to-speech synthesis process, so that the rhythm of the synthesized speech cannot completely match the text information, thereby reducing the richness and authenticity of the synthesized speech. SUMMARY

[0004] To solve the above problems, the embodiments of the present application provide a text processing method, device and electronic equipment, which can consider the rhythm characteristics corresponding to the text information in the text-to-speech synthesis process to improve the authenticity and richness of the synthesized speech. Specifically, the embodiments of the present application disclose the following technical solutions:

[0005] The first aspect of the embodiments of the present application provides a text processing method, which comprises: first, obtaining a to-be-processed text and a target reference speech; second, determining text information and phoneme information corresponding to the to-be-processed text according to the to-be-processed text; wherein the text information includes text content and boundary information; then, generating a target syntax graph corresponding to the to-be-processed text based on the text information and a target syntax graph construction network in a target speech generation model; wherein the text content includes a plurality of characters, and the target syntax graph is used to represent the syntactic relationship between the characters; third, generating a target word-level code corresponding to the to-be-processed text based on the phoneme information, the boundary information, the target syntax graph, and a target encoding network in the target speech generation model; and finally, generating a target synthesized speech based on the target word-level code, the target reference speech, and a target speech generation network in the target speech generation model; wherein the target synthesized speech includes the rhythm characteristics of the to-be-processed text.

[0006] In some embodiments, the target syntax graph corresponding to the text to be processed is generated based on the text information and the target syntax graph construction network in the target speech generation model, including: generating an initial dependency graph corresponding to the text information based on the text content, the word boundary in the boundary information, and a dependency parser in the syntax graph construction network; wherein the initial dependency graph includes a plurality of word nodes and a unidirectional connection relationship between each word node in the plurality of word nodes; and generating a target syntax graph corresponding to the initial dependency graph based on the initial dependency graph and a syntax graph constructor in the syntax graph construction network.

[0007] In some embodiments, the target syntax graph corresponding to the initial dependency graph is generated based on the initial dependency graph and the syntax graph constructor in the syntax graph construction network, including: dividing each word node in the initial dependency graph into at least one character node based on the syntax graph constructor and the character boundary in the boundary information, and determining a first character node in each word node; establishing a first dependency connection relationship between the plurality of first character nodes, a start node and an end node; establishing a second dependency connection relationship between at least one character node in each word node according to the intra-word order of the at least one character node; and obtaining the target syntax graph according to the first dependency connection relationship and the second dependency connection relationship; wherein the first dependency connection relationship and / or the second dependency connection relationship are used to represent a bidirectional connection relationship between the character nodes.

[0008] In some embodiments, the target character-level encoding corresponding to the text to be processed is generated based on the phoneme information, the boundary information, the target syntax graph, and the target encoding network in the target speech generation model, including: generating an initial character-level encoding corresponding to the text to be processed based on the phoneme information, the character boundary in the boundary information, and a character-level average pooling layer in the target encoding network; and generating the target character-level encoding based on the initial character-level encoding, the target syntax graph, and a plurality of gated graph convolution layers in the target encoding network.

[0009] In some embodiments, the initial character-level encoding corresponding to the text to be processed is generated based on the phoneme information, the character boundary in the boundary information, and the character-level average pooling layer in the target encoding network, including: determining at least one phoneme feature corresponding to each character node in the text to be processed based on the phoneme information and the character boundary; performing an average pooling operation on the at least one phoneme feature corresponding to each character node to determine a comprehensive phoneme feature corresponding to each character node; and determining the initial character-level encoding corresponding to the text to be processed according to the comprehensive phoneme feature corresponding to each character node.

[0010] In some embodiments, the target word-level encoding is generated based on the initial word-level encoding, the target syntax graph, and a plurality of gated graph convolution layers in the target encoding network, including: generating a first word-level encoding based on the initial word-level encoding, the target syntax graph, and a first gated graph convolution layer in the plurality of gated graph convolution layers; generating a second word-level encoding based on the first word-level encoding, the target syntax graph, and a second gated graph convolution layer in the plurality of gated graph convolution layers; and determining the target word-level encoding based on the initial word-level encoding, the first word-level encoding, and the second word-level encoding.

[0011] In some embodiments, the target synthesized speech is generated based on the target word-level encoding, the target reference speech, and a target speech generation network in the target speech generation model, including: generating a target style vector based on the target word-level encoding, the target reference speech, and a style vector unit in the target speech generation network; wherein the target style vector is used to represent prosodic information corresponding to the target reference speech and prosodic information corresponding to the to-be-processed text; and generating the target synthesized speech based on the target style vector, the to-be-processed text, and a generation unit in the target speech generation network.

[0012] In some embodiments, the method further includes: obtaining sample text data and sample reference speech data; generating predicted synthesized speech based on the sample text data, the sample reference speech data, and the to-be-trained speech generation model; wherein the to-be-trained speech generation model includes a to-be-trained syntax graph construction network and a to-be-trained encoding network; obtaining sample synthesized speech data; and iteratively training the to-be-trained speech generation model with the predicted synthesized speech as initial training output information of the to-be-trained speech generation model and the sample synthesized speech data as supervision information to obtain a target speech generation model; wherein the target speech generation model includes a target syntax graph construction network and a target encoding network.

[0013] The second aspect of the embodiment of the present application provides a text processing device, comprising an acquisition module, a determination module, a first generation module, a second generation module and a third generation module. The acquisition module is configured to acquire a text to be processed and a target reference voice. The determination module is configured to determine text information and phoneme information corresponding to the text to be processed according to the text to be processed; wherein the text information comprises text content and boundary information. The first generation module is configured to generate a target syntax graph corresponding to the text to be processed based on the text information and a target syntax graph construction network in a target speech generation model; wherein the text content comprises a plurality of characters, and the target syntax graph is used to represent the syntactic relationship between the characters. The second generation module is configured to generate target word-level coding corresponding to the text to be processed based on the phoneme information, the boundary information, the target syntax graph, and a target coding network in the target speech generation model. The third generation module is configured to generate a target synthesized voice based on the target word-level coding, the target reference voice, and a target speech generation network in the target speech generation model; wherein the target synthesized voice comprises prosodic features of the text to be processed.

[0014] The third aspect of the embodiment of the present application provides an electronic device, comprising a processor and a memory, the memory is used to store computer executable instructions; the processor is used to read the instructions from the memory and execute the instructions to realize the text processing method of the first aspect.

[0015] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, the storage medium stores computer program instructions, when the computer reads the instructions, the text processing method of the first aspect is executed.

[0016] The fifth aspect of the embodiment of the present application provides a computer program product, the computer program product comprises a computer program stored on a non-transitory computer readable storage medium, the computer program comprises program instructions, when the program instructions are executed by a computer, the computer executes the text processing method of the first aspect.

[0017] The sixth aspect of the embodiment of the present application provides a computer program, when the computer program is executed by a processor, the text processing method of the first aspect can be realized.

[0018] The text processing method provided in the embodiments of the present application generates a target synthesized speech corresponding to the target reference speech and the to-be-processed text through a pre-trained target speech generation model. First, a target syntax graph corresponding to the to-be-processed text is generated based on text information corresponding to the to-be-processed text and a target syntax graph construction network in the target speech generation model. Then, target word-level coding corresponding to the to-be-processed text is generated based on phoneme information and boundary information corresponding to the to-be-processed text, the target syntax graph, and a target coding network in the target speech generation model. Finally, the target synthesized speech is generated based on the target word-level coding, the target reference speech, and a target speech generation network in the target speech generation model. In the text processing method provided in the embodiments of the present application, the target syntax graph generated by the target syntax graph construction network can represent the syntax relationship between each character in a plurality of characters, is more consistent with the syntax information of Chinese syntax, and can provide prosodic features that are more consistent with the text information. Therefore, the target word-level coding generated based on the target syntax graph can have the prosodic features corresponding to the to-be-processed text, so that the target synthesized speech generated based on the target word-level coding and the target reference speech also has the prosodic features of the to-be-processed text. Therefore, the text processing method provided in the embodiments of the present application can extract the prosodic features of the text information in the text-to-speech synthesis process, so as to improve the authenticity and richness of the synthesized speech. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 A schematic diagram of a text processing method provided in the embodiments of the present application;

[0021] Figure 2 A schematic diagram of a target speech generation model provided in the embodiments of the present application;

[0022] Figure 3 A schematic diagram of another text processing method provided in the embodiments of the present application;

[0023] Figure 4 A schematic diagram of an initial dependency graph and a target syntax graph provided in the embodiments of the present application;

[0024] Figure 5 A schematic diagram of another text processing method provided in the embodiments of the present application;

[0025] Figure 6Another text processing method provided by an embodiment of the present application is shown in the flowchart of FIG. 6.

[0026] Figure 7 Another text processing method provided by an embodiment of the present application is shown in the flowchart of FIG. 6.

[0027] Figure 8 Another target speech generation model provided by an embodiment of the present application is shown in the flowchart of FIG. 7.

[0028] Figure 9 Another text processing method provided by an embodiment of the present application is shown in the flowchart of FIG. 6.

[0029] Figure 10 Another text processing method provided by an embodiment of the present application is shown in the flowchart of FIG. 6.

[0030] Figure 11 An X-vector feature vector extractor provided by an embodiment of the present application is shown in the flowchart of FIG. 8.

[0031] Figure 12 Another text processing method provided by an embodiment of the present application is shown in the flowchart of FIG. 6.

[0032] Figure 13 A target speech generation network provided by an embodiment of the present application is shown in the flowchart of FIG. 9.

[0033] Figure 14 A text processing apparatus provided by an embodiment of the present application is shown in the flowchart of FIG. 10.

[0034] Figure 15 An electronic device provided by an embodiment of the present application is shown in the flowchart of FIG. 11. DETAILED DESCRIPTION

[0035] In order to make the technical solutions in the embodiments of the present application better understood by those skilled in the art, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more apparent and easy to understand, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the drawings.

[0036] Text to speech (TTS) is to convert written text data into audible speech data for output. Generally, in the process of realizing text to speech, more attention is often paid to the replication of the voice of a reference speaker, that is, to make the synthesized speech as close as possible to the voice characteristics of the reference speaker, such as the pitch, timbre and rhythm of the reference speaker.

[0037] However, the presentation effect of a synthesized voice depends not only on the voice characteristics of a reference speaker, but also on the text information corresponding to the voice. For example, when the text information is a thriller novel, the synthesized voice is more expected to have fast and deep prosody characteristics in terms of presentation effect; and when the text information is a documentary literature, the synthesized voice is more expected to have slow and smooth prosody characteristics in terms of presentation effect. However, the synthesized voice in the related art cannot generate a voice that matches the text information, so that the synthesized voice is not real and rich enough, and cannot provide a better experience for users.

[0038] To solve the above problems, an embodiment of the present application provides a text processing method, which can extract prosody characteristics of text information in the process of synthesizing a voice, and generate a synthesized voice based on the prosody characteristics of the text information, so as to ensure that the synthesized voice has the prosody characteristics of the text information, improve the authenticity and richness of the synthesized voice, and provide a better user experience.

[0039] The following will be described in conjunction with the accompanying Figures 1 to 13 The text processing method provided by the embodiment of the present application is described.

[0040] Figure 1 A schematic diagram of a text processing method provided by the embodiment of the present application is shown in FIG. 1. As shown in the figure, the text-to-speech method includes the following steps 110 to 150. Figure 1

[0041] Step 110, obtaining a to-be-processed text and a target reference voice.

[0042] For example, in the process of generating a text-to-speech, the voice information of the target reference voice is needed to express the information of the to-be-processed text, so as to obtain the finally generated target synthesized voice. Therefore, before generating the target synthesized voice, the to-be-processed text and the target reference voice need to be obtained first.

[0043] In some examples, the to-be-processed text is used to provide text information to the target synthesized voice. The to-be-processed text can be a piece of text, such as a thriller novel segment or a documentary literature segment, etc.; or the to-be-processed text can also be a sentence or a group of words, such as the to-be-processed text can be "monkeys like to eat bananas". The to-be-processed text is not limited by the embodiment of the present application.

[0044] In some examples, the target reference voice is used to provide voice information to the target synthesized voice, and the voice information includes prosody information of the voice of the target speaker, such as tone and pitch, etc. The target reference voice can be the voice of the target speaker, such as a piece of voice of the target speaker in a preset time length. The target speaker can be a real person or a virtual person, and the embodiment of the present application does not limit it.

[0045] ​Step 120: Determine the text information and phoneme information corresponding to the text to be processed based on the text to be processed.

[0046] For example, the text to be processed can be parsed to obtain text information and phoneme information corresponding to the text, with the phoneme information corresponding to the text information. The text information is the written form of the language, reflecting its direct meaning, grammatical structure, and contextual relationships. The phoneme information is the actual phoneme sequence of pronunciation formed after further transformation of the text information, reflecting the pronunciation characteristics and acoustic performance of the language.

[0047] In some examples, after obtaining the text information from the text to be processed, each word in the text information can be converted into its corresponding phoneme sequence based on the text information and a phoneme dictionary, thereby obtaining the phoneme information corresponding to the text information. The phoneme dictionary can be a pre-defined dictionary library, such as a resource library built according to the target language; or it can be an existing phoneme dictionary library in related technologies, which is not limited in this embodiment.

[0048] For example, text information may include text content and boundary information. Boundary information includes word boundaries and character boundaries.

[0049] In some examples, the text content is the plain text corresponding to the text information. The text content is used to express the meaning of the text and can include multiple characters, such as words (words or letters), numbers, and punctuation marks. Boundary information refers to the separating positions between words or characters within the text content. For example, boundary information can also be the separation between two adjacent words or two adjacent characters. The separation between words can be called word boundaries, and the separation between characters can be called character boundaries.

[0050] For example, when the text to be processed is "monkeys like to eat bananas", the word boundaries are the separations between "monkey" and "like", "like" and "eat", and "eat" and "banana"; the character boundaries are the separations between "monkey" and "child", "child" and "like", "like" and "happy", "happy" and "eat", "eat" and "fragrant", and "fragrant" and "banana".

[0051] This application embodiment converts the text to be processed into text information and phoneme information, which can characterize the essential features of the text to be processed from two levels. The text information focuses on characterizing the semantic structure and lexical meaning of the text to be processed, while the phoneme information focuses on the pronunciation details and phonetic characteristics of the text to be processed, providing the necessary input information for the subsequent text-to-speech generation process.

[0052] Figure 2Schematic diagram of a target speech generation model provided by an embodiment of the present application. As Figure 2 shown, the target speech generation model 1 includes a syntactic graph construction sub-model 100 and a generation sub-model 200. It should be noted that the text processing method provided by the embodiment of the present application can be implemented based on the target speech generation model 1.

[0053] Exemplarily, the syntactic graph construction sub-model 100 includes a target syntactic graph construction network 10 and a target encoding network 20. Among them, the target syntactic graph construction network 10 may include a dependency parser 11 and a syntactic graph builder 12; the target encoding network 20 may include a word-level average pooling layer 21 and multiple gated graph convolutional layers 22.

[0054] Exemplarily, the generation sub-model 200 includes a target speech generation network 30. It should be noted that the generation sub-model 200 can be the generator in the speech generation model in the related art, or the improved generator in the embodiment of the present application. The embodiment of the present application does not limit this.

[0055] Next, the text processing method provided by the embodiment of the present application will be further described in conjunction with Figure 2 the target speech generation model 1 in

[0056] Step 130, based on the text information and the target syntactic graph construction network in the target speech generation model, generate a target syntactic graph corresponding to the text to be processed.

[0057] Exemplarily, the text content includes multiple characters, and the target syntactic graph is used to represent the syntactic relationship between each character in the multiple characters. Among them, when the text content is a Chinese sentence, the character is used to represent the Chinese character in the Chinese sentence.

[0058] In some examples, the syntactic graph is a graphical representation of the text information, and the syntactic graph can represent the syntactic relationship between each word in the text information. Through the syntactic relationship between each word, the prosodic features of each word can be obtained, including pronunciation features, pitch features, etc.

[0059] The target syntactic graph generated by the target syntactic construction network in the embodiment of the present application includes the syntactic relationship between each character. Compared with the syntactic graph in the related art that can only represent the syntactic relationship between each word, it is more in line with the syntactic information of the Chinese syntax and can provide prosodic features more in line with the text information.

[0060] For example, for a Chinese sentence, the prosody corresponding to the same character in different sentences may be different, that is, the same character may have different pronunciations in different sentences. For example, the pronunciation of the character "xian" in "fresh" and "rarely known" is different, that is, the prosody is different.

[0061] Therefore, when determining the prosody of a Chinese character, it is necessary to consider the sentence in which it is located, that is, the dependency connections between the character and other characters in the sentence. The target syntax graph provided in this application can represent the dependency connections between characters, thus enabling the acquisition of more accurate and richer prosodic features of the text to be processed.

[0062] Figure 3 This is a schematic diagram illustrating another text processing method provided in an embodiment of this application. For example... Figure 3 As shown, step 130 above includes steps 1310 to 1320 as shown below.

[0063] Step 1310: Based on the text content, word boundaries in the boundary information, and the syntactic graph, construct the dependency parser in the network to generate the initial dependency graph corresponding to the text information.

[0064] Continue to refer to Figure 2 The text content and boundary information, such as word boundaries in the text content and boundary information, are input into the dependency parser 11 in the target syntax graph construction network 10. After processing by the dependency parser 11, the initial dependency graph corresponding to the text information is generated.

[0065] In some examples, the initial dependency graph is a directed graph based on a unidirectional structure. The initial dependency graph includes multiple word nodes and unidirectional connections between the word nodes.

[0066] Figure 4 This is a schematic diagram of an initial dependency graph and a target syntax graph provided for embodiments of this application. Wherein, Figure 4 (a) in the diagram is a schematic of an initial dependency graph. For example, when the text information is "Monkeys like to eat bananas", the initial dependency graph generated after passing through dependency resolver 11 is as follows: Figure 4 As shown in (a) in the figure.

[0067] like Figure 4 As shown in (a), the initial dependency graph includes four word nodes: "monkey", "like", "eat", and "banana". "Like" is the parent node, and "monkey", "eat", and "banana" are three byte nodes. One-way connections are established between adjacent word nodes. Since "like" is the parent node, the one-way connection between "monkey" and "like" is "like" pointing to "monkey", the one-way connection between "like" and "eat" is "like" pointing to "eat", and the one-way connection between "eat" and "banana" is "eat" pointing to "banana".

[0068] In some examples, the initial dependency graph can represent the prosodic features of the text information. However, the prosodic information determined by the initial dependency graph is not accurate enough. On the one hand, since the connection between each word node in the initial dependency graph is a one-way connection, that is, each word node in the sentence cannot obtain any information from other word nodes, that is, from other non-adjacent word nodes in the graph aggregation process, the prosodic features of the text information obtained are not accurate enough. On the other hand, since each node in the initial dependency graph is a word node, the pronunciation rules of each character in a Chinese word depend on its position in a sentence or word, and therefore, the initial dependency graph cannot represent the pronunciation features of each character in the text information, which may affect the effect of the subsequent synthesized speech. Therefore, when a relatively complex sentence is processed, the prosody of the sentence, such as pronunciation, pause, and stress, is also relatively complex, and the initial dependency graph cannot better represent the prosodic features of the sentence.

[0069] In order to more accurately obtain the prosodic features of the text information, the syntax graph constructor 12 is introduced into the target syntax graph construction network 10 in the embodiments of the present application, and the initial dependency graph is further processed by the syntax graph constructor 12, so as to obtain more rich prosodic features in the text information.

[0070] Step 1320, based on the initial dependency graph and the syntax graph constructor in the syntax graph construction network, a target syntax graph corresponding to the initial dependency graph is generated.

[0071] Exemplarily, the syntax graph constructor 12 is used to convert the initial dependency graph into a target syntax graph. The target syntax graph is a directed graph based on a bidirectional structure, and the target syntax graph includes a plurality of character nodes and a bidirectional connection relationship between each character node in the plurality of character nodes.

[0072] Continuing to refer to Figure 2 After obtaining the initial dependency graph output by the dependency parser 11, the initial dependency graph can be input into the syntax graph constructor 12, so as to obtain the target syntax graph.

[0073] In some examples, the target syntax graph is obtained by adding bidirectional edges between each word node on the basis of the initial dependency graph, so that the information flow in the target syntax graph becomes a bidirectional information flow. That is, for any pair of parent nodes and character nodes, the target syntax graph includes both the forward edge from the parent node to the character node and the reverse edge from the character node to the parent node.

[0074] Figure 5 A schematic diagram of another text processing method provided by the embodiments of the present application is shown in FIG. 13B. Figure 5 As shown in FIG. 13B, the above step 1320 includes steps 1321 to 1324 as shown below.

[0075] Step 1321: Based on the word boundaries in the syntactic graph builder and boundary information, divide each word node in the initial dependency graph into at least one byte point, and determine the first byte point in each word node.

[0076] For example, the syntactic graph builder 12 can further split each word node in the initial dependency graph into at least one byte point based on word boundaries and establish dependencies between the byte points.

[0077] For example, in a Chinese sentence (i.e., text content), the same character may have different pronunciations or intonations in different words or sentences, thus determining that the phonemes of that character may differ. To enable the target syntactic graph to more accurately represent the pronunciation rules of Chinese, the syntactic graph builder 12 provided in this application can expand each word in the initial dependency graph as a node to each character as a node. That is, after splitting each word node into byte points, the dependency relationships between each byte point are established, thereby obtaining more accurate pronunciation and prosodic information.

[0078] Continue to refer to Figure 4 , Figure 4 (b) in the diagram is a schematic representation of a target syntax graph. For example, Figure 4 The initial dependency graph in (a) can be generated as follows after processing by the syntactic graph builder 12: Figure 4 The target syntax graph is shown in (b) of the diagram.

[0079] like Figure 4 As shown in (b) in the target syntax graph, the word node “monkey” is split into two byte nodes “monkey” and “child”, the word node “like” is split into two byte nodes “like” and “happy”, the word node “like banana” is split into two byte nodes “fragrant” and “banana”, and the byte node corresponding to the word node “eat” is also “eat”.

[0080] In some examples, after dividing each word node into at least one byte point, the first byte point in each word node can be determined according to the word order of each byte point within its respective word node. For example, in the word node "monkey", the word order is "monkey" and "child", therefore, the first byte point in this word node is "monkey". Similarly, the first byte points corresponding to each word node can be determined as "monkey", "happy", "eat", and "fragrant".

[0081] Step 1322: Establish the first dependency connection relationship between multiple first byte points, start nodes and end nodes.

[0082] Exemplarily, the target syntax graph further comprises a start node (which can also be referred to as BOS) and an end node (which can also be referred to as EOS). The start node is connected to the first character node in the first word node in the text content, and the end node is connected to the first character node in the last word node in the text content.

[0083] In some examples, a first dependency connection relationship can be established between the first character nodes corresponding to the plurality of word nodes, the start node and the end node. The first dependency connection relationship can be a bidirectional connection relationship. For example, the first dependency connection relationship between the first character nodes in adjacent nodes can be established according to the adjacency relationship of the plurality of word nodes in the initial dependency graph. It should be noted that the target syntax graph is established on the basis of the initial dependency graph, and therefore the order of the word nodes is consistent with the order of the word nodes in the initial dependency graph.

[0084] As shown in (b) of FIG. 13, in the target syntax graph, a solid arrow is used to represent the first dependency connection relationship, and the “BOS” and “monkey”, the “monkey” and “happy”, the “happy” and “eat”, the “eat” and “fragrant”, and the “fragrant” and “EOS” are all bidirectional connections (i.e., the first dependency connection relationship). Figure 4

[0085] Step 1323: According to the intra-word order of at least one character node in each word node, a second dependency connection relationship between at least one character node in each word node is established.

[0086] In some examples, after the plurality of word nodes are divided into at least one character node, the second dependency connection relationship between at least one character node in each word node can be established according to the intra-word order of the character nodes in the word node. The second dependency connection relationship can be a bidirectional connection relationship.

[0087] As shown in (b) of FIG. 13, a dashed arrow is used to represent the second dependency connection relationship, and the character node “monkey” and the character node “child” in the word node “monkey child” are bidirectional connections (i.e., the second dependency connection relationship). Similarly, the character node “happy” and the character node “happy” are bidirectional connections (i.e., the second dependency connection relationship), and the character node “fragrant” and the character node “banana” are bidirectional connections (i.e., the second dependency connection relationship). Figure 4

[0088] Step 1324: According to the first dependency connection relationship and the second dependency connection relationship, a target syntax graph is obtained.

[0089] The first dependency connection relationship and / or the second dependency connection relationship are used to represent the bidirectional connection relationship between the character nodes.

[0090] In some examples, after the first dependency connection relationship and the second dependency connection relationship between the character nodes are established, a target syntax graph as shown in (b) of FIG. 13 can be obtained. Figure 4 ​​the target syntax graph shown in (b) in FIG. 13.

[0091] The initial dependency graph can be converted into the target syntax graph by the syntax graph constructor 12 in the embodiment of the present application. Since the target syntax graph is bidirectional connection between each word node, more accurate pronunciation and prosody features in the text information can be obtained, and better input is provided for subsequent generation of target word-level coding. That is, the improvement of the syntax graph extraction process in the embodiment of the present application obtains syntax information more consistent with Chinese syntax, and further makes the prosody feature more consistent with the prosody feature of the input text in the subsequent prosody feature.

[0092] In step 140, the target word-level coding corresponding to the text to be processed is generated based on the phoneme information, the boundary information, the target syntax graph, and a target coding network in the target speech generation model.

[0093] Referring to Figure 2 After obtaining the target syntax graph, the phoneme information, the boundary information, and the target syntax graph corresponding to the text to be processed can be input into the target coding network 20, and the target coding network 20 outputs the target word-level coding.

[0094] Figure 6 Another text processing method provided by the embodiment of the present application is shown in the flowchart of FIG. 14. As shown in Figure 6 The step 140 includes steps 1410 to 1420 as shown below.

[0095] In step 1410, the initial word-level coding corresponding to the text to be processed is generated based on the phoneme information, the word boundary in the boundary information, and a word-level average pooling layer in the target coding network.

[0096] In some examples, the boundary information includes the word boundary. As Figure 2 As shown, the phoneme information and the word boundary can be input into the word-level average pooling layer 21 in the target coding network 20 to generate the initial word-level coding.

[0097] In some embodiments, the step 1410 includes: determining at least one phoneme feature corresponding to each word node in the text to be processed based on the phoneme information and the word boundary; performing an average pooling operation on the at least one phoneme feature corresponding to each word node to determine a comprehensive phoneme feature corresponding to each word node; and determining the initial word-level coding corresponding to the text to be processed according to the comprehensive phoneme feature corresponding to each word node.

[0098] In some examples, after obtaining the phoneme information corresponding to the text to be processed, further processing of this phoneme information is required. The phoneme information corresponding to the text to be processed can be a phoneme encoding sequence. Since each byte (such as a Chinese character in a sentence) can typically be composed of multiple phonemes, the encoding sequence corresponding to each byte is determined within the phoneme encoding sequence corresponding to the text to be processed.

[0099] For example, word boundaries can be used to determine which adjacent phonemes in the phoneme encoding sequence corresponding to the text to be processed correspond to a single byte point. After determining the phonemes corresponding to each byte point, average pooling can be performed on at least one phoneme corresponding to the same byte point to obtain the comprehensive phoneme features of each byte point. After obtaining the comprehensive phoneme features of each byte point, each byte point is encoded to determine the initial character-level encoding corresponding to the text to be processed. That is, in this embodiment, encoding is performed on a character-by-character basis, rather than on a word-by-word basis.

[0100] The embodiments of this application employ character-level average pooling to generate character-level codes at the character unit level. Compared with word-level coding in related technologies, this approach can obtain more accurate prosodic information from the text to be processed.

[0101] Step 1420: Generate the target character-level code based on the initial character-level code, the target syntax graph, and multiple gated graph convolutional layers in the target encoding network.

[0102] In some examples, such as Figure 2 As shown, after obtaining the initial character-level encoding from the output of the character-level average pooling layer 21, the initial character-level encoding and the target syntactic graph can be input into multiple gated graph convolutional layers 22 in the target encoding network 20, and the target character-level encoding can be determined based on the output of each gated graph convolutional layer 22. The target character-level encoding can represent the prosodic features corresponding to the text to be processed.

[0103] For example, the target encoding network 20 may include multiple gated graph convolutional layers 22. For instance, the target encoding network 20 may include two gated graph convolutional layers, such as a first gated graph convolutional layer and a second gated graph convolutional layer. Alternatively, the target encoding network 20 may include more gated graph convolutional layers; the number of gated graph convolutional layers 22 is not limited in this embodiment. This embodiment uses the example of the target encoding network 20 including a first gated graph convolutional layer and a second gated graph convolutional layer for illustrative purposes.

[0104] In some examples, the target syntax graph input to each gated graph convolution layer 22 can be a target syntax graph processed from the output of the syntax graph constructor 12. That is, the output of the syntax graph constructor 12 can be converted into a data type that can be processed by the gated graph convolution layer 22 and input to each gated graph convolution layer 22.

[0105] For example, the target syntax graph input to the gated graph convolution layer 22 can include a node embedding feature vector and a graph structure feature vector corresponding to the target syntax graph. The node embedding feature vector is used to represent the features of each word node in the target syntax graph, and the graph structure feature vector is used to represent the dependency connection relationship between each word node in the target syntax graph.

[0106] In some examples, the gated graph convolution layer 22 is used to extract the syntax information of each word node and the dependency connection relationship between each word node in the target syntax graph. For example, each gated graph convolution layer 22 can include 5 iterations to extract the dependency connection relationship between each word node in the target syntax graph.

[0107] The embodiment of the present application can further extract the phoneme features of each word node in the text to be processed according to the initial word-level encoding and the target syntax graph through the gated graph convolution layer 22, so as to obtain more rich and accurate prosody information of each word node in the text to be processed.

[0108] Figure 7 Another example of a text processing method provided by the embodiment of the present application is shown in the flowchart of FIG. 14. As shown in FIG. 14, the above step 1420 includes steps 1421 to 1423 as shown below. Figure 7

[0109] Step 1421: generating a first word-level encoding based on the initial word-level encoding, the target syntax graph, and a first gated graph convolution layer in the plurality of gated graph convolution layers.

[0110] Step 1422: generating a second word-level encoding based on the first word-level encoding, the target syntax graph, and a second gated graph convolution layer in the plurality of gated graph convolution layers.

[0111] Step 1423: determining a target word-level encoding based on the initial word-level encoding, the first word-level encoding, and the second word-level encoding.

[0112] ​In some examples, first, the initial word-level encoding output by the word-level average pooling layer 21 and the target syntax graph output by the target syntax graph construction network 10 are input into a first gated convolutional layer, and after processing by the first gated convolutional layer, a first word-level encoding is output; then, the first word-level encoding output by the first gated convolutional layer and the target syntax graph output by the target syntax graph construction network 10 are input into a second gated graph convolutional layer, and after processing by the second gated convolutional layer, a second word-level encoding is output. Finally, the initial word-level encoding output by the word-level average pooling layer 21, the first word-level encoding output by the first gated convolutional layer, and the second word-level encoding output by the second gated convolutional layer are summed to obtain the target word-level encoding.

[0113] It should be noted that more numbers of gated graph convolutional layers 22 can also be set in the target encoding network 20 for superposition, and the outputs of each gated graph convolutional layer are aggregated to output the target word-level encoding. The processing mode of the more numbers of gated graph convolutional layers is similar to the above steps 1421 to 1423, which will not be described here.

[0114] The text processing method provided by the embodiment of the present application obtains the target syntax graph corresponding to the text to be processed through the target syntax graph construction network in the target speech generation model, and then obtains the target word-level encoding corresponding to the text to be processed according to the target syntax graph and the target encoding network 20. Since the bidirectional dependency connection relationship between the word nodes in the text information is represented in the target syntax graph, which is more in line with the syntax information of Chinese syntax, it can provide more prosodic features in line with the text information, and the target word-level encoding obtained based on the target syntax graph can better reflect the prosodic features of the word nodes in the text to be processed. Therefore, when the target word-level encoding is used as the input of the subsequent generation sub-model 200 for speech synthesis, the generated target synthesized speech can have the prosodic features of the text to be processed, improving the authenticity and richness of the target synthesized speech.

[0115] Step 150, generating a target synthesized speech based on the target word-level encoding, the target reference speech, and the target speech generation network in the target speech generation model.

[0116] Exemplarily, the target synthesized speech includes the prosodic features of the text to be processed.

[0117] In some examples, as shown in Figure 2 After the target word-level encoding is obtained by the syntax graph construction sub-model 100, the target word-level encoding and the target reference speech can be input into the target speech generation network 30 in the generation sub-model 200 to generate a target synthesized speech corresponding to the text to be processed and the target reference speech.

[0118] It should be noted that the target speech generation network 30 can be a generator used for speech synthesis in related technologies, or it can be a generator improved in the embodiments of this application. In the embodiments of this application, the input to the generator is the target word-level code. Therefore, whether it is a generator in related technologies or a generator improved in the embodiments of this application, the final synthesized speech can have the prosodic features of the text to be processed, improving the realism and richness of the final synthesized speech. For example, the embodiments of this application use the improved generator (i.e., the target speech generation network 30) as an example to illustrate the process of generating target synthesized speech.

[0119] In some embodiments, the target speech generation network 30 includes a style vector unit and a generation unit.

[0120] The following is combined with Figures 8 to 13 The target speech generation network 30 provided in the embodiments of this application will be further described.

[0121] Figure 8 This is a schematic diagram of another target speech generation model provided in an embodiment of this application. For example... Figure 8 As shown, the target speech generation network 30 includes a style vector unit 31 and a generation unit 32. The style vector unit 31 is used to further extract prosodic information corresponding to the text to be processed and prosodic information corresponding to the target reference speech; the generation unit 32 is used to generate the final target synthesized speech based on the prosodic information extracted by the style vector unit (such as the first prosodic information and the second prosodic information).

[0122] Figure 9 This is a schematic diagram illustrating another text processing method provided in an embodiment of this application. For example... Figure 9 As shown, step 150 above includes steps 1510 to 1520 as shown below.

[0123] Step 1510: Generate a target style vector based on the target word-level encoding, the target reference speech, and the style vector unit in the target speech generation network.

[0124] For example, the target style vector can be used to represent the first prosodic information corresponding to the text to be processed and the second prosodic information corresponding to the target reference speech.

[0125] In some embodiments, the style vector unit 31 includes a style diffusion sampler and a feature extractor.

[0126] Figure 10 This is a schematic diagram illustrating another text processing method provided in an embodiment of this application. For example... Figure 10 As shown, step 1510 above includes steps 1511 to 1513 as shown below.

[0127] At step 1511, the first prosody vector corresponding to the to-be-processed text is generated based on the target word-level coding and the style diffusion sampler in the style vector unit.

[0128] The first prosody vector is used to represent the first prosody information. That is, the style diffusion sampler in the style vector unit 31 can extract the prosody features (i.e., the first prosody features) corresponding to the to-be-processed text based on the target word-level coding.

[0129] It should be noted that the style diffusion sampler in the embodiment of the present application can be a pre-trained style diffusion sampler. The style diffusion sampler gradually converts sample data (such as target word-level coding) into a Gaussian distribution, learns the relationship between sample data and noisy data, and recovers sample data from the Gaussian distribution. The style diffusion sampler can improve the de-noising capability of the target speech generation model, so that the noise resistance of the target speech generation model is stronger.

[0130] In some examples, in the training process of the style diffusion sampler, first, the sample word-level coding is obtained (wherein the sample word-level coding can be obtained by the above steps 110 to 140); then, the sample word-level coding is gradually added with Gaussian noise in an iterative manner until the sample vector corresponding to the sample word-level coding becomes a pure Gaussian noise vector.

[0131] For example, the sample word-level coding is y0, the iteration number is t, and the sample vector after adding noise is y t . t = k t * y t-1 . That is, based on each y t-1 , a small Gaussian noise k t is added, until y t becomes a pure Gaussian noise vector. Wherein k indicates the variance, k t indicates the variance value corresponding to different cycle numbers t, and k t can be calculated based on the standard Gaussian distribution in advance, that is, for any t, a certain k t can be directly calculated.

[0132] On the basis of the above, t is randomly selected, and the corresponding k t is determined through the above calculation process. y t , the above parameters are used as the basis for training the model. Wherein y0 and t are training inputs, and y tTo train the output, the style diffusion sampler (also referred to as a diffusion model) is trained, so that the style diffusion sampler learns the relationship between the sample data and the noisy data. It should be noted that, during the training process, due to the large variation of the model loss in the early training stage, the random selection of t can effectively avoid the model from excessively focusing on the information of a certain time period in the early training stage, and thus ignoring the information of other time periods. Based on this, the random distribution of t should be as uniform as possible.

[0133] After the training is completed based on sufficient sample data, the trained parameters are processed in reverse, so that the style diffusion sampler recovers the data from the Gaussian distribution through the reverse process, thereby obtaining the prosody vector (i.e., the first prosody vector) corresponding to the text to be processed.

[0134] The prosody vector (i.e., the first prosody vector) corresponding to the text to be processed output by the syntax graph construction sub-model 100 can be further improved to improve the accuracy of the prosody information corresponding to the text to be processed, and provide a more optimal input for subsequent generation of target synthesized speech.

[0135] Step 1512, based on the target reference speech and the feature extractor in the style vector unit, a second prosody vector corresponding to the target reference speech is generated.

[0136] The second prosody vector is used to represent the second prosody information. That is, the feature extractor in the style vector unit 31 can extract the prosody features (i.e., the second prosody features) corresponding to the target reference speech.

[0137] In some embodiments, the feature extractor can include an X-vector feature extractor. The X-vector feature extractor can include a plurality of time delay network layers and a plurality of fully connected network layers.

[0138] In some embodiments, the above step 1512 includes: based on the target reference speech, generating reference audio features corresponding to the target reference speech; and sequentially passing the reference audio features through a plurality of time delay network layers and a plurality of fully connected network layers to generate an X-vector of a preset dimension.

[0139] For example, the reference audio features can include Mel Frequency Cepstral Coefficients (MFCCs) features, and the second prosody vector includes an X-vector.

[0140] In some examples, the X-vector feature vector extractor employs a deep neural network (DNN) to map the target reference speech to a preset fixed-dimension embedding, referred to as an X-vector. The X-vector can better represent the prosody information (i.e., the second prosody information) corresponding to the target reference speech, and can represent the timbre information, the pitch information, and the energy information of the target reference speech.

[0141] Figure 11 An X-vector feature vector extractor provided by an embodiment of the present application is shown in a schematic diagram. As shown in the diagram, the X-vector feature vector extractor includes multiple time-delay network layers and multiple fully connected network layers. Figure 11

[0142] It should be noted that the number of the time-delay network layers and the fully connected network layers in the X-vector feature vector extractor is not limited by the embodiments of the present application. Figure 11 For example, the X-vector feature vector extractor includes five time-delay network layers (i.e., time-delay network layer 1 to time-delay network layer 5) and three fully connected network layers (i.e., fc 1 to fc 3).

[0143] As shown in the diagram, the parameters of each time-delay network layer can be represented as F (N, D, K), the parameters F of each time-delay network layer are activated by the context with a width of K and an extension length of D from the previous layer, and N is the dimension of the output vector. Figure 11 For example, when the features of the target reference speech are 24-dimension mel-frequency cepstral coefficient (MFCC) features, and the frame length is 25 ms, the feature map output by the time-delay network layer 1 is subjected to 1-dimension convolution, and the input channel x convolution kernel size x output channel = 24 x 5 x 512; the feature map output by the time-delay network layer 2 and the time-delay network layer 3 is subjected to 1-dimension convolution, and the input channel x convolution kernel size x output channel = 512 x 3 x 512; the feature map output by the time-delay network layer 4 is subjected to 1-dimension convolution, and the input channel x convolution kernel size x output channel = 512 x 1 x 512; and the feature map output by the time-delay network layer 5 is subjected to 1-dimension convolution, and the input channel x convolution kernel size x output channel = 512 x 1 x 1500. After the output of the time-delay network layer 5, the output feature vector is subjected to further statistical pooling processing by fc1, fc2, and fc3.

[0144]

[0145] ​​For example, the statistical pooling processing can be to calculate the mean and standard deviation in the dimension of sequence length T, and then concatenate the mean and standard deviation. Therefore, after statistical pooling, the dimension of sequence length T disappears, and 1500 mean values and 1500 standard deviations are obtained, which are concatenated to obtain a vector with a length of 3000. Among them, fc1, fc2 and fc3 can all be standard fully connected network layers, and fc3 can be a feature vector representation of the target reference speech, which contains the prosody information and phoneme information corresponding to the target reference speech.

[0146] The embodiment of the present application inputs the target reference speech into the feature extractor (such as the X-vector feature extractor) in the style vector unit 31 to extract the prosody vector (i.e. the second prosody vector) corresponding to the target reference speech, and provides a more optimal input for subsequent generation of the target synthesized speech.

[0147] Step 1513, generating a target style vector based on the first prosody vector and the second prosody vector.

[0148] After obtaining the first prosody vector through step 1511 and obtaining the second prosody vector through step 1512, the target style vector can be generated based on the first prosody vector and the second prosody vector.

[0149] In some embodiments, the above step 1513 includes: performing a summation operation on the first prosody vector and the second prosody vector to obtain an initial style vector; and performing a temporal average pooling processing on the initial style vector to determine the target style vector.

[0150] In some examples, the temporal average pooling processing can determine an optimal one of the plurality of initial style vectors as the target style vector.

[0151] In some examples, the temporal average pooling (Temporal Average Pooling) can be an operation performed on time series data, and the main purpose is to reduce the complexity of the time dimension (such as T) while retaining important information in each feature channel (such as F). For example, the temporal average pooling processing can average all values in each time dimension (T), and each feature channel becomes a single value.

[0152] For example, temporal average pooling can transform a feature map of dimension (batchsize, F, T) into a feature vector of dimension (batchsize, F). During temporal average pooling, the dimension T disappears. Therefore, temporal average pooling can be viewed as extracting the most representative features from a series of frames, transforming the variable-length frame sequence corresponding to the most representative features into a fixed-length feature vector (i.e., a target style vector). For example, the target style vector can be determined by averaging the feature vectors of multiple frames.

[0153] Step 1520: Generate target synthesized speech based on the target style vector, the text to be processed, and the generation unit in the target speech generation network.

[0154] For example, the target synthesized speech has prosodic features corresponding to the text to be processed (i.e., the first prosodic feature) and prosodic features corresponding to the reference speech (i.e., the second prosodic feature).

[0155] In some embodiments, the generation unit 32 includes a phoneme encoder, a variance adapter, a melm encoder, and a vocoder.

[0156] Figure 12 This is a schematic diagram illustrating another text processing method provided in an embodiment of this application. The following is in conjunction with... Figure 12 The process of generating the target synthesized speech by the generation unit 32 is explained. For example... Figure 12 As shown, step 1520 above includes steps 1521 to 1523 as shown below.

[0157] Step 1521: Based on the text to be processed, the target style vector, and the phoneme encoder, generate the phoneme state sequence corresponding to the text to be processed.

[0158] In some embodiments, the phoneme encoder includes at least one style-adaptive normalization layer.

[0159] For example, the phoneme encoder further includes a first convolutional layer, a first fully connected layer, a plurality of first conversion modules, and a second fully connected layer.

[0160] In some embodiments, step 1521 includes: determining an initial phoneme vector and a phoneme position encoding corresponding to the text to be processed based on the text to be processed; generating a first intermediate sequence based on the initial phoneme vector and the processing of the first convolutional layer and the first fully connected layer in the phoneme encoder; and generating a phoneme state sequence based on the first intermediate sequence, the phoneme position encoding, the target style vector, and multiple first transformation modules and the second fully connected layer in the phoneme encoder.

[0161] In some examples, the phoneme encoder can convert the input text to be processed into a phoneme sequence, and further convert the phoneme sequence into a phoneme encoding result and output, which is a phoneme hidden state sequence.

[0162] Exemplarily, the phoneme position encoding is used to represent the position information of each phoneme corresponding to the text to be processed in the text to be processed. Wherein, the phoneme position encoding can be determined in different ways.

[0163] In some examples, the phoneme position encoding can be determined by absolute position encoding. For example, an index (such as an integer index) can be assigned to each phoneme in the text to be processed, such as the index of the first phoneme is 1, the index of the second phoneme is 2, and so on. The position encoding of each phoneme can be determined by the index information of each phoneme.

[0164] In other examples, the phoneme position encoding can be determined by relative position encoding. For example, the relative order between phonemes can be used for encoding, such as based on the distance between phonemes. For example, the first phoneme encoding can be [1, 0, 0,...], the second phoneme encoding can be [0, 1, 0,...], and so on. The position encoding of each phoneme corresponding to the text to be processed can be determined. Alternatively, a function can also be used for encoding, such as using sin / cos position embedding to determine the position encoding of each phoneme.

[0165] In yet other examples, the phoneme position encoding can also be determined by combining absolute position and relative position encoding. For example, in some cases, the absolute position of the phoneme and the position relative to other phonemes can be considered at the same time, that is, the position encoding of each phoneme can be determined by combining the two encoding methods.

[0166] It should be noted that in the embodiments of the present application, the position encoding of each phoneme in the text to be processed can be determined by any one of the above three methods, or other methods can also be used to determine the position encoding of each phoneme. The method of phoneme position encoding is not limited in the embodiments of the present application.

[0167] Exemplarily, the first conversion module can include a first attention mechanism layer, at least one style adaptive normalization layer, and a second convolutional layer. Wherein, the number of the first conversion module in the phoneme encoder can be multiple, for example, it can be 4, or it can be greater than 4, and the present application is not limited thereto.

[0168] Step 1522, based on the phoneme state sequence and the variance adapter, adding the preset variance information to the phoneme state sequence to obtain an adjusted phoneme state sequence.

[0169] Exemplarily, the preset variance information comprises at least one of duration information, pitch information and energy information.

[0170] In some examples, after obtaining the phoneme state sequence (which can also be referred to as a phoneme hidden state sequence) through the phoneme encoder, the phoneme state sequence is input into a variance adapter. The variance adapter is used to add additional variance information to the phoneme state sequence, thereby obtaining an adjusted phoneme state sequence. The variance adapter can add duration, pitch and energy information to the phoneme state sequence.

[0171] For example, the variance adapter can predict the duration of the phoneme in each duration through a duration predictor, so as to adjust the length of the speech frame in the phoneme state sequence, and predict and adjust the corresponding pitch value and energy value through a pitch predictor and an energy predictor respectively.

[0172] In step 1523, based on the adjusted phoneme state sequence, the target style vector and the mel-spectrogram encoder, the adjusted phoneme state sequence is converted into a mel-spectrogram sequence.

[0173] In some embodiments, the mel-spectrogram encoder comprises at least one style-adaptive normalization layer.

[0174] Exemplarily, the mel-spectrogram encoder further comprises a third fully connected layer, a plurality of second conversion modules and a fourth fully connected layer.

[0175] In some embodiments, the above step 1523 comprises: generating a second intermediate sequence based on the adjusted phoneme state sequence and the third fully connected layer in the mel-spectrogram encoder; and generating the mel-spectrogram sequence based on the second intermediate sequence, the target style vector, and the plurality of second conversion modules and the fourth fully connected layer in the mel-spectrogram encoder.

[0176] Exemplarily, the second conversion module comprises a second attention mechanism layer, at least one style-adaptive normalization layer and a third convolutional layer. The number of the second conversion modules in the mel-spectrogram encoder can be multiple, for example, can be 4, or can be greater than 4, and the embodiments of the present application do not limit this.

[0177] It should be noted that the structure of the second conversion module in the mel-spectrogram encoder and the first conversion module in the phoneme encoder can be the same, and the number of the second conversion module and the first conversion module can also be the same.

[0178] Exemplarily, at least one style-adaptive normalization layer is arranged in the mel-spectrogram encoder and the phoneme encoder, and the target style vector generated by the style vector unit 31 can be input into each style-adaptive normalization layer to further process the target style vector.

[0179] The following describes the processing procedure of the style-adaptive normalization layer on the target style vector. It should be noted that the processing procedure of each style-adaptive normalization layer in the mel-spectrogram encoder and the phoneme encoder is similar, and the style-adaptive normalization layer mentioned in the following embodiments can be any style-adaptive normalization layer in the mel-spectrogram encoder or any style-adaptive normalization layer in the phoneme encoder.

[0180] In some examples, the style-adaptive normalization layer can be used to add the target style vector generated by the style vector unit 32 to the process of voice synthesis. For example, the style-adaptive normalization layer can adjust the synthesized voice based on the target style vector after receiving the target style vector. For example, assuming that the target style feature vector h = (h1, h2,..., hH), where H is the dimension of the style feature vector, and the normalized vector y = (y1, y2,..., yH) output after the style-adaptive normalization layer is y = (y1, y2,..., yH). H H

[0181] In some examples, the normalized vector y can be determined by the following formula (1) and formula (2).

[0182]

[0183] where μ is the mean of the normal distribution, and σ is the standard deviation of the normal distribution.

[0184] On this basis, the gain and bias of the target style vector can be further determined by the following formula (3).

[0185] SALN(h, w) = g(w) y + b(w) formula (3) ;

[0186] where g(w) and b(w) are the gain and bias parameters of the style-adaptive normalization layer (which can also be called an affine layer and a SALN layer), and the SALN layer is a single fully connected layer.

[0187] In some examples, unlike the fixed gain and bias in the traditional normalization layer, g(w) and b(w) can adaptively perform scaling and moving of the features in the target voice generation network 30 according to the target style vector itself. The embodiments of the present application replace the original normalization layer with a style-adaptive normalization layer (SALN), and construct an affine layer that converts the target style vector into the corresponding bias and gain according to the above method.

[0188] Step 1524, generating a target synthesized voice based on the mel-spectrogram sequence and the vocoder.

[0189] ​​For example, after obtaining the sequence of mel-spectrogram output by the mel-spectrum encoder, the sequence of mel-spectrogram can be input to the vocoder to determine the finally generated target synthesized speech.

[0190] The text processing method provided by the embodiments of the present application extracts the prosody of the target word-level code and the target reference speech by using the style vector unit in the process of generating the target synthesized speech by the target speech generation network, and then calculates the target style vector. On the other hand, for the target style vector, the embodiments of the present application use a style-adaptive manner to embed it into the generation process of the generation unit (such as the style-adaptive normalization layer), so that the generated effect is more close to the target synthesized speech of the target reference speech and the text to be processed.

[0191] Figure 13 A schematic diagram of a target speech generation network is provided by the embodiments of the present application. As shown in Figure 13 , the target speech generation network 30 includes a style vector unit 31 Figure 13 (left part in the middle) and a generation unit 32 Figure 13 (right part in the middle). The style vector unit 31 includes a style diffusion sampler, a prosody text encoder, and a V-vector feature vector extractor. The generation unit 32 includes a phoneme encoder, a variance adapter, a mel-spectrum encoder, and a vocoder.

[0192] The process of generating the target synthesized speech by the target speech generation network 30 will be described below. Figure 13

[0193] After obtaining the text to be processed, the text to be processed is converted into a phoneme vector (such as 256 dimensions). The phoneme vector is input to the style vector unit 31. The prosody text encoder and the style diffusion sampler in the style vector unit 31 extract a first prosody vector corresponding to the text to be processed according to the phoneme vector. The target reference speech is input to the style vector unit 31, the MCFF feature of the target reference speech is obtained, and the MCFF feature of the target reference speech is input to the V-vector feature vector extractor in the style vector unit 31 to extract a second prosody vector corresponding to the target reference speech. After summing the first prosody vector and the second prosody vector, the style vector is obtained. And the style vector is processed by time sequence average pooling to obtain the target style vector.

[0194] ​The phoneme vector is input to the phoneme encoder in the generation unit 32, specifically, the phoneme vector can be input to 2 one-dimensional convolution layers with residual connection, and then input to a fully connected layer. Among them, the filter number of the one-dimensional convolution layer in the phoneme encoder is 256, and the convolution kernel size is 3. Then, the phoneme vector processed by the fully connected layer is added to the phoneme position encoding (the phoneme position encoding is used for the position of the phoneme in the sentence), and the result (i.e. the first intermediate sequence) and the target style vector are input to 4 FFT blocks for calculation to obtain the phoneme encoding result, and then the phoneme encoding result (i.e. the phoneme state sequence) is output by the fully connected layer.

[0195] The phoneme encoding result output by the phoneme encoder is input to the variance adapter, and the variance adapter is used to extract three kinds of information in the phoneme encoding result as output, including phoneme duration, pitch (used to express emotional and prosodic information), and energy (used to express volume information).

[0196] After the variance adapter extracts the above information, the extracted information (such as the adjusted phoneme state sequence) is input to the mel-spectrogram encoder. Specifically, the output result of the variance adapter can be input to 2 fully connected layers (such as [128, 256] fully connected layers) for processing, and the processed result (i.e. the second intermediate sequence) is added to the phoneme position encoding. After adding, the added result and the target style vector are input to 4 FFT blocks for calculation, and finally the dimension of the mel-spectrogram is adapted by a fully connected layer to output the mel-spectrogram (i.e. the mel-spectrogram sequence).

[0197] As shown in Figure 13 In the above process, each FFT block is composed of a multi-head attention mechanism, a style adaptive normalization, a one-dimensional convolution, and another style adaptive normalization in turn. The target style vector output by the style vector unit 31 is embedded into each style adaptive normalization for processing. Among them, in each FFT block, the hidden layer size of the one-dimensional convolution can be 256, the number of attention heads can be 2, the convolution kernel size can be 9, and the filter number can be 1024. All activation functions can be Mish activation functions, and the parameter size of the random inactivation layer can be set to 0.1.

[0198] Finally, the mel-spectrogram output by the mel-spectrogram encoder is output to the vocoder as the final synthesized speech, i.e. the target synthesized speech.

[0199] The text processing method provided in the embodiments of the present application generates a target synthesized speech corresponding to the target reference speech and the text to be processed through a pre-trained target speech generation model. First, a network is constructed based on the text information corresponding to the text to be processed and a target syntax graph in the target speech generation model, to generate a target syntax graph corresponding to the text to be processed. Then, a target character-level coding corresponding to the text to be processed is generated based on the phoneme information and the boundary information corresponding to the text to be processed, the target syntax graph, and a target coding network in the target speech generation model. Finally, a target synthesized speech is generated based on the target character-level coding, the target reference speech, and a target speech generation network in the target speech generation model. In the text processing method provided in the embodiments of the present application, the target syntax graph generated by the target syntax graph construction network can represent the syntax relationship between each character in the plurality of characters, is more consistent with the syntax information of the Chinese syntax, and can provide more consistent prosodic features of the text information. Therefore, the target character-level coding generated according to the target syntax graph can have the prosodic features corresponding to the text to be processed, so that the target synthesized speech generated based on the target character-level coding and the target reference speech also has the prosodic features of the text to be processed. Therefore, the text processing method provided in the embodiments of the present application can extract the prosodic features of the text information in the text-to-speech synthesis process, to improve the authenticity and richness of the synthesized speech.

[0200] In some embodiments, the text processing method further includes: obtaining sample text data and sample reference speech data; generating a predicted synthesized speech based on the sample text data, the sample reference speech data, and a speech generation model to be trained; wherein the speech generation model to be trained includes a syntax graph construction network to be trained and a coding network to be trained; obtaining sample synthesized speech data; iteratively training the speech generation model to be trained, taking the predicted synthesized speech as initial training output information of the speech generation model to be trained and taking the sample synthesized speech data as supervision information, to obtain a target speech generation model; wherein the target speech generation model includes a target syntax graph construction network and a target coding network.

[0201] Exemplarily, the target speech generation model 1 involved in the above embodiments is a pre-trained model. The syntax graph construction sub-model 100 and the generation sub-model 200 in the target speech generation model 1 are both trained models. It should be noted that the syntax graph construction sub-model 100 and the generation sub-model 200 can be trained as a whole or can be trained separately, and the embodiments of the present application do not limit this.

[0202] In some examples, the speech generation model to be trained can include a syntax graph construction network to be trained and a coding network to be trained, and training the speech generation model to be trained includes training the syntax graph construction network to be trained and the coding network to be trained, to obtain a target syntax graph construction network and a target coding network.

[0203] In some examples, the to-be-trained speech generation model further includes a to-be-trained speech generation subnetwork, and training the training speech generation model includes training the to-be-trained speech generation subnetwork, so as to obtain the target speech generation network.

[0204] Exemplarily, in the model pre-training process, the audio corresponding to at least one reference speaker can be taken as sample reference speech data for training of the model. For example, the number of reference speakers can be greater than a preset number, such as greater than 100. The reference speaker corresponds to at least 100 pieces of sample speech, and the audio length of each piece of sample speech is greater than a preset audio length, such as greater than 3 seconds.

[0205] In some examples, after the sample text data and the sample reference speech data are obtained, a predicted synthesized speech can be generated based on the sample text data, the sample reference speech data, and the to-be-trained speech generation model. Then, the predicted synthesized speech is taken as initial training output information of the to-be-trained speech generation model, the sample synthesized speech data is taken as supervision information, and the to-be-trained speech generation model is iterated to obtain the target speech generation model.

[0206] It should be noted that the text processing method provided by the embodiments of the present application can not only make the prosodic features of the obtained target synthesized speech more ideal, but also can synthesize synthesized speeches of different speech styles in the case of giving other reference speeches.

[0207] Figure 14 A schematic diagram of a text processing apparatus provided by an embodiment of the present application is shown in FIG. 14. As shown in FIG. 14, the text processing apparatus 1400 includes an obtaining module 1410, a determining module 1420, a first generating module 1430, a second generating module 1440, and a third generating module 1450. Wherein: Figure 14

[0208] The obtaining module 1410 is configured to obtain a to-be-processed text and a target reference speech.

[0209] The determining module 1420 is configured to determine text information and phoneme information corresponding to the to-be-processed text according to the to-be-processed text; wherein the text information includes text content and boundary information.

[0210] The first generating module 1430 is configured to generate a target syntax graph corresponding to the to-be-processed text based on the text information and a target syntax graph construction network in a target speech generation model; wherein the text content includes a plurality of characters, and the target syntax graph is used to represent the syntactic relationship between the characters.

[0211] ​The second generation module 1440 is configured to generate the target word-level code corresponding to the to-be-processed text based on the phoneme information, the boundary information, the target syntax graph, and a target encoding network in the target speech generation model.

[0212] The third generation module 1450 is configured to generate a target synthesized speech based on the target word-level code, the target reference speech, and a target speech generation network in the target speech generation model; wherein the target synthesized speech comprises prosodic features of the to-be-processed text.

[0213] In some embodiments, the first generation module 1430 is configured to generate an initial dependency graph corresponding to the text information based on the text content, the word boundaries in the boundary information, and a dependency parser in the syntax graph construction network; wherein the initial dependency graph comprises a plurality of word nodes and a unidirectional connection relationship between each word node in the plurality of word nodes; and generate the target syntax graph corresponding to the initial dependency graph based on the initial dependency graph and a syntax graph constructor in the syntax graph construction network.

[0214] In some embodiments, the first generation module 1430 is configured to divide each word node in the initial dependency graph into at least one character node and determine a first character node in each word node based on the syntax graph constructor and the character boundaries in the boundary information; establish a first dependency connection relationship between the plurality of first character nodes, the start node and the end node; establish a second dependency connection relationship between at least one character node in each word node according to the intra-word order of the at least one character node in each word node; and obtain the target syntax graph according to the first dependency connection relationship and the second dependency connection relationship; wherein the first dependency connection relationship and / or the second dependency connection relationship are used to represent a bidirectional connection relationship between each character node.

[0215] In some embodiments, the second generation module 1440 is configured to generate an initial word-level code corresponding to the to-be-processed text based on the phoneme information, the character boundaries in the boundary information, and a word-level average pooling layer in the target encoding network; and generate the target word-level code based on the initial word-level code, the target syntax graph, and a plurality of gated graph convolution layers in the target encoding network.

[0216] In some embodiments, the second generation module 1440 is configured to determine at least one phoneme feature corresponding to each character node in the to-be-processed text based on the phoneme information and the character boundaries; perform an average pooling operation on the at least one phoneme feature corresponding to each character node to determine a comprehensive phoneme feature corresponding to each character node; and determine the initial word-level code corresponding to the to-be-processed text according to the comprehensive phoneme feature corresponding to each character node.

[0217] In some embodiments, the second generation module 1440 is configured to generate a first word-level encoding based on the initial word-level encoding, the target syntax graph, and a first gated graph convolution layer in the plurality of gated graph convolution layers; generate a second word-level encoding based on the first word-level encoding, the target syntax graph, and a second gated graph convolution layer in the plurality of gated graph convolution layers; and determine the target word-level encoding based on the initial word-level encoding, the first word-level encoding, and the second word-level encoding.

[0218] In some embodiments, the third generation module 1450 is configured to generate a target style vector based on the target word-level encoding, the target reference speech, and a style vector unit in the target speech generation network, wherein the target style vector is used to represent prosodic information corresponding to the target reference speech and prosodic information corresponding to the text to be processed; and generate the target synthesized speech based on the target style vector, the text to be processed, and a generation unit in the target speech generation network.

[0219] In some embodiments, the text processing apparatus 1400 further includes a training module. The acquisition module 1410 is configured to acquire sample text data and sample reference speech data, and acquire sample synthesized speech data. The training module is configured to generate a predicted synthesized speech based on the sample text data, the sample reference speech data, and a to-be-trained speech generation model; iteratively train the to-be-trained speech generation model to obtain a target speech generation model, taking the predicted synthesized speech as initial training output information of the to-be-trained speech generation model, and taking the sample synthesized speech data as supervision information; wherein the to-be-trained speech generation model includes a to-be-trained syntax graph construction network and a to-be-trained encoding network, and the target speech generation model includes a target syntax graph construction network and a target encoding network.

[0220] Figure 15 A schematic diagram of an electronic device is provided for the embodiments of the present application. In some embodiments, the electronic device includes one or more processors and a memory. The memory is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the text processing method in the above embodiments.

[0221] As shown in Figure 15 The electronic device 1000 includes a processor 1001 and a memory 1002. The electronic device 1000 may, for example, also include a communications interface 1003 and a communications bus 1004. The processor 1001, the memory 1002, and the communications interface 1003 communicate with each other through the communications bus 1004. The communications interface 1003 is used to communicate with network elements such as clients or other servers.

[0222] In some embodiments, the processor 1001 is configured to execute programs 1005, and specifically can execute the related steps in the text processing method embodiments described above. Specifically, the programs 1005 can include program codes including computer-executable instructions.

[0223] For example, the processor 1001 can be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application. The one or more processors of the electronic device 1000 can be of the same type, such as one or more CPUs, or can be of different types, such as one or more CPUs and one or more ASICs.

[0224] In some embodiments, the memory 1002 is configured to store the programs 1005. The memory 1002 can include a high-speed RAM memory, and can also include a non-volatile memory (NVM), such as at least one disk memory. The programs 1005 can specifically be invoked by the processor 1001 to cause the electronic device 1000 to perform the operations of the text processing method.

[0225] The embodiments of the present application provide a computer-readable storage medium storing at least one executable instruction, which, when executed on the electronic device 1000, causes the electronic device 1000 to perform the text processing method in the above-described embodiments.

[0226] The executable instructions can specifically be used to cause the electronic device 1000 to perform the operations of the text processing method.

[0227] For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0228] In some embodiments, the embodiments of the present application provide a computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the text processing method described in any of the above embodiments.

[0229] In some embodiments, the embodiments of the present application further provide a computer program which, when executed by a processor, can implement the text processing method of any of the above-mentioned embodiments.

[0230] The text processing apparatus, electronic device, computer readable storage medium, computer program product and computer program provided by the embodiments of the present application can achieve the beneficial effects provided by the text processing method, which will not be described here again.

[0231] It should be noted that, in the application, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0232] Each of the embodiments in the specification is described in a relevant manner, and the same or similar parts between each of the embodiments can be referred to each other, and each of the embodiments focuses on the difference from other embodiments. Especially, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the part of the method embodiment.

[0233] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be specifically embodied in any computer readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from an instruction execution system, device or apparatus) or in conjunction with these instructions execution system, device or apparatus.

[0234] For the purpose of the present specification, "computer readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in conjunction with an instruction execution system, device or apparatus.

[0235] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM).

[0236] In addition, a computer readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for instance via an optical scanner, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory. It will be understood that parts of this application can be implemented by hardware, software, firmware or a combination thereof.

[0237] In the embodiments described above, the steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0238] The embodiments of the application described above do not represent a limitation in the scope of protection of the application.

Claims

1. A text processing method, characterized in that, The method includes: Obtain the text to be processed and the target reference speech; Based on the text to be processed, determine the text information and phoneme information corresponding to the text to be processed; wherein, the text information includes text content and boundary information, and the text content includes multiple characters; Based on the text content, word boundaries in the boundary information, and the dependency parser in the target syntactic graph construction network of the target speech generation model, an initial dependency graph corresponding to the text information is generated; wherein, the initial dependency graph includes multiple word nodes and unidirectional dependency connections between each word node; Based on the initial dependency graph and the target syntactic graph, a syntactic graph builder is constructed in the network to construct the target syntactic graph corresponding to the text to be processed; wherein, the target syntactic graph is used to represent the syntactic relationships between characters among the plurality of characters; The step of constructing the target syntactic graph corresponding to the text to be processed by the syntactic graph builder in the network based on the initial dependency graph and the target syntactic graph includes: Based on the syntactic graph builder and the word boundaries in the boundary information, each word node in the initial dependency graph is divided into at least one byte point, and the first byte point in each word node is determined. Establish a bidirectional connection between the starting node and the first byte of the first word node among the plurality of word nodes; according to the order of the word nodes among the plurality of word nodes, establish a bidirectional connection between the first byte of the first word node and the first byte of the other word nodes in sequence; and establish a bidirectional connection between the first byte of the last word node among the plurality of word nodes and the ending node, thereby obtaining the first dependency connection relationship; Based on the intra-word order of multiple byte points in each of the word nodes, a second dependency connection relationship between two byte points in each of the word nodes is established sequentially. The target syntax graph is obtained based on the first dependency connection relationship and the second dependency connection relationship; wherein the first dependency connection relationship and / or the second dependency connection relationship are used to characterize the bidirectional connection relationship between each byte point.

2. The method according to claim 1, characterized in that, After constructing the target syntactic graph corresponding to the text to be processed, the method further includes: Based on the phoneme information, the boundary information, the target syntax graph, and the target encoding network in the target speech generation model, the target character-level encoding corresponding to the text to be processed is generated; Based on the target word-level encoding, the target reference speech, and the target speech generation network in the target speech generation model, target synthesized speech is generated; wherein, the target synthesized speech includes the prosodic features of the text to be processed.

3. The method according to claim 2, characterized in that, The step of generating the target character-level code corresponding to the text to be processed based on the phoneme information, the boundary information, the target syntax graph, and the target encoding network in the target speech generation model includes: Based on the phoneme information, the character boundaries in the boundary information, and the character-level average pooling layer in the target encoding network, an initial character-level encoding corresponding to the text to be processed is generated; The target character-level code is generated based on the initial character-level code, the target syntax graph, and multiple gated graph convolutional layers in the target encoding network.

4. The method according to claim 3, characterized in that, The step of generating the initial character-level encoding corresponding to the text to be processed based on the phoneme information, the character boundaries in the boundary information, and the character-level average pooling layer in the target encoding network includes: Based on the phoneme information and the word boundary, at least one phoneme feature corresponding to each byte point in the text to be processed is determined; Perform average pooling on at least one phoneme feature corresponding to each byte point to determine the comprehensive phoneme feature corresponding to each byte point; The initial character-level encoding corresponding to the text to be processed is determined based on the comprehensive phoneme features corresponding to each byte point.

5. The method according to claim 3, characterized in that, The step of generating the target character-level encoding based on the initial character-level encoding, the target syntax graph, and multiple gated graph convolutional layers in the target encoding network includes: Based on the initial character-level encoding, the target syntax graph, and the first gated graph convolutional layer among the plurality of gated graph convolutional layers, a first character-level encoding is generated; Based on the first character-level encoding, the target syntax graph, and the second gated graph convolutional layer among the plurality of gated graph convolutional layers, a second character-level encoding is generated; The target character encoding is determined based on the initial character encoding, the first character encoding, and the second character encoding.

6. The method according to claim 2, characterized in that, The step of generating target synthesized speech based on the target word-level encoding, the target reference speech, and the target speech generation network in the target speech generation model includes: Based on the target character-level encoding, the target reference speech, and the style vector unit in the target speech generation network, a target style vector is generated; wherein, the target style vector is used to represent the prosodic information corresponding to the target reference speech and the prosodic information corresponding to the text to be processed; The target synthesized speech is generated based on the target style vector, the text to be processed, and the generation unit in the target speech generation network.

7. The method according to claim 2, characterized in that, The method further includes: Obtain sample text data and sample reference speech data; Based on the sample text data, the sample reference speech data, and the speech generation model to be trained, a predicted synthesized speech is generated; wherein, the speech generation model to be trained includes a syntactic graph construction network to be trained and an encoding network to be trained. Acquire sample synthesized speech data; Using the predicted synthesized speech as the initial training output information of the speech generation model to be trained, and the sample synthesized speech data as supervision information, the speech generation model to be trained is iterated to obtain the target speech generation model; wherein, the target speech generation model includes the target syntax graph construction network and the target encoding network.

8. A text processing device, characterized in that, include: The acquisition module is configured to acquire the text to be processed and the target reference speech. The determining module is configured to: determine the text information and phoneme information corresponding to the text to be processed based on the text to be processed; wherein, the text information includes text content and boundary information, and the text content includes multiple characters; The generation module is configured to: generate an initial dependency graph corresponding to the text information based on the text content, word boundaries in the boundary information, and the dependency parser in the target syntactic graph construction network of the target speech generation model; wherein, the initial dependency graph includes multiple word nodes and unidirectional dependency connections between each word node; The construction module is configured to: construct a syntactic graph builder in the network based on the initial dependency graph and the target syntactic graph, and construct the target syntactic graph corresponding to the text to be processed; wherein, the target syntactic graph is used to represent the syntactic relationships between characters among the plurality of characters; The construction module is specifically configured to: divide each word node in the initial dependency graph into at least one byte point based on the syntactic graph builder and the word boundaries in the boundary information, and determine the first byte point in each word node; establish a bidirectional connection between the start node and the first byte point of the first word node among the multiple word nodes; establish a bidirectional connection between the first byte point of the first word node and the first byte point of other word nodes according to the order of the word nodes among the multiple word nodes; and establish a bidirectional connection between the first byte point of the last word node and the end node among the multiple word nodes to obtain a first dependency connection relationship; establish a second dependency connection relationship between two byte points in each word node according to the intra-word order of the multiple byte points in each word node; and obtain the target syntactic graph according to the first dependency connection relationship and the second dependency connection relationship; wherein the first dependency connection relationship and / or the second dependency connection relationship are used to characterize the bidirectional connection relationship between each byte point.

9. An electronic device, characterized in that, include: One or more processors; and The memory is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the text processing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the text processing method according to any one of claims 1-7.

Images