Speech synthesis method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a speech synthesis method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first text, and extracting the semantic features of the first text through a speech synthesis model; obtaining first voice data with a first tone, and generating a first acoustic feature through the voice synthesis model according to the semantic feature of the first text and the first voice data; the first acoustic feature is at least used for representing semantic information of the first text and timbre information of the first timbre; generating second voice data with the first tone according to the first acoustic feature through the voice synthesis model; the text corresponding to the second voice data is the first text.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech synthesis method, device, electronic device and storage medium. Background Art

[0002] Text-to-speech (TTS) refers to the synthesis of text into speech data. Speech synthesis can be achieved through a speech synthesis model. The speech synthesis model can be trained based on the speech data of a single speaker, and the text can be synthesized into speech data according to the timbre of the single speaker during speech synthesis. Alternatively, the speech synthesis model can be trained based on the speech data of multiple speakers, and the text can be synthesized into speech data according to the timbre of one of the multiple speakers during speech synthesis. Alternatively, the speech synthesis model can be trained based on the speech data of multiple speakers, and the text can be synthesized into speech data according to the timbre of any speaker during speech synthesis, and the speech data of any speaker can be located in the training set or outside the training set. Summary of the invention

[0003] The embodiments of the present disclosure provide a speech synthesis method, device, electronic device and storage medium, which can synthesize text into speech data according to the timbre of any speaker and improve the pronunciation accuracy of the synthesized speech data.

[0004] In a first aspect, an embodiment of the present disclosure provides a speech synthesis method, including:

[0005] Acquire a first text, and extract semantic features of the first text through a speech synthesis model;

[0006] Acquire first speech data with a first timbre, and generate a first acoustic feature according to the semantic feature of the first text and the first speech data through the speech synthesis model; the first acoustic feature is used to at least represent the semantic information of the first text and the timbre information of the first timbre;

[0007] Through the speech synthesis model, second speech data having the first timbre is generated according to the first acoustic feature; the text corresponding to the second speech data is the first text.

[0008] In a second aspect, an embodiment of the present disclosure provides a speech synthesis device, including:

[0009] A semantic extraction module, used to obtain a first text and extract semantic features of the first text through a speech synthesis model;

[0010] An acoustic generation module, used for acquiring first speech data having a first timbre, and generating a first acoustic feature according to the semantic feature of the first text and the first speech data through the speech synthesis model; the first acoustic feature is used for representing at least the semantic information of the first text and the timbre information of the first timbre;

[0011] A speech generation module is used to generate second speech data having the first timbre according to the first acoustic feature through the speech synthesis model; the text corresponding to the second speech data is the first text.

[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0013] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in the first aspect above.

[0014] In one or more embodiments of the present disclosure, first, a first text is obtained, and the semantic features of the first text are extracted through a speech synthesis model. Then, a first speech data with a first timbre is obtained. A first acoustic feature is generated based on the semantic features of the first text and the first speech data through the speech synthesis model. The first acoustic feature is used to represent at least the semantic information of the first text and the timbre information of the first timbre. Finally, a second speech data with a first timbre is generated based on the first acoustic feature through the speech synthesis model. The text corresponding to the second speech data is the first text. It can be seen that through this embodiment, the semantic features of the first text can be extracted through the speech synthesis model. Based on the semantic features of the first text and the first speech data with the first timbre, the second speech data with the first timbre is synthesized. The text corresponding to the second speech data is the first text. The effect of synthesizing text into speech data according to the timbre of any given speaker is achieved, and the second speech data is synthesized based on the semantic features of the first text, so that the second speech data can accurately represent the semantic information of the first text, thereby improving the pronunciation accuracy of the synthesized speech data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate one or more embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0016] Figure 1 A flowchart of a speech synthesis method provided by an embodiment of the present disclosure;

[0017] Figure 2 A schematic diagram of the speech synthesis principle of a speech synthesis model provided in one embodiment of the present disclosure;

[0018] Figure 3 A schematic diagram of the speech synthesis principle of a speech synthesis model provided in yet another embodiment of the present disclosure;

[0019] Figure 4 A schematic diagram of the principle of performing autoregressive processing on text features of a first text provided by an embodiment of the present disclosure;

[0020] Figure 5 A schematic diagram of the principle of performing autoregressive processing on the first splicing feature provided by another embodiment of the present disclosure;

[0021] Figure 6a A schematic diagram of the training principle of a speech synthesis model provided in one embodiment of the present disclosure;

[0022] Figure 6b A schematic diagram of the training principle of a speech synthesis model provided in another embodiment of the present disclosure;

[0023] Figure 7 A schematic diagram of the training principle of the encoding unit and the decoding unit provided in one embodiment of the present disclosure;

[0024] Figure 8 A schematic diagram of the structure of a speech synthesis device provided in one embodiment of the present disclosure;

[0025] Fig. 9 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the protection scope of the present disclosure.

[0027] The embodiments of the present disclosure provide a speech synthesis method, which can synthesize text into speech data according to the timbre of any speaker and improve the pronunciation accuracy of the synthesized speech data. The semantic features involved in the various embodiments of the present disclosure are at least used to represent semantic information, and the semantic information at least includes the content expressed by the language, such as text content. The semantic features and semantic information can also refer to the general explanation of the prior art, which will not be repeated here.

[0028] Figure 1 A flowchart of a speech synthesis method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the process includes:

[0029] Step S102, obtaining a first text, and extracting semantic features of the first text through a speech synthesis model;

[0030] Step S104, obtaining first speech data with a first timbre, and generating a first acoustic feature according to the semantic feature of the first text and the first speech data through a speech synthesis model; the first acoustic feature is used to represent at least the semantic information of the first text and the timbre information of the first timbre;

[0031] Step S106: Generate second speech data with the first timbre through the speech synthesis model according to the first acoustic feature; the text corresponding to the second speech data is the first text.

[0032] In this embodiment, first, a first text is obtained, and the semantic features of the first text are extracted through a speech synthesis model. Then, first speech data with a first timbre is obtained. A first acoustic feature is generated based on the semantic features of the first text and the first speech data through the speech synthesis model. The first acoustic feature is used to represent at least the semantic information of the first text and the timbre information of the first timbre. Finally, a second speech data with a first timbre is generated based on the first acoustic feature through the speech synthesis model. The text corresponding to the second speech data is the first text. It can be seen that through this embodiment, the semantic features of the first text can be extracted through the speech synthesis model. Based on the semantic features of the first text and the first speech data with a first timbre, the second speech data with a first timbre is synthesized. The text corresponding to the second speech data is the first text. The effect of synthesizing text into speech data according to the timbre of any given speaker is achieved. The second speech data is synthesized based on the semantic features of the first text, so that the second speech data can accurately represent the semantic information of the first text, thereby improving the pronunciation accuracy of the synthesized speech data.

[0033] Below Figure 1 The process is described in detail.

[0034] In the above step S102, a first text to be synthesized into speech is obtained. The first text may include any text to be synthesized into speech data, and the first text may include a paragraph, one or more paragraphs, and the like.

[0035] In the above step S102, the semantic features of the first text are extracted by the speech synthesis model. The semantic features of the first text at least represent the semantic information of the first text, and the semantic information of the first text at least includes the text content of the first text. The semantic features of the first text can be represented in the form of a matrix, for example, by a matrix of size a1*a2, where a1 represents the length of the semantic features and a2 represents the dimension of the semantic features.

[0036] In the above step S104, first voice data with a first timbre is obtained. The first voice data has a first timbre, and the first timbre is the timbre that the synthesized voice data should have, so the first voice data is equivalent to data of the first timbre. The first voice data can be provided by a speaker with the first timbre. In one embodiment, in order to better learn the first timbre and synthesize voice data with the first timbre, the duration of the first voice data is required to be greater than a certain duration, such as greater than 10 seconds.

[0037] In the above step S104, a first acoustic feature is generated according to the semantic features of the first text and the first voice data through a speech synthesis model. In one embodiment, the timbre features of the first timbre are extracted according to the first voice data through a speech synthesis model, and the first acoustic feature is generated by feature generation based on the semantic features of the first text and the timbre features of the first timbre. Since the semantic features of the first text at least represent the semantic information of the first text, and the timbre features of the first timbre represent the timbre information of the first timbre, the first acoustic feature generated based on the semantic features of the first text and the timbre features of the first timbre can at least represent the semantic information of the first text and the timbre information of the first timbre, and the first acoustic feature can also represent the prosodic information when the first text is read aloud through the first timbre, and the prosodic information includes the pronunciation duration of each word, the pronunciation tone of each word, the pause duration between words, and other information when the first text is read aloud through the first timbre.

[0038] The timbre feature of the first timbre can be represented in the form of a matrix, for example, by a matrix of size b1*b2, where b1 represents the length of the timbre feature, and b2 represents the dimension of the timbre feature, wherein b1 can be equal to 1, indicating that the timbre feature is a global timbre feature without a length concept. The first acoustic feature can be represented in the form of a matrix, for example, by a matrix of size c1*c2, where c1 represents the length of the first acoustic feature, and c2 represents the dimension of the first acoustic feature.

[0039] In the above step S106, the second speech data is generated according to the first acoustic feature through the speech synthesis model. Since the first acoustic feature is used to represent at least the semantic information of the first text and the timbre information of the first timbre, the second speech data generated according to the first acoustic feature also has the semantic information and the first timbre of the first text, and the text corresponding to the second speech data is the first text, thereby achieving the effect of collectively referring to the first text as the second speech data according to the first timbre.

[0040] In this embodiment, it is not required that the speech data with the first timbre is in the training set for training the speech synthesis model, that is, the training set of the speech synthesis model may include the speech data with the first timbre or may not include the speech data with the first timbre, thereby achieving the effect of synthesizing the text into speech data according to the timbre of any given speaker, and because the second speech data is generated according to the semantic features of the first text, the second speech data can accurately represent the semantic information of the first text, thereby improving the pronunciation accuracy of the synthesized speech data.

[0041] In one embodiment, extracting semantic features of the first text through a speech synthesis model includes:

[0042] Obtaining text features of a first text;

[0043] The semantic features of the first text are extracted based on the text features of the first text by a text semantic extraction unit in the speech synthesis model.

[0044] Figure 2 A schematic diagram of the speech synthesis principle of a speech synthesis model provided in an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the speech synthesis model includes a text feature extraction unit and a text semantic extraction unit. The text feature extraction unit can be any neural network capable of extracting text features. The text semantic extraction unit can include an autoregressive model and can include a neural network stacked by multiple transformers (neural networks based on attention mechanisms). In this embodiment, the first text is input into the text feature extraction unit, the text features of the first text are extracted by the text feature extraction unit, the text features of the first text are input into the text semantic extraction unit, and the semantic features of the first text are extracted by the text semantic extraction unit based on the text features of the first text.

[0045] The text features of the first text can be represented in the form of a matrix, for example, by a matrix of size d1*d2, where d1 represents the length of the text feature, d2 represents the dimension of the text feature, and the data of all dimensions corresponding to any length in the matrix represent the features of one or more words in the first text.

[0046] It can be seen that through this embodiment, the text features of the first text can be obtained, and the text semantics extraction unit in the speech synthesis model can accurately and efficiently extract the semantic features of the first text based on the text features of the first text. By stacking multiple transformers to form a text semantics extraction unit, the accuracy of extracting the semantic features of the first text can be improved.

[0047] In one embodiment, extracting semantic features of the first text through a speech synthesis model includes:

[0048] Acquire text features of the first text, text features of the second text, and semantic features of the first voice data; the second text is the text corresponding to the first voice data;

[0049] The semantic features of the first text are extracted by a text semantic extraction unit in the speech synthesis model based on the text features of the first text, the text features of the second text and the semantic features of the first speech data.

[0050] The present embodiment is different from the above-mentioned embodiment in that not only the first voice data with the first timbre is acquired, but also the text corresponding to the first voice data, ie, the second text, is acquired. Figure 3 A schematic diagram of the speech synthesis principle of a speech synthesis model provided in another embodiment of the present disclosure is shown in FIG. Figure 3As shown, the speech synthesis model includes a text feature extraction unit, a text semantic extraction unit and a speech semantic extraction unit for speech data. The text feature extraction unit can be any neural network that can extract text features. The text semantic extraction unit can include an autoregressive model and can include a neural network formed by stacking multiple transformers. The speech semantic extraction unit can include a self-supervised network.

[0051] In this embodiment, the first text is input into the text feature extraction unit, and the text feature of the first text is extracted by the text feature extraction unit. The second text is input into the text feature extraction unit, and the text feature of the second text is extracted by the text feature extraction unit. The first voice data is input into the voice semantic extraction unit for voice data, and the semantic feature of the first voice data is extracted by the voice semantic extraction unit. The semantic feature of the first text is extracted by the text semantic extraction unit based on the text feature of the first text, the text feature of the second text, and the semantic feature of the first voice data.

[0052] The text features of the first text can be represented in the form of a matrix, for example, by a matrix of size d1*d2, where d1 represents the length of the text feature, d2 represents the dimension of the text feature, and the data of all dimensions corresponding to any length in the matrix represent the features of one or more characters in the first text. The text features of the second text can be represented in the form of a matrix, for example, by a matrix of size e1*e2, where e1 represents the length of the text feature, e2 represents the dimension of the text feature, and the data of all dimensions corresponding to any length in the matrix represent the features of one or more characters in the second text. The semantic features of the first speech data can also be represented by a matrix of a certain length and a certain dimension. The semantic features of the first speech data are at least used to represent the semantic information of the text corresponding to the first speech data, that is, the semantic information of the second text, and the semantic information of the second text at least includes the text content of the second text.

[0053] In this embodiment, the text semantic extraction unit can accurately extract the semantic features of the text based on the input text features, such as Figure 3 As shown, when the input of the text semantic feature extraction unit includes the text features of the first text and the text features of the second text, the text semantic extraction unit can predict the semantic features of the first text and the semantic features of the second text. Since the input of the text semantic extraction unit also includes the semantic features of the first voice data, the semantic features of the first voice data are equivalent to the semantic features of the second text. Therefore, the text semantic extraction unit focuses on predicting the semantic features of the first text through feature alignment, thereby achieving the effect of extracting the semantic features of the first text.

[0054] Moreover, through this embodiment, the input of the text semantic extraction unit includes the text features of the second text and the semantic features of the first voice data. The semantic features of the first voice data are equivalent to the semantic features of the second text. On the one hand, the text semantic extraction unit can learn the association between the text features and the corresponding accurate semantic features based on the text features of the input second text and the semantic features of the first voice data, and accurately predict the semantic features of the first text based on the association.

[0055] On the other hand, the text semantic extraction unit can learn the semantic features of the second text when the first timbre expresses the second text based on the text features of the input second text and the semantic features of the first voice data, so that the text semantic extraction unit can learn the semantic features of the first text when the first timbre expresses the first text based on the semantic features of the second text when the first timbre expresses the second text, thereby improving the accuracy of extracting the semantic features of the first text.

[0056] It can be seen that through this embodiment, the text semantic extraction unit can accurately extract the semantic features of the first text based on the text features of the first text, the text features of the second text and the semantic features of the first voice data.

[0057] In one embodiment, generating a first acoustic feature according to a semantic feature of a first text and first speech data by a speech synthesis model includes:

[0058] Acquire a timbre feature of a first timbre according to the first voice data;

[0059] The first acoustic feature is obtained by performing feature concatenation and feature conversion on the timbre feature of the first timbre, the semantic feature, the time feature and the first noise feature of the first text through the acoustic unit in the speech synthesis model.

[0060] like Figure 2 and Figure 3 As shown, when the speech synthesis model includes or does not include a speech semantic extraction unit for speech data, the speech synthesis model also includes a timbre extraction unit and an acoustic unit. The timbre extraction unit may include a neural network of any structure capable of extracting timbre features in speech data, such as a multi-layer convolutional neural network. The acoustic unit may include a Diffusion model, which may adopt a Unet structure or a Dual-path diffusion structure.

[0061] In this embodiment, the first voice data is input into the timbre extraction unit, and the timbre extraction unit extracts the timbre features of the first timbre based on the first voice data. The timbre features can also be represented in the form of a matrix with a certain length and a certain dimension. When the length is 1, the timbre features can be considered as global timbre features. In this embodiment, the first noise feature is also obtained, and the first noise feature can include the noise feature of standard Gaussian noise. In this embodiment, the semantic features of the first text are exemplified as the matrix [T1, C], the time features, the first noise features, and the timbre features of the first timbre are exemplified as [1, C], respectively, that is, the length of these three features is 1 and the dimension is C.

[0062] Next, the timbre feature of the first timbre, the semantic feature of the first text, the time feature and the first noise feature are input into the acoustic unit. The acoustic unit upsamples the semantic feature [T1, C] of the first text to obtain the feature [T2, C]. The upsampling process can extend the length of the semantic feature, and the extended length T2 is equal to the length T2 of the first acoustic feature. The acoustic unit also extends the length of the time feature, the length of the timbre feature and the length of the first noise feature to the length of the first acoustic feature, and obtains the extended time feature [T2, C], the extended timbre feature [T2, C] and the extended first noise feature [T2, C]. The acoustic unit also concatenates the semantic feature, the extended timbre feature, the extended time feature and the extended first noise feature in terms of dimension, and obtains the concatenated feature [T2, 4C]. The acoustic unit also performs feature conversion on the concatenated feature [T2, 4C] through a plurality of conformer blocks connected in sequence, and obtains the first acoustic feature [T2, 4C].

[0063] It can be seen that through this embodiment, the timbre features of the first timbre can be obtained according to the first speech data, and the timbre features of the first timbre, the semantic features of the first text, the time features and the first noise features can be feature concatenated and feature converted through the acoustic unit in the speech synthesis model to obtain the first acoustic features that can represent the semantic information of the first text and the timbre information of the first timbre. Setting the acoustic unit to a Diffusion model can utilize the Diffusion model's more powerful imagination and expression capabilities to better reconstruct the acoustic features based on the timbre features and semantic features, so that the acoustic features can better express the semantic information of the first text and the timbre information of the first timbre.

[0064] In one embodiment, generating second speech data having a first timbre according to a first acoustic feature through a speech synthesis model includes:

[0065] The first acoustic feature is decoded by a decoding unit in the speech synthesis model to obtain second speech data with a first timbre, and the text corresponding to the second speech data is the first text.

[0066] like Figure 2 and Figure 3 As shown, when the speech synthesis model includes or does not include a speech semantic extraction unit for speech data, the speech synthesis model also includes a decoding unit. The decoding unit may include a decoder decoding network of a variational auto-encoder (VAE). The first acoustic feature is input to the decoding unit, and the decoding unit decodes the first acoustic feature to obtain second speech data with a first timbre, and the text corresponding to the second speech data is the first text.

[0067] It can be seen that through this embodiment, the decoding unit can be used to decode the first acoustic feature, and the second voice data with the first timbre can be accurately and quickly obtained, thereby improving the efficiency of synthesizing the second voice data.

[0068] In one embodiment, reference Figure 2 As shown, the text semantic extraction unit in the speech synthesis model extracts the semantic features of the first text based on the text features of the first text, including:

[0069] By means of a text semantic extraction unit, an autoregressive process is performed on a text feature of the first text to obtain a first feature; a feature length of the first feature is greater than a feature length of the text feature of the first text;

[0070] The first feature is compared with the text feature of the first text according to the feature length to obtain redundant features in the first feature, and the redundant features in the first feature are used as semantic features of the first text.

[0071] In this embodiment, the text semantic extraction unit may include an autoregressive model, such as Figure 2 As shown, the text features of the first text are input into the text semantic extraction unit, and the text semantic extraction unit performs autoregressive processing on the text features of the first text. Autoregressive processing refers to splicing the data in the output features of the text semantic extraction unit back into the input features of the text semantic extraction unit for cyclic processing. In this embodiment, the length and dimension of the input features of the text semantic extraction unit are the same as the length and dimension of the output features of the text semantic extraction unit. On this basis, the text features of the first text are input into the text semantic extraction unit, and the text semantic extraction unit processes the features to obtain outputs, and the data of each dimension corresponding to the last length in the output features of the text semantic extraction unit are spliced ​​back into the input features of the text semantic extraction unit, and the spliced ​​features are processed by the text semantic extraction unit. After repeating the above processing method several times, that is, after the text semantic extraction unit processes the input features several times, the output features obtained are the first features.

[0072] Figure 4 A schematic diagram of the principle of performing autoregressive processing on text features of a first text provided by an embodiment of the present disclosure, such as Figure 4 As shown, in one example, the text feature of the first text is a matrix of size [n, m], where n is the feature length and m is the feature dimension. The text semantic extraction unit can process the text feature of the first text to obtain a matrix of size [n, m]. The data of each dimension corresponding to the last length in the matrix of size [n, m] is spliced ​​to the last bit of the text feature of the input first text to obtain the spliced ​​feature, that is, a matrix of size [n+1, m]. The spliced ​​feature, that is, the matrix of size [n+1, m] is input to the text semantic extraction unit for further processing to obtain a matrix of size [n+1, m]. The data of each dimension corresponding to the last length in the matrix of size [n+1, m] is spliced ​​to the last bit of the feature last input to the text semantic extraction unit to obtain the spliced ​​feature, that is, a matrix of size [n+2, m]. The spliced ​​feature, that is, the matrix of size [n+2, m] is input to the text semantic extraction unit for further processing. The above process is repeated several times, and the output obtained is the first feature.

[0073] It can be seen from the process of autoregression processing of the text features of the first text that the feature length of the first feature will be greater than the feature length of the text features of the first text. In this embodiment, the text semantic extraction unit compares the first feature with the text features of the first text according to the feature length to obtain redundant features in the first feature, and the redundant features are the features corresponding to the extra length of the first feature relative to the text features of the first text after aligning the first feature and the text features of the first text according to the feature length. For example, if the first feature is a matrix of size [n+20,m] and the text features of the first text are a matrix of size [n,m], then the redundant features are the last 20 features in the length direction of the first feature. The redundant features in the first feature are used as the semantic features of the first text.

[0074] It can be seen that through this embodiment, the text features of the first text can be processed by the text semantic extraction unit through autoregression, and the semantic features of the first text can be accurately obtained through the autoregression processing.

[0075] In one embodiment, reference Figure 3 As shown, the text semantic extraction unit in the speech synthesis model extracts the semantic features of the first text based on the text features of the first text, the text features of the second text and the semantic features of the first speech data, including:

[0076] Splicing the text features of the first text, the text features of the second text, and the semantic features of the first voice data to obtain a first splicing feature;

[0077] The first concatenated feature is subjected to autoregressive processing by a text semantic extraction unit to obtain a second feature; the feature length of the second feature is greater than the feature length of the first concatenated feature;

[0078] The second feature is compared with the first concatenated feature according to feature length to obtain redundant features in the second feature, and the redundant features in the second feature are used as semantic features of the first text.

[0079] In this embodiment, the text semantic extraction unit may include an autoregressive model, such as Figure 3 As shown, the text features of the first text, the text features of the second text and the semantic features of the first speech data are spliced ​​to obtain a first spliced ​​feature, and the first spliced ​​feature is input into a text semantic extraction unit, and the text semantic extraction unit performs autoregressive processing on the first spliced ​​feature. Autoregressive processing refers to splicing the data in the output features of the text semantic extraction unit back into the input features of the text semantic extraction unit for cyclic processing. In this embodiment, the length and dimension of the input features of the text semantic extraction unit are the same as the length and dimension of the output features of the text semantic extraction unit. On this basis, the first spliced ​​feature is input into the text semantic extraction unit, and the text semantic extraction unit processes the feature to obtain an output, and the data of each dimension corresponding to the last length in the output features of the text semantic extraction unit is spliced ​​back into the input features of the text semantic extraction unit, and the spliced ​​features are processed by the text semantic extraction unit. After repeating the above processing method several times, that is, after the text semantic extraction unit processes the input features several times, the output feature obtained is the second feature.

[0080] Figure 5 A schematic diagram of the principle of performing autoregressive processing on the first splicing feature provided by another embodiment of the present disclosure is as follows: Figure 5As shown, in one example, the text features of the first text, the text features of the second text, and the semantic features of the first speech data are spliced ​​to obtain a first spliced ​​feature, the first spliced ​​feature is a matrix of size [p, q], where p is the feature length and q is the feature dimension. The text semantic extraction unit can process the first spliced ​​feature to obtain a matrix of size [p, q], and splice the data of each dimension corresponding to the last length in the matrix of size [p, q] to the last bit of the input first spliced ​​feature to obtain the spliced ​​feature, that is, a matrix of size [p+1, q], and then The concatenated features, i.e., a matrix of size [p+1,q], are input into the text semantic extraction unit for further processing to obtain a matrix of size [p+1,q]. The data of each dimension corresponding to the last length in the matrix of size [p+1,q] are concatenated to the last bit of the features previously input into the text semantic extraction unit to obtain the concatenated features, i.e., a matrix of size [p+2,q]. The concatenated features, i.e., a matrix of size [p+2,q], are input into the text semantic extraction unit for further processing. The above process is repeated several times, and the output obtained is the second feature.

[0081] It can be seen from the process of autoregression processing of the first splicing feature that the feature length of the second feature will be greater than the feature length of the first splicing feature. In this embodiment, the second feature is compared with the first splicing feature according to the feature length by the text semantic extraction unit to obtain redundant features in the second feature. The redundant feature is the feature corresponding to the extra length of the second feature relative to the first splicing feature after the second feature and the first splicing feature are aligned according to the feature length. For example, the second feature is a matrix of size [p+20,q], and the first splicing is a matrix of size [p,q], then the redundant feature is the last 20 features in the length direction of the second feature. The redundant features in the second feature are used as the semantic features of the first text.

[0082] It can be seen that through this embodiment, the text features of the first text, the text features of the second text and the semantic features of the first speech data can be spliced ​​to obtain a first spliced ​​feature, and the first spliced ​​feature is autoregressively processed by a text semantic extraction unit. Through the autoregressive processing, the semantic features of the first text can be accurately obtained.

[0083] The above describes the process of synthesizing the second speech data through the speech synthesis model. The following describes the training process of the speech synthesis model. Figure 2 and Figure 3 It can be seen that the speech synthesis model at least includes a text feature extraction unit, a text semantic extraction unit, a timbre extraction unit, an acoustic unit and a decoding unit, and may also include or not include a speech semantic extraction unit.

[0084] Figure 6a A schematic diagram of the training principle of a speech synthesis model provided in an embodiment of the present disclosure, Figure 6b A schematic diagram of the training principle of a speech synthesis model provided in another embodiment of the present disclosure is shown below in conjunction with Figure 6a and Figure 6b This section describes the training process of the speech synthesis model in detail.

[0085] The text feature extraction unit in the speech synthesis model can be trained in a general manner, which is not limited in this embodiment. For the speech semantics extraction unit in the speech synthesis model, the speech semantics extraction unit can include a self-supervisory model, which can be a commonly used self-supervisory model such as Hubert and bestRQ. The speech semantics extraction unit can also be trained by a general training method, for example, based on a large amount of speech data and the semantic features corresponding to the speech data. In this embodiment, the speech semantics extraction unit can output continuous features as semantic features, or it can output discrete features, and the discrete features are clustered to form semantic features.

[0086] like Figure 6a As shown, after the speech semantic extraction unit and the text feature extraction unit are trained, the text semantic extraction unit can be trained based on the trained speech semantic extraction unit. In one embodiment, the above method flow also includes:

[0087] Acquire first sample voice data and sample text corresponding to the first sample voice data;

[0088] Extracting semantic features of the first sample speech data and text features of the sample text;

[0089] A text semantic extraction unit is trained according to the semantic features of the first sample speech data and the text features of the sample text.

[0090] In this embodiment, a plurality of first sample voice data and a sample text corresponding to each first sample voice data are obtained. The plurality of first sample voice data may include voice data of different timbres. Next, the semantic features of each first sample voice data are extracted by a trained voice semantic extraction unit, and the semantic features of the first sample voice data are equivalent to the semantic features of the sample text. The text features of each trained sample text are extracted by a trained text feature extraction unit. Finally, according to the semantic features of the first sample voice data and the text features of the sample text, the text semantic extraction unit is trained so that the text semantic extraction unit learns the ability to extract the semantic features of the text.

[0091] It can be seen that through this embodiment, the text semantic extraction unit can be accurately and efficiently trained according to the semantic features of the first sample speech data and the text features of the sample text.

[0092] like Figure 6a As shown, in one embodiment, training a text semantic extraction unit according to the semantic features of the first sample speech data and the text features of the sample text includes:

[0093] Splicing the semantic features of the first sample speech data and the text features of the sample text to obtain a second splicing feature;

[0094] The second concatenated feature is processed based on the attention mechanism through the text semantic extraction unit to obtain a third feature; the feature length of the third feature is equal to the feature length of the second concatenated feature;

[0095] The text semantic extraction unit is trained according to the semantic feature of the first sample speech data and the feature in the third feature corresponding to the semantic feature of the first sample speech data.

[0096] In this embodiment, when training the text semantic extraction unit, the semantic features of the first sample speech data and the text features of the sample text are first concatenated in terms of feature length to obtain a second concatenated feature, and the second concatenated feature is input into the text semantic extraction unit. The text semantic extraction unit can be an autoregressive model based on a transformer network architecture. The text semantic extraction unit processes the second concatenated feature based on an attention mechanism, such as a self-attention mechanism, to obtain a third feature. After processing based on the attention mechanism, the feature length of the third feature is equal to the feature length of the second concatenated feature, and the feature dimension of the third feature is equal to the feature dimension of the second concatenated feature.

[0097] Determine in the third feature the feature corresponding to the semantic feature of the first sample speech data. In a specific example, assume that the semantic feature of the first sample speech data is a feature in the matrix form [r1, r2], r1 is the feature length and r2 is the feature dimension, and the text feature of the sample text is a feature in the matrix form [r3, r2], r3 is the feature length and r2 is the feature dimension. Then, the semantic feature of the first sample speech data and the text feature of the sample text are concatenated in terms of feature length, and the obtained second concatenated feature is a feature in the matrix form [r1+r3, r2]. The text semantic extraction unit processes the second concatenated feature based on the attention mechanism, and the obtained third feature is also a feature in the matrix form [r1+r3, r2].

[0098] In this embodiment, in order to enable the text semantic extraction unit to have the ability to predict the semantic features of the text, the text semantic extraction unit can be trained by the next-token prediction method, and the data of the features from r1-1 to r1+r3-1 in length are extracted from the third feature. These data can also form a feature in the form of a matrix [r3, r2], which is the feature in the third feature corresponding to the semantic feature of the first sample voice data. According to the difference between the feature and the semantic feature of the first sample voice data, the text semantic extraction unit is fed back for training. When training the text semantic extraction unit, if the text semantic extraction unit outputs a discrete feature, the loss function can select the cross-entropy loss, and if the output is a continuous feature, the loss function can select the MAE loss. From this training process, it can be seen that since the data of the features from r1-1 to r1+r3-1 in length are extracted from the third feature and compared with the semantic features of the first sample voice data, so that the difference between the two is minimized, the trained text semantic extraction unit has the effect of predicting the semantic features of the next moment, and the length concept in the semantic feature can be understood as the moment concept.

[0099] It can be seen that through this embodiment, the semantic features of the first sample speech data and the text features of the sample text can be spliced ​​to obtain a second spliced ​​feature, and the second spliced ​​feature can be processed by a text semantic extraction unit to obtain a third feature. Based on the semantic features of the first sample speech data and the features in the third feature that correspond to the semantic features of the first sample speech data, the efficiency and accuracy of training the text semantic extraction unit can be improved.

[0100] This embodiment includes a first training process and a second training process. The first training process is as follows: Figure 6a As shown: first train the text feature extraction unit and the speech semantic extraction unit, and after these two units are trained, train the text semantic extraction unit based on the trained two units. The second training process is: before training the timbre extraction unit and the acoustic unit, it is necessary to first train the encoding unit and the decoding unit, and then train the timbre extraction unit and the acoustic unit based on the trained encoding unit and the speech semantic extraction unit. The above introduction completes the first training process, and the second training process is introduced below.

[0101] In the second training process, the encoding unit and the decoding unit are trained first. In one embodiment, the encoding unit is configured to include an encoder encoding network of a variational autoencoder, and the decoding unit is configured to include a decoder decoding network of the variational autoencoder. Figure 7 A schematic diagram of the training principle of the encoding unit and the decoding unit provided in an embodiment of the present disclosure is shown in FIG. Figure 7As shown, the encoding unit and the decoding unit are trained by the speech data and the acoustic features corresponding to the speech data, so that the encoding unit has the ability to extract the acoustic features of the speech data, and the decoding unit has the ability to restore the speech data according to the acoustic features. When training the encoding unit and the decoding unit, in addition to the training method of the variational autoencoder, the discriminator network Discriminator can also be introduced. The discriminator network Discriminator, the encoding unit and the decoding unit are adversarially trained through the adversarial loss function. The adversarial training method can refer to the training method of HifiGAN.

[0102] After the encoding unit and the decoding unit are obtained through training, the above method flow further includes:

[0103] Acquire second sample voice data;

[0104] Extracting the acoustic features of the second sample speech data and the semantic features of the second sample speech data; the acoustic features of the second sample speech data may be referred to as second acoustic features, which are at least used to represent the semantic information of the second sample speech data and the timbre information of the second sample speech data;

[0105] The timbre extraction unit and the acoustic unit are trained according to the second sample speech data, the semantic feature of the second sample speech data, the second acoustic feature, the time feature and the second noise feature.

[0106] like Figure 6b As shown, in this embodiment, a plurality of second sample speech data are obtained, and the second sample speech data may be the same as or different from the first sample speech data. Then, the acoustic features of the second sample speech data are extracted as the second acoustic features through the previously trained encoding unit, and the semantic features of the second sample speech data are extracted through the previously trained speech semantic extraction unit, wherein the second acoustic features are at least used to represent the semantic information of the second sample speech data and the timbre information of the second sample speech data, and the semantic information of the second sample speech data at least includes the text content of the text corresponding to the second sample speech data. Finally, the timbre extraction unit and the acoustic unit are trained according to the second sample speech data, the semantic features of the second sample speech data, the second acoustic features, the time features, and the second noise features.

[0107] It can be seen that through this embodiment, the second acoustic features and the semantic features of the second sample speech data can be extracted, and the timbre extraction unit and the acoustic unit can be trained efficiently and accurately based on the second sample speech data, the semantic features of the second sample speech data, the second acoustic features, the time features and the second noise features.

[0108] Based on the aforementioned process of preferentially training the encoding unit and the decoding unit, in one embodiment, the above method flow further includes:

[0109] Train the encoding unit and decoding unit;

[0110] extracting a second acoustic feature through an encoding unit;

[0111] The decoding unit is used to generate second voice data with a first timbre according to the first acoustic feature, and the text corresponding to the second voice data is the first text.

[0112] According to the above introduction, it is necessary to first train the encoding unit and the decoding unit, and use the encoding unit to extract the second acoustic feature to train the timbre extraction unit and the acoustic unit, wherein the decoding unit is Figure 2 Figure 3 The decoding unit in can decode the first acoustic feature to obtain second voice data with the first timbre, and the text corresponding to the second voice data is the first text.

[0113] It can be seen that through this embodiment, the encoding unit and the decoding unit can be preferentially trained, so that the encoding unit has the ability to generate acoustic features, and the decoding unit has the ability to decode acoustic features, thereby further training the acoustic unit and the timbre extraction unit to improve the training accuracy of the model.

[0114] In one embodiment, training the timbre extraction unit and the acoustic unit according to the second sample speech data, the semantic feature of the second sample speech data, the second acoustic feature, the time feature and the second noise feature includes:

[0115] Predicting the timbre characteristics of the second sample speech data according to the second sample speech data by the timbre extraction unit to obtain a first prediction result;

[0116] Predicting the second acoustic feature according to the first prediction result, the semantic feature of the second sample speech data, the second acoustic feature, the time feature, and the second noise feature by the acoustic unit to obtain a second prediction result;

[0117] The timbre extraction unit and the acoustic unit are trained according to the second prediction result and the second acoustic feature.

[0118] In this embodiment, the timbre extraction unit may include a neural network of any structure capable of extracting timbre features from speech data, such as a multi-layer convolutional neural network. The acoustic unit may include a diffusion model, which may adopt a Unet structure or a dual-path diffusion structure. Figure 6b As shown, the timbre extraction unit and the acoustic unit are trained synchronously. During the training, the second sample speech data is input into the timbre extraction unit, and the timbre extraction unit predicts the timbre characteristics of the second sample speech data to obtain a first prediction result.

[0119] Next, the second noise feature is superimposed on the second acoustic feature, and the second noise feature can be a noise feature of Gaussian noise with random amplitude. The first prediction result, the semantic feature of the second sample speech data, the second acoustic feature after superimposed noise, and the time feature are input into the acoustic unit. The acoustic unit performs feature splicing on the first prediction result, the semantic feature of the second sample speech data, the second acoustic feature after superimposed noise, and the time feature, and performs feature conversion on the spliced ​​features through multiple conformer blocks connected in sequence, predicts the second acoustic feature, and obtains the second prediction result.

[0120] Finally, the timbre extraction unit and the acoustic unit are fed back for training according to the difference between the second prediction result and the second acoustic feature.

[0121] It can be seen that through this embodiment, the timbre extraction unit and the acoustic unit can be synchronously trained according to the second sample speech data, the semantic features of the second sample speech data, the second acoustic features, the time features and the second noise features, thereby improving the model training efficiency.

[0122] In summary, through the speech synthesis method provided above:

[0123] 1. It can improve the pronunciation accuracy of synthesized speech data and avoid misreading of text;

[0124] 2. Support speech synthesis in two scenarios. When the first text and the first voice data are obtained, speech synthesis can be performed. When the first text, the first voice data and the second text corresponding to the first voice data are obtained, speech synthesis can also be performed, thereby expanding the applicable scenarios of speech synthesis.

[0125] Figure 8 A schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the device comprises:

[0126] A semantic extraction module 81, used to obtain a first text and extract semantic features of the first text through a speech synthesis model;

[0127] The acoustic generation module 82 is used to obtain first speech data with a first timbre, and generate a first acoustic feature according to the semantic feature of the first text and the first speech data through the speech synthesis model; the first acoustic feature is used to represent at least the semantic information of the first text and the timbre information of the first timbre;

[0128] The speech generation module 83 is used to generate second speech data having the first timbre according to the first acoustic feature through the speech synthesis model; the text corresponding to the second speech data is the first text.

[0129] Optionally, the semantic extraction module 81 is specifically used for:

[0130] Acquire text features of the first text;

[0131] The semantic features of the first text are extracted based on the text features of the first text by the text semantic extraction unit in the speech synthesis model.

[0132] Optionally, the semantic extraction module 81 is specifically used for:

[0133] Acquire text features of the first text, text features of the second text, and semantic features of the first voice data; the second text is the text corresponding to the first voice data;

[0134] The semantic features of the first text are extracted by the text semantic extraction unit in the speech synthesis model based on the text features of the first text, the text features of the second text and the semantic features of the first speech data.

[0135] Optionally, the acoustic generation module 82 is specifically used for:

[0136] Acquire a timbre feature of the first timbre according to the first voice data;

[0137] The first acoustic feature is obtained by performing feature concatenation and feature conversion on the timbre feature of the first timbre, the semantic feature, the time feature and the first noise feature of the first text through the acoustic unit in the speech synthesis model.

[0138] Optionally, the speech generation module 83 is specifically used for:

[0139] The first acoustic feature is decoded by a decoding unit in the speech synthesis model to obtain the second speech data having the first timbre.

[0140] Optionally, the semantic extraction module 81 is further specifically used for:

[0141] By means of the text semantic extraction unit, an autoregressive process is performed on the text feature of the first text to obtain a first feature; the feature length of the first feature is greater than the feature length of the text feature of the first text;

[0142] The first feature is compared with the text feature of the first text according to feature length to obtain redundant features in the first feature, and the redundant features in the first feature are used as semantic features of the first text.

[0143] Optionally, the semantic extraction module 81 is further specifically used for:

[0144] splicing the text features of the first text, the text features of the second text, and the semantic features of the first voice data to obtain a first splicing feature;

[0145] The text semantic extraction unit performs autoregressive processing on the first splicing feature to obtain a second feature; the feature length of the second feature is greater than the feature length of the first splicing feature;

[0146] The second feature is compared with the first concatenated feature according to feature length to obtain redundant features in the second feature, and the redundant features in the second feature are used as semantic features of the first text.

[0147] Optionally, the method further includes a first training unit, configured to:

[0148] Acquire first sample voice data and a sample text corresponding to the first sample voice data;

[0149] Extracting semantic features of the first sample speech data and text features of the sample text;

[0150] The text semantic extraction unit is trained according to the semantic features of the first sample speech data and the text features of the sample text.

[0151] Optionally, the first training unit is specifically used for:

[0152] splicing the semantic features of the first sample speech data and the text features of the sample text to obtain a second splicing feature;

[0153] The text semantic extraction unit processes the second splicing feature based on an attention mechanism to obtain a third feature; the feature length of the third feature is equal to the feature length of the second splicing feature;

[0154] The text semantic extraction unit is trained according to the semantic feature of the first sample speech data and a feature among the third features corresponding to the semantic feature of the first sample speech data.

[0155] Optionally, the method further includes a second training unit, configured to:

[0156] Acquire second sample voice data;

[0157] Extracting a second acoustic feature of the second sample speech data and a semantic feature of the second sample speech data; the second acoustic feature is at least used to represent the semantic information of the second sample speech data and the timbre information of the second sample speech data;

[0158] The timbre extraction unit and the acoustic unit are trained according to the second sample speech data, the semantic features of the second sample speech data, the second acoustic features, the time features and the second noise features.

[0159] Optionally, a third training unit is further included, configured to:

[0160] Train the encoding unit and decoding unit;

[0161] extracting the second acoustic feature by the encoding unit;

[0162] The decoding unit is used to generate the second speech data having the first timbre according to the first acoustic feature.

[0163] Optionally, the second training unit is specifically used for:

[0164] Predicting the timbre characteristics of the second sample voice data according to the second sample voice data by the timbre extraction unit to obtain a first prediction result;

[0165] Predicting the second acoustic feature by the acoustic unit according to the first prediction result, the semantic feature of the second sample speech data, the second acoustic feature, the time feature and the second noise feature to obtain a second prediction result;

[0166] The timbre extraction unit and the acoustic unit are trained according to the second prediction result and the second acoustic feature.

[0167] The speech synthesis device in the embodiment of the present disclosure can implement each process of the above-mentioned speech synthesis method embodiment and achieve the same effect and function, which will not be repeated here.

[0168] An embodiment of the present disclosure further provides an electronic device, Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, such as Fig. 9As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 901 and memory 902, and one or more applications or data may be stored in the memory 902. Among them, the memory 902 may be a temporary storage or a permanent storage. The application stored in the memory 902 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 901 can be configured to communicate with the memory 902 to execute a series of computer executable instructions in the memory 902 on the electronic device. The electronic device may also include one or more power supplies 903, one or more wired or wireless network interfaces 904, one or more input or output interfaces 905, one or more keyboards 906, etc.

[0169] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the following process:

[0170] Acquire a first text, and extract semantic features of the first text through a speech synthesis model;

[0171] Acquire first speech data with a first timbre, and generate a first acoustic feature according to the semantic feature of the first text and the first speech data through the speech synthesis model; the first acoustic feature is used to at least represent the semantic information of the first text and the timbre information of the first timbre;

[0172] Through the speech synthesis model, second speech data having the first timbre is generated according to the first acoustic feature; the text corresponding to the second speech data is the first text.

[0173] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned speech synthesis method embodiment and achieve the same effect and function, which will not be repeated here.

[0174] Another embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:

[0175] Acquire a first text, and extract semantic features of the first text through a speech synthesis model;

[0176] Acquire first speech data with a first timbre, and generate a first acoustic feature according to the semantic feature of the first text and the first speech data through the speech synthesis model; the first acoustic feature is used to at least represent the semantic information of the first text and the timbre information of the first timbre;

[0177] Through the speech synthesis model, second speech data having the first timbre is generated according to the first acoustic feature; the text corresponding to the second speech data is the first text.

[0178] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned speech synthesis method embodiment and achieve the same effect and function, which will not be repeated here.

[0179] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0180] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0181] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0182] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0183] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0184] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0188] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0189] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0190] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0191] The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A speech synthesis method, characterized in that: include: Acquire a first text, and extract semantic features of the first text through a speech synthesis model; Acquire first speech data with a first timbre, and generate a first acoustic feature according to the semantic feature of the first text and the first speech data through the speech synthesis model; The first acoustic feature is used to represent at least semantic information of the first text and timbre information of the first timbre; Generate second speech data having the first timbre according to the first acoustic feature by using the speech synthesis model; The text corresponding to the second voice data is the first text.

2. The method according to claim 1, characterized in that The extracting the semantic features of the first text by using a speech synthesis model includes: Acquire text features of the first text; The semantic features of the first text are extracted based on the text features of the first text by the text semantic extraction unit in the speech synthesis model.

3. The method according to claim 1, characterized in that: The extracting the semantic features of the first text by using a speech synthesis model includes: Acquire text features of the first text, text features of the second text, and semantic features of the first voice data; the second text is the text corresponding to the first voice data; The semantic features of the first text are extracted by the text semantic extraction unit in the speech synthesis model based on the text features of the first text, the text features of the second text and the semantic features of the first speech data.

4. The method according to claim 1, characterized in that: The step of generating a first acoustic feature according to the semantic feature of the first text and the first speech data by using the speech synthesis model includes: Acquire a timbre feature of the first timbre according to the first voice data; The first acoustic feature is obtained by performing feature concatenation and feature conversion on the timbre feature of the first timbre, the semantic feature, the time feature and the first noise feature of the first text through the acoustic unit in the speech synthesis model.

5. The method according to claim 1, characterized in that The step of generating second speech data having the first timbre according to the first acoustic feature by using the speech synthesis model includes: The first acoustic feature is decoded by a decoding unit in the speech synthesis model to obtain the second speech data having the first timbre.

6. The method according to claim 2, characterized in that The extracting the semantic features of the first text based on the text features of the first text by the text semantic extraction unit in the speech synthesis model includes: By means of the text semantic extraction unit, an autoregressive process is performed on the text feature of the first text to obtain a first feature; the feature length of the first feature is greater than the feature length of the text feature of the first text; The first feature is compared with the text feature of the first text according to feature length to obtain redundant features in the first feature, and the redundant features in the first feature are used as semantic features of the first text.

7. The method according to claim 3, characterized in that The extracting the semantic features of the first text based on the text features of the first text, the text features of the second text and the semantic features of the first speech data by the text semantic extraction unit in the speech synthesis model includes: splicing the text features of the first text, the text features of the second text, and the semantic features of the first voice data to obtain a first splicing feature; The text semantic extraction unit performs autoregressive processing on the first splicing feature to obtain a second feature; the feature length of the second feature is greater than the feature length of the first splicing feature; The second feature is compared with the first concatenated feature according to feature length to obtain redundant features in the second feature, and the redundant features in the second feature are used as semantic features of the first text.

8. The method according to claim 2 or 3, characterized in that: Also includes: Acquire first sample voice data and a sample text corresponding to the first sample voice data; Extracting semantic features of the first sample speech data and text features of the sample text; The text semantic extraction unit is trained according to the semantic features of the first sample speech data and the text features of the sample text.

9. The method according to claim 8, characterized in that The step of training the text semantic extraction unit according to the semantic features of the first sample speech data and the text features of the sample text comprises: splicing the semantic features of the first sample speech data and the text features of the sample text to obtain a second splicing feature; The text semantic extraction unit processes the second splicing feature based on an attention mechanism to obtain a third feature; the feature length of the third feature is equal to the feature length of the second splicing feature; The text semantic extraction unit is trained according to the semantic feature of the first sample speech data and a feature among the third features corresponding to the semantic feature of the first sample speech data.

10. The method according to claim 4, characterized in that Also includes: Acquire second sample voice data; Extracting a second acoustic feature of the second sample speech data and a semantic feature of the second sample speech data; The second acoustic feature is used to represent at least the semantic information of the second sample speech data and the timbre information of the second sample speech data; The timbre extraction unit and the acoustic unit are trained according to the second sample speech data, the semantic features of the second sample speech data, the second acoustic features, the time features and the second noise features.

11. The method according to claim 10, characterized in that The method further comprises: Train the encoding unit and decoding unit; extracting the second acoustic feature by the encoding unit; The decoding unit is used to generate the second speech data having the first timbre according to the first acoustic feature.

12. The method according to claim 10, characterized in that The training of the timbre extraction unit and the acoustic unit according to the second sample speech data, the semantic feature of the second sample speech data, the second acoustic feature, the time feature and the second noise feature comprises: Predicting the timbre characteristics of the second sample voice data according to the second sample voice data by the timbre extraction unit to obtain a first prediction result; Predicting the second acoustic feature by the acoustic unit according to the first prediction result, the semantic feature of the second sample speech data, the second acoustic feature, the time feature and the second noise feature to obtain a second prediction result; The timbre extraction unit and the acoustic unit are trained according to the second prediction result and the second acoustic feature.

13. A speech synthesis device, characterized in that: include: A semantic extraction module, used to obtain a first text and extract semantic features of the first text through a speech synthesis model; An acoustic generation module, configured to obtain first speech data having a first timbre, and generate a first acoustic feature according to the semantic feature of the first text and the first speech data through the speech synthesis model; The first acoustic feature is used to represent at least semantic information of the first text and timbre information of the first timbre; A speech generation module is used to generate second speech data having the first timbre according to the first acoustic feature through the speech synthesis model; the text corresponding to the second speech data is the first text.

14. An electronic device, characterized in that: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 12.

Images