A speech synthesis method and apparatus

Abstract

The embodiment of the present application discloses a speech synthesis method and device, including: obtaining text data and pitch data corresponding to the text data, where the text data contains multiple texts, and the pitch data represents the pitch corresponding to each text; encoding the text data and the pitch data to obtain text feature data and pitch feature data; performing duration prediction on the multiple texts to obtain a predicted duration of each text, where the predicted duration represents the number of frames corresponding to the text; merging the text feature data and the pitch feature data to obtain first feature data; using the predicted duration of each text to perform duration expansion on the first feature data to obtain second feature data; decoding the second feature data to obtain speech spectrum parameters, and inputting the speech spectrum parameters into a pre-trained neural network vocoder to generate and obtain the target speech.

Description

Technical Field

[0001] The present application relates to the technical fields of data processing, deep learning and speech synthesis, and in particular to a speech synthesis method and device. Background Art

[0002] Currently, there are two methods for synthesizing songs from text. One uses a hidden Markov model. However, because this model doesn't consider pitch characteristics, the resulting synthesized songs lack emotion. The other uses existing neural network models. However, because the model considers too few features, it can overfit the model, making the synthesized songs less stable and realistic. Summary of the Invention

[0003] The present application provides a speech synthesis method and device to solve the above-mentioned technical problems.

[0004] To this end, an embodiment of the present application provides a speech synthesis method, the method comprising:

[0005] Acquire text data and pitch data corresponding to the text data, wherein the text data includes a plurality of texts, and the pitch data represents the pitch corresponding to each text;

[0006] Encoding the text data and the pitch data to obtain text feature data and pitch feature data;

[0007] Performing duration prediction on the multiple texts to obtain a predicted duration of each text, where the predicted duration represents the number of frames corresponding to the text;

[0008] Merging the text feature data and the pitch feature data to obtain first feature data;

[0009] Extending the first feature data by using the predicted duration of each text to obtain second feature data;

[0010] The second feature data is decoded to obtain speech spectrum parameters, and the speech spectrum parameters are input into a pre-trained neural network vocoder to generate and obtain the target speech.

[0011] The merging of the text feature data and the pitch feature data comprises:

[0012] Determining text sub-feature data and pitch sub-feature data corresponding to each text from the text feature data and pitch feature data;

[0013] Determining first sub-feature data of the text based on the text sub-feature data and the pitch sub-feature data of the text;

[0014] All first sub-feature data are merged according to the order of corresponding texts in the text data to obtain first feature data.

[0015] The method of using the predicted duration of each text to extend the duration of the first feature data to obtain the second feature data includes:

[0016] Determining first sub-feature data corresponding to each text from the first feature data;

[0017] According to the predicted duration of the text, the first sub-feature data corresponding to the text is expanded to the number of frames indicated by the predicted duration to obtain the second sub-feature data;

[0018] The second sub-feature data are merged according to the order of the corresponding texts in the text data to obtain second feature data.

[0019] After obtaining the second characteristic data, the method further includes:

[0020] Determining second sub-feature data corresponding to each text from the second feature data;

[0021] Traverse all second sub-feature data;

[0022] Determining a similarity between the current second sub-feature data and each other second sub-feature data, and determining a weight according to the similarity;

[0023] Adjusting the current second sub-feature data according to the current second sub-feature data, all other second sub-feature data, and the weight of each of the other second sub-feature data of the current second sub-feature data;

[0024] After traversing all the second sub-feature data, all the adjusted second sub-feature data are merged according to the order of the corresponding texts in the text data to obtain the second feature data.

[0025] Another aspect of the present invention provides a method for training a speech synthesis model, the method comprising:

[0026] Obtain multiple text sample data and label speech spectrum parameters corresponding to each text sample data;

[0027] Inputting the text sample data into an initial speech synthesis model to obtain predicted feature data of the text sample data;

[0028] Determine the loss value of the text sample data according to the label speech spectrum parameters and the predicted speech spectrum parameters of the text sample data;

[0029] The initial speech synthesis model is optimized according to the loss values of the multiple text sample data to obtain a speech synthesis model.

[0030] Another aspect of the present invention provides a speech synthesis device, comprising:

[0031] A first acquisition module is configured to acquire text data and pitch data corresponding to the text data, wherein the text data includes a plurality of texts, and the pitch data represents the pitch corresponding to each text;

[0032] An encoding module, configured to encode the text data and the pitch data to obtain text feature data and pitch feature data;

[0033] A first deep learning module is configured to perform duration prediction on the plurality of texts to obtain a predicted duration of each text, where the predicted duration represents the number of frames corresponding to the text;

[0034] a calculation module, configured to combine the text feature data and the pitch feature data to obtain first feature data;

[0035] The first deep learning module is further configured to perform duration expansion on the first feature data using the predicted duration of each text to obtain second feature data;

[0036] A decoding module is used to decode the second feature data to obtain speech spectrum parameters, and input the speech spectrum parameters into a pre-trained neural network vocoder to generate and obtain the target speech.

[0037] The calculation module is further configured to determine text sub-feature data and pitch sub-feature data corresponding to each text from the text feature data and pitch feature data;

[0038] The calculation module is further configured to determine first sub-feature data of the text based on the text sub-feature data and the pitch sub-feature data of the text;

[0039] The calculation module is further configured to merge all first sub-feature data according to the order of corresponding texts in the text data to obtain first feature data.

[0040] The first deep learning module is further configured to determine first sub-feature data corresponding to each text from the first feature data;

[0041] The first deep learning module is further configured to expand the first sub-feature data corresponding to the text to a number of frames indicated by the predicted duration according to the predicted duration, to obtain second sub-feature data;

[0042] The first deep learning module is further configured to merge the second sub-feature data according to the order of the corresponding texts in the text data to obtain second feature data.

[0043] The first deep learning module is further configured to determine second sub-feature data corresponding to each text from the second feature data;

[0044] The first deep learning module is further configured to traverse all second sub-feature data;

[0045] The first deep learning module is further configured to determine a similarity between the current second sub-feature data and each other second sub-feature data, and determine a weight based on the similarity;

[0046] The first deep learning module is further configured to adjust the current second sub-feature data according to the current second sub-feature data, all other second sub-feature data, and the weight of each of the other second sub-feature data of the current second sub-feature data;

[0047] The first deep learning module is further configured to, after traversing all the second sub-feature data, merge all the adjusted second sub-feature data in the order of the corresponding texts in the text data to obtain the second feature data.

[0048] Another aspect of the present invention provides a speech synthesis model training device, comprising:

[0049] The second acquisition module is used to obtain a plurality of text sample data and a label speech spectrum parameter corresponding to each text sample data;

[0050] A second deep learning module is used to input the text sample data into an initial speech synthesis model to obtain predicted feature data of the text sample data;

[0051] The second deep learning module is further used to determine the loss value of the text sample data based on the label speech spectrum parameters and the predicted speech spectrum parameters of the text sample data;

[0052] The second deep learning module is further used to optimize the initial speech synthesis model according to the loss values of the multiple text sample data to obtain a speech synthesis model.

[0053] In the above scheme, text feature data and pitch feature data are obtained by encoding the text data and the pitch data corresponding to the text data. The text feature data and the pitch feature data are merged so that the obtained first feature data can contain the feature data of both text and pitch. The predicted duration of each text is used to extend the duration of the corresponding part of the first feature data, so that the second feature data obtained after the duration extension can contain the feature data of text, pitch and rhythm at the same time. The target speech obtained after decoding the second feature data and inputting it into the neural network vocoder can fully consider the text, pitch and rhythm features, can have emotions, and has higher stability and authenticity. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0055] Figure 1 A flowchart of a speech synthesis method according to an embodiment of the present application is shown;

[0056] Figure 2 A flowchart showing a method for merging text feature data and pitch feature data according to another embodiment of the present application is shown;

[0057] Figure 3 A flowchart of a method for extending the duration of first feature data according to another embodiment of the present application is shown;

[0058] Figure 4 A flow chart showing a method for adjusting second feature data according to another embodiment of the present application is shown;

[0059] Figure 5 A flowchart of a speech synthesis model training method according to an embodiment of the present application is shown;

[0060] Figure 6 A schematic structural diagram of a speech synthesis device according to an embodiment of the present application is shown;

[0061] Figure 7 A structural diagram of a speech synthesis model training device according to another embodiment of the present application is shown. DETAILED DESCRIPTION

[0062] To make the objectives, features, and advantages of this application more obvious and understandable, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of this application.

[0063] Technical problem.

[0064] In order to make the synthesized song have emotions and improve the stability and authenticity of the song, an embodiment of this application provides a voice synthesis method, as Figure 1 shown, the method includes:

[0065] Step 101: Obtain text data and pitch data corresponding to the text data. The text data contains multiple texts, and the pitch data represents the pitch corresponding to each text.

[0066] Step 102: Encode the text data and pitch data to obtain text feature data and pitch feature data.

[0067] Step 103: Predict the duration of the multiple texts to obtain the predicted duration of each text. The predicted duration represents the number of frames corresponding to the text.

[0068] For example, if a certain text data is "today", which contains two texts "今" and "天", predicting the duration of these two texts, the predicted duration of "今" is 5 speech frames, and the predicted duration of "天" is 4 speech frames.

[0069] Preferably, if a certain text is "我", and there are two phonemes in the text "我", namely "w" and "o", the predicted duration of this text is [2, 2]. The predicted duration of this text can further represent the number of frames of the phonemes in this text, indicating that the phoneme "w" of this text is 2 speech frames, and the phoneme "o" of this text is 2 speech frames.

[0070] Step 104: Merge the text feature data and pitch feature data to obtain first feature data.

[0071] Merging the text feature data and pitch feature data enables the obtained first feature data to simultaneously contain the feature data of text and pitch.

[0072] Step 105: Use the predicted duration of each text to perform duration expansion on the first feature data to obtain second feature data.

[0073] The predicted duration of each text is used to extend the duration of the corresponding partial data of the text in the first feature data, and the corresponding partial data is extended to the number of frames represented by the predicted duration.

[0074] Step 106: decode the second feature data to obtain speech spectrum parameters, and input the speech spectrum parameters into a pre-trained neural network vocoder to generate and obtain the target speech.

[0075] The speech spectrum parameters obtained after decoding the second feature data cannot be played. It is necessary to use a trained neural network vocoder to process the speech spectrum parameters to obtain a target speech that can be played.

[0076] In this embodiment, a pre-trained Neural Vocoder is used to process speech spectrum parameters. Neural Vocoder uses a convolutional neural network and directly models the speech at the sampling points, so the generated speech is more realistic.

[0077] By encoding the text data and the pitch data corresponding to the text data, text feature data and pitch feature data are obtained. The text feature data and pitch feature data are merged so that the obtained first feature data can contain the feature data of both text and pitch. The predicted duration of each text is used to extend the duration of the corresponding part of the first feature data, so that the second feature data obtained after the duration extension can contain the feature data of text, pitch and rhythm at the same time. The target speech obtained after decoding the second feature data and inputting it into the neural network vocoder can fully consider the text, pitch and rhythm features, can have emotions, and has higher stability and authenticity.

[0078] like Figure 2 As shown, this embodiment also provides a method for merging text feature data and pitch feature data, including:

[0079] Step 201 : Determine text sub-feature data and pitch sub-feature data corresponding to each text from the text feature data and pitch feature data.

[0080] The text sub-feature data corresponding to each text is determined from the text feature data.

[0081] For example, a certain text data is "The weather is great today", and the text feature data obtained after encoding this text data is 48-bit feature data. Each text in the text data corresponds to 8-bit feature data respectively. The text "今 (today)" corresponds to the feature data from the 1st bit to the 8th bit in the text feature data; the text "天 (weather)" corresponds to the feature data from the 9th bit to the 16th bit in the text feature data; the text "天 (weather)" corresponds to the feature data from the 17th bit to the 24th bit in the text feature data; the text "气 (air)" corresponds to the feature data from the 25th bit to the 32nd bit in the text feature data; the text "真 (really)" corresponds to the feature data from the 33rd bit to the 40th bit in the text feature data; the text "好 (good)" corresponds to the feature data from the 41st bit to the 48th bit in the text feature data. Determine the text feature data corresponding to these texts as the text sub-feature data corresponding to these texts.

[0082] Determine the pitch sub-feature data corresponding to each text from the pitch feature data.

[0083] For example, a certain text data is "Hello", and the pitch feature data obtained after encoding the pitch data corresponding to this text data is 20-bit feature data. Each text in the text data corresponds to 10-bit feature data respectively. The pitch data corresponding to the text "你 (you)" corresponds to the feature data from the 1st bit to the 10th bit in the pitch feature data; the pitch data corresponding to the text "好 (good)" corresponds to the feature data from the 11th bit to the 20th bit in the pitch feature data; Determine the pitch feature data corresponding to these texts as the pitch sub-feature data corresponding to these texts.

[0084] Step 202, determine the first sub-feature data of the text according to the text sub-feature data and the pitch sub-feature data of the text.

[0085] In this embodiment, there are two methods to determine the first sub-feature data of the text according to the text sub-feature data and the pitch sub-feature data of the text:

[0086] The first method: Add the text sub-feature data and the pitch sub-feature data of the text to obtain the first sub-feature data of the text.

[0087] For example, the text sub-feature data of a certain text is "21322148", and the pitch sub-feature data is "12856231", then add the two to obtain the first sub-feature data "34178379" of the text.

[0088] The second method: Combine the text sub-feature data and the pitch sub-feature data of the text to obtain the first sub-feature data of the text.

[0089] For example, the text sub-feature data of a certain text is 8-bit data, which is "21322148", and the pitch sub-feature data is 8-bit data, which is "12856231". Then the two are combined to obtain the 16-bit first sub-feature data "2132214812856231" of this text.

[0090] In other embodiments, other methods that can determine the first sub-feature data of the text according to the text sub-feature data and pitch sub-feature data of the text can also be used.

[0091] Step 203: Combine all the first sub-feature data in the order of the corresponding text in the text data to obtain the first feature data.

[0092] For example, a certain text data is "Hello", and the corresponding texts are "你 (nǐ)", "好 (hǎo)", and "啊 (ā)". The first sub-feature data corresponding to the text "你 (nǐ)" is "21322148"; the first sub-feature data corresponding to the text "好 (hǎo)" is "12856231"; the first sub-feature data corresponding to the text "啊 (ā)" is "14257855". Then the first sub-feature data corresponding to these three texts are combined in the order of the corresponding text in the text data to obtain the first feature data "213221481285623114257855".

[0093] Determine the first feature data according to the text feature data and pitch feature data, so that the first feature data can contain the feature data of both text and pitch at the same time. Make the target speech obtained after the subsequent steps of processing the first feature data fully contain the feature data of pitch, so that the synthesized target speech has emotion.

[0094] As Figure 3 shown, this embodiment also provides a method for expanding the duration of the first feature data, including:

[0095] Step 301: Determine the first sub-feature data corresponding to each text from the first feature data.

[0096] In this embodiment, the first sub-feature data corresponding to each text obtained in step 202 can be directly used, or the corresponding first sub-feature data can be determined from the first feature data according to the text.

[0097] Step 302: Expand the first sub-feature data corresponding to the text to the number of frames indicated by the predicted duration according to the predicted duration of the text to obtain the second sub-feature data.

[0098] For example, a certain text data is "Hello", and the corresponding texts are "你 (nǐ)" and "好 (hǎo)". The predicted duration corresponding to the text "你 (nǐ)" is 3 frames, and the first sub-feature data is "21322148"; the predicted duration corresponding to the text "好 (hǎo)" is 2 frames, and the first sub-feature data is "12856231".

[0099] According to the predicted duration corresponding to the text "你 (nǐ)", copy the first sub-feature data "21322148" corresponding to this text 3 times and merge them to obtain the second sub-feature data "213221482132214821322148" corresponding to this text;

[0100] According to the predicted duration corresponding to the text "好 (hǎo)", copy the first sub-feature data "12856231" corresponding to this text 2 times and merge them to obtain the second sub-feature data "1285623112856231" corresponding to this text.

[0101] Step 303, merge the second sub-feature data according to the order of the corresponding texts in the text data to obtain the second feature data.

[0102] For example, a certain text data is "Hello", and the corresponding texts are "你 (nǐ)" and "好 (hǎo)". The second sub-feature data corresponding to the text "你 (nǐ)" is "213221482132214821322148"; the second sub-feature data corresponding to the text "好 (hǎo)" is "1285623112856231". Then merge these two second sub-feature data according to the order of the corresponding texts in the text data to obtain the second feature data "2132214821322148213221481285623112856231".

[0103] By using the predicted duration corresponding to the text to perform duration expansion on the first sub-feature data corresponding to this text, it is possible to make the length of some of the corresponding data in the obtained second feature data more in line with the length of the real pronunciation, thereby making the stability and authenticity of the synthesized target voice higher.

[0104] As Figure 4 shown, this embodiment also provides a method for adjusting the second feature data, including:

[0105] Step 401, determine the second sub-feature data corresponding to each text from the second feature data.

[0106] In this embodiment, the second sub-feature data corresponding to each text obtained in step 302 can be directly used, or the corresponding second sub-feature data can be determined from the second feature data according to the text.

[0107] Step 402: traverse all second sub-feature data.

[0108] Step 403: Determine the similarity between the current second sub-feature data and each other second sub-feature data, and determine a weight according to the similarity.

[0109] Step 404 : Adjust the current second sub-feature data according to the current second sub-feature data, all other second sub-feature data, and the weights between the current second sub-feature data and each of the other second sub-feature data.

[0110] Step 405: After traversing all the second sub-feature data, all the adjusted second sub-feature data are merged according to the order of the corresponding texts in the text data to obtain the second feature data.

[0111] The trained attention layer is used to calculate the similarity between the current second sub-feature data and each other second sub-feature data, and a weight is determined based on the similarity. Finally, the current second sub-feature data is adjusted based on the current second sub-feature data, all other second sub-feature data, and the weights between the current second sub-feature data and each other second sub-feature data.

[0112] By calculating the similarity between the current second sub-feature data and each other's second sub-feature data, determining weights based on the similarities, and finally adjusting the current second sub-feature data based on all the second sub-feature data and weights, the contextual relationship between the current second sub-feature data and the other second sub-feature data can be fully integrated, further improving the authenticity of the final synthesized target speech.

[0113] like Figure 5 As shown, this embodiment also provides a speech synthesis model training method, including:

[0114] Step 501: Acquire multiple text sample data and label speech spectrum parameters corresponding to each text sample data.

[0115] Step 502: Input the text sample data into an initial speech synthesis model to obtain predicted speech spectrum parameters of the text sample data.

[0116] Step 503: Determine the loss value of the text sample data according to the label speech spectrum parameters and the predicted speech spectrum parameters of the text sample data.

[0117] Step 504: Optimize the initial speech synthesis model according to the loss values of the plurality of text sample data to obtain a speech synthesis model.

[0118] The parameters in the initial speech synthesis model are optimized based on multiple loss values. If the model does not converge, the optimized model is used to re-predict based on all text sample data and the loss value is calculated. A new round of optimization is performed on the parameters in the optimized model based on the recalculated multiple loss values until the model converges.

[0119] In order to implement the above-mentioned speech synthesis method, Figure 6 As shown, an example of the present application further provides a speech synthesis device, comprising:

[0120] A first acquisition module 10 is configured to acquire text data and pitch data corresponding to the text data, wherein the text data includes a plurality of texts, and the pitch data represents the pitch corresponding to each text;

[0121] An encoding module 20, configured to encode the text data and the pitch data to obtain text feature data and pitch feature data;

[0122] A first deep learning module 30 is configured to perform duration prediction on the plurality of texts to obtain a predicted duration of each text, where the predicted duration represents the number of frames corresponding to the text;

[0123] A calculation module 40 is used to combine the text feature data and the pitch feature data to obtain first feature data;

[0124] The first deep learning module 30 is further configured to perform duration expansion on the first feature data using the predicted duration of each text to obtain second feature data;

[0125] The decoding module 50 is used to decode the second feature data to obtain speech spectrum parameters, and input the speech spectrum parameters into a pre-trained neural network vocoder to generate and obtain the target speech.

[0126] The calculation module 40 is further configured to determine text sub-feature data and pitch sub-feature data corresponding to each text from the text feature data and pitch feature data;

[0127] The calculation module 40 is further configured to determine first sub-feature data of the text based on the text sub-feature data and the pitch sub-feature data of the text;

[0128] The calculation module 40 is further configured to merge all first sub-feature data according to the order of corresponding texts in the text data to obtain first feature data.

[0129] The first deep learning module 30 is further configured to determine first sub-feature data corresponding to each text from the first feature data;

[0130] The first deep learning module 30 is further configured to expand the first sub-feature data corresponding to the text to the number of frames indicated by the predicted duration according to the predicted duration, to obtain second sub-feature data;

[0131] The first deep learning module 30 is further configured to merge the second sub-feature data according to the order of the corresponding texts in the text data to obtain second feature data.

[0132] The first deep learning module 30 is further configured to determine second sub-feature data corresponding to each text from the second feature data;

[0133] The first deep learning module 30 is further configured to traverse all second sub-feature data;

[0134] The first deep learning module 30 is further configured to determine a similarity between the current second sub-feature data and each other second sub-feature data, and determine a weight based on the similarity;

[0135] The first deep learning module 30 is further configured to adjust the current second sub-feature data according to the current second sub-feature data, all other second sub-feature data, and the weight of each of the other second sub-feature data of the current second sub-feature data;

[0136] The first deep learning module 30 is further configured to, after traversing all the second sub-feature data, merge all the adjusted second sub-feature data according to the order of the corresponding texts in the text data to obtain the second feature data.

[0137] In order to implement the above-mentioned speech synthesis model training method, such as Figure 7 As shown, an example of the present application further provides a speech synthesis model training device, comprising:

[0138] The second acquisition module 60 is used to obtain a plurality of text sample data and a label speech spectrum parameter corresponding to each text sample data;

[0139] A second deep learning module 70 is configured to input the text sample data into an initial speech synthesis model to obtain predicted feature data of the text sample data;

[0140] The second deep learning module 70 is further configured to determine a loss value of the text sample data based on the label speech spectrum parameters and the predicted speech spectrum parameters of the text sample data;

[0141] The second deep learning module 70 is further configured to optimize the initial speech synthesis model according to the loss values of the plurality of text sample data to obtain a speech synthesis model.

[0142] In one example, an embodiment of the present application further provides a mobile terminal, which includes at least one memory and a processor communicatively connected to the at least one memory; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are configured to execute the above Figures 1 to 4 The speech synthesis method described in any one of the embodiments and the above Figure 5 The speech synthesis model training method described in the embodiment.

[0143] In addition, the present invention also provides a computer-readable storage medium storing computer-executable instructions for executing the above-mentioned Figures 1 to 4 The speech synthesis method process described in any one of the embodiments and the above Figure 5 The speech synthesis model training method process described in the embodiment.

[0144] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0145] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of the embodiments of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0146] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0147] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interface, indirect coupling or communication connection of the device or unit, which may be electrical, mechanical or other forms.

[0149] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0150] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0151] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Acquire text data and pitch data corresponding to the text data, wherein the text data includes a plurality of texts, and the pitch data represents the pitch corresponding to each text; Encoding the text data and the pitch data to obtain text feature data and pitch feature data; Performing duration prediction on the multiple texts to obtain a predicted duration of each text, where the predicted duration represents the number of frames corresponding to the text; Merging the text feature data and the pitch feature data to obtain first feature data; Extending the first feature data by using the predicted duration of each text to obtain second feature data; Decoding the second feature data to obtain speech spectrum parameters, and inputting the speech spectrum parameters into a pre-trained neural network vocoder to generate and obtain target speech; The merging of the text feature data and the pitch feature data comprises: Determining text sub-feature data and pitch sub-feature data corresponding to each text from the text feature data and pitch feature data; Determining first sub-feature data of the text based on the text sub-feature data and the pitch sub-feature data of the text; Merging all first sub-feature data according to the order of corresponding texts in the text data to obtain first feature data; The method of performing duration expansion on the first feature data by using the predicted duration of each text to obtain second feature data includes: Determining first sub-feature data corresponding to each text from the first feature data; According to the predicted duration of the text, the first sub-feature data corresponding to the text is expanded to the number of frames indicated by the predicted duration to obtain the second sub-feature data; The second sub-feature data are merged according to the order of the corresponding texts in the text data to obtain second feature data.

2. The speech synthesis method according to claim 1, characterized in that: After obtaining the second characteristic data, the method further includes: Determining second sub-feature data corresponding to each text from the second feature data; Traverse all second sub-feature data; Determining a similarity between the current second sub-feature data and each other second sub-feature data, and determining a weight according to the similarity; Adjusting the current second sub-feature data according to the current second sub-feature data, all other second sub-feature data, and the weight of each of the other second sub-feature data of the current second sub-feature data; After traversing all the second sub-feature data, all the adjusted second sub-feature data are merged according to the order of the corresponding texts in the text data to obtain the second feature data.

3. A speech synthesis device, characterized in that: The device comprises: A first acquisition module is configured to acquire text data and pitch data corresponding to the text data, wherein the text data includes a plurality of texts, and the pitch data represents the pitch corresponding to each text; An encoding module, configured to encode the text data and the pitch data to obtain text feature data and pitch feature data; A first deep learning module is configured to perform duration prediction on the plurality of texts to obtain a predicted duration of each text, where the predicted duration represents the number of frames corresponding to the text; a calculation module, configured to combine the text feature data and the pitch feature data to obtain first feature data; The first deep learning module is further configured to perform duration expansion on the first feature data using the predicted duration of each text to obtain second feature data; A decoding module, configured to decode the second feature data to obtain speech spectrum parameters, and input the speech spectrum parameters into a pre-trained neural network vocoder to generate and obtain the target speech; The calculation module is further used to determine text sub-feature data and pitch sub-feature data corresponding to each text from the text feature data and pitch feature data; The calculation module is further configured to determine first sub-feature data of the text based on the text sub-feature data and the pitch sub-feature data of the text; The calculation module is further configured to merge all first sub-feature data according to the order of corresponding texts in the text data to obtain first feature data; The first deep learning module is further configured to determine first sub-feature data corresponding to each text from the first feature data; The first deep learning module is further configured to expand the first sub-feature data corresponding to the text to a number of frames indicated by the predicted duration according to the predicted duration, to obtain second sub-feature data; The first deep learning module is further configured to merge the second sub-feature data according to the order of the corresponding texts in the text data to obtain second feature data.

4. The speech synthesis device according to claim 3, characterized in that: include: The first deep learning module is further configured to determine second sub-feature data corresponding to each text from the second feature data; The first deep learning module is further configured to traverse all second sub-feature data; The first deep learning module is further configured to determine a similarity between the current second sub-feature data and each other second sub-feature data, and determine a weight based on the similarity; The first deep learning module is further configured to adjust the current second sub-feature data according to the current second sub-feature data, all other second sub-feature data, and the weight of each of the other second sub-feature data of the current second sub-feature data; The first deep learning module is further configured to, after traversing all the second sub-feature data, merge all the adjusted second sub-feature data in the order of the corresponding texts in the text data to obtain the second feature data.

Images