A speech synthesis method and system

Abstract

An embodiment of the present specification provides a speech synthesis method, which includes obtaining a text to be processed; determining prediction results of multiple tasks based on the text to be processed, the multiple tasks including at least two of the rhythm of characters in the text to be processed, the pronunciation of characters in the text to be processed, the type of character strings in the text to be processed, and the emotion of the character strings in the text to be processed; and determining the speech corresponding to the text to be processed based on the prediction results of the multiple tasks.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a speech synthesis method and system. Background Art

[0002] With the development of artificial intelligence technology, speech synthesis technology has attracted more and more attention. Speech synthesis technology can convert text into speech output. The existing speech synthesis technology synthesizes speech that is relatively stiff and has a poor user experience.

[0003] Therefore, there is a need to provide a speech synthesis method and system for improving the quality of synthesized speech. Summary of the invention

[0004] One of the embodiments of the present specification provides a speech synthesis method, which includes: obtaining a text to be processed; based on the text to be processed, determining prediction results of multiple tasks, the multiple tasks including at least two of the rhythm of characters in the text to be processed, the pronunciation of characters in the text to be processed, the type of character strings in the text to be processed, and the emotion of the character strings in the text to be processed; based on the prediction results of the multiple tasks, determining the speech corresponding to the text to be processed.

[0005] One of the embodiments of the present specification provides a speech synthesis system, the system comprising: a text acquisition module, used to acquire a text to be processed; a task prediction module, used to determine the prediction results of multiple tasks based on the text to be processed, the multiple tasks including at least two of the rhythm of the characters in the text to be processed, the pronunciation of the characters in the text to be processed, the type of the character string in the text to be processed and the emotion of the character string in the text to be processed; and a result generation module, used to determine the speech corresponding to the text to be processed based on the prediction results of the multiple tasks.

[0006] One of the embodiments of the present specification provides a speech synthesis device, including a processor, wherein the processor is used to execute a speech synthesis method.

[0007] One of the embodiments of the present specification provides a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes a speech synthesis method. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0009] Figure 1 is a schematic diagram of an application scenario of a speech synthesis system according to some embodiments of this specification;

[0010] Figure 2 is an exemplary block diagram of a processing device according to some embodiments of this specification;

[0011] Figure 3 is an exemplary flow chart of a speech synthesis method according to some embodiments of this specification;

[0012] Figure 4 is a schematic diagram of the structure of the prediction model shown in some embodiments of this specification;

[0013] Figure 5 is another schematic diagram of a prediction model according to some embodiments of this specification;

[0014] Figure 6 is another schematic diagram of a prediction model according to some embodiments of this specification;

[0015] Figure 7 It is a schematic diagram of a training prediction model according to some embodiments of this specification. DETAILED DESCRIPTION

[0016] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0017] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0018] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0019] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0020] Figure 1 It is a schematic diagram of an application scenario of the speech synthesis system 100 according to some embodiments of the present specification.

[0021] In some embodiments, the speech synthesis system 100 can determine the speech 160 corresponding to the to-be-processed text 150 by implementing the methods and / or processes disclosed in this specification.

[0022] like Figure 1 As shown, the speech synthesis system 100 involved in the embodiment of this specification may include a processing device 110 , a network 120 , a terminal 130 and a storage device 140 .

[0023] The processing device 110 can be used to process data and / or information from at least one component of the speech synthesis system 100 or an external data source (e.g., a cloud data center). The processing device 110 can access data and / or information from the terminal 130 and / or the storage device 140 through the network 120. The processing device 110 can be directly connected to the terminal 130 and / or the storage device 140 to access information and / or data. For example, the processing device 110 can obtain the text to be processed 150 from the terminal 130 and / or the storage device 140. The processing device 110 can process the acquired data and / or information. For example, the processing device 110 can process the text to be processed 150 based on the prediction model, determine the prediction results of multiple tasks, and determine the speech 160 corresponding to the text to be processed 150 based on the prediction results of multiple tasks. In some embodiments, the processing device 110 can be a single server or a server group. The processing device 110 can be local or remote. The processing device 110 can be implemented on a cloud platform. For more description of the processing device 110, see Figure 2 The related descriptions will not be repeated here.

[0024] The network 120 may include any suitable network that provides information and / or data exchange that can facilitate the speech synthesis system 100. In some embodiments, one or more components of the speech synthesis system 100 (e.g., the processing device 110, the terminal 130, and the storage device 140) may exchange information and / or data through the network 120. The network 120 may include a local area network (LAN), a wide area network (WAN), a wired network, a wireless network, etc., or any combination thereof.

[0025] The terminal 130 refers to one or more terminal devices or software used by the user. In some embodiments, the terminal 130 may be a mobile device, a tablet computer, etc. or any combination thereof. In some embodiments, the terminal 130 may interact with other components in the speech synthesis system 100 through the network 120. For example, the terminal 130 may send one or more control instructions to the processing device 110 to control the processing device 110 to process the text to be processed 150, thereby determining the speech 160 corresponding to the text to be processed 150. In some embodiments, the terminal 130 may be a part of the processing device 110. In some embodiments, the terminal 130 may be integrated with the processing device 110 as an input terminal of the text to be processed 150.

[0026] The storage device 140 may be used to store data, instructions and / or any other information. In some embodiments, the storage device 140 may store data and / or information obtained from, for example, the processing device 110, the terminal 130, etc. For example, the storage device 140 may store a predetermined text to be processed 150. For another example, the storage device 140 may store a trained prediction model. In some embodiments, the storage device 140 may include a large-capacity memory, a removable memory, etc., or any combination thereof.

[0027] Figure 2 is an exemplary block diagram of a processing device 110 according to some embodiments of the present specification.

[0028] In some embodiments, the processing device 110 may include a text acquisition module 210 , a task prediction module 220 , and a result generation module 230 .

[0029] The text acquisition module 210 can be used to acquire the text to be processed. For more information about the text to be processed, see Figure 3 and its related description.

[0030] The task prediction module 220 can be used to determine the prediction results of multiple tasks based on the text to be processed. For more description of the prediction results, please refer to Figure 3 In some embodiments, the task prediction module 220 may obtain reference results of preset tasks among multiple tasks; determine initial prediction results of multiple tasks based on the text to be processed; replace the initial prediction results of the preset tasks based on the reference results of the preset tasks to determine the prediction results of the multiple tasks. For more descriptions of reference results and initial prediction results, please refer to Figure 3 and its related description.

[0031] In some embodiments, the task prediction module 220 may process the text to be processed using a prediction model to determine prediction results for multiple tasks. For more information about the prediction model, see Figure 3In some embodiments, the task prediction module 220 can obtain reference results of preset tasks among multiple tasks; process the to-be-processed text and the reference results of the preset tasks through the prediction model to determine the prediction results of the multiple tasks. For more descriptions of the preset tasks, see Figure 3 and its related description.

[0032] In some embodiments, the task prediction module 220 can obtain a basic model based on the first training set, where the first training set includes multiple training samples and their corresponding labels; based on the basic model, the parameters of the prediction model are determined. For more information about the first training set and the basic model, see Figure 3 and its related description.

[0033] The result generation module 230 can be used to determine the speech corresponding to the text to be processed based on the prediction results of multiple tasks. For more information about determining the speech corresponding to the text to be processed, please refer to Figure 3 and its related description.

[0034] It should be noted that the above description of the processing device 110 and its modules is only for convenience of description and does not limit the present specification to the scope of the embodiments. It is understandable that, after understanding the principle of the processing device 110, those skilled in the art may arbitrarily combine the modules or form a subsystem to connect with other modules without deviating from the principle. In some embodiments, Figure 2 The text acquisition module 210, task prediction module 220 and result generation module 230 disclosed in the specification can be different modules in a processing device 110, or a module can realize the functions of two or more modules. For example, each module can share a storage module, or each module can have its own storage module. Such variations are within the scope of protection of this specification.

[0035] Figure 3 FIG. 3 is an exemplary flow chart of a speech synthesis method 300 according to some embodiments of the present specification. Figure 3 As shown, the process 300 includes the following steps: In some embodiments, the process 300 may be executed by the speech synthesis system 100 .

[0036] Step 310 , obtaining the text to be processed. In some embodiments, step 310 may be performed by the text obtaining module 210 .

[0037] The text to be processed may be a text for synthesizing speech. In some embodiments, the text to be processed may be composed of at least one character, wherein the character may include letters, words, numbers, operation symbols, punctuation marks and other symbols, and some functional symbols.

[0038] Multiple characters can form a string. For example, a string can be a sentence, and specific punctuation marks (e.g., ".", "!", "?", "……", etc.) can divide the text to be processed into at least one string. For example, the text to be processed can be "Really? I don't believe it!", and the text to be processed can be divided into two strings by the punctuation marks "?" and "!". In some embodiments, a string can also be a phrase with multiple meanings. For example, for the string "14:15", its meaning can be that the time is 2:15 pm or the score is 14 to 15.

[0039] In some embodiments, the text acquisition module 210 can obtain the text to be processed from the terminal 130, the storage device 140, or an external data source.

[0040] Step 320, based on the text to be processed, determine the prediction results of multiple tasks. In some embodiments, step 320 can be executed by the task prediction module 220.

[0041] A task can be the information required to convert the text to be processed into speech. In some embodiments, a task can be to determine one or more of the prosody of the characters in the text to be processed, the pronunciation of the characters in the text to be processed, the type of the string in the text to be processed, and the emotion of the string in the text to be processed. For example, multiple includes two, three, etc.

[0042] In some embodiments, the task prediction module 220 can obtain information from the terminal 130, the storage device 140, or an external data source to determine multiple tasks. For example, the task prediction module 220 can obtain a task table from the terminal 130, the storage device 140, or an external data source, and the task table can record the quantity and type of tasks. By way of example, the numbers of the character prosody determination task, the character pronunciation determination task, the string emotion determination task, and the string type determination task are 1, 2, 3, and 4 in sequence. If the task table is [1, 3], then the task prediction module 220 needs to perform 2 tasks on the text to be processed: the character prosody determination task and the string emotion determination task.

[0043] Determining the prosody of the characters in the text to be processed can be determining the pause length after pronouncing the characters. In some embodiments, the prosody can be divided into multiple levels of prosody. For example, the prosody can be represented by #1, #2, #3, *, and [PAD], where the pauses of #1, #2, and #3 gradually increase, * represents no pause, and [PAD] represents skipping the character.

[0044] Determining the pronunciation of the characters in the text to be processed can be determining the pronunciation of the characters. For example, the pronunciation of "你" is "ni3", where "3" indicates the third tone.

[0045] Determining the type of the string in the text to be processed may be determining the true meaning of a phrase with multiple meanings in the text to be processed. For example, for the string "14:15", the task prediction module 220 needs to determine whether the meaning of the string in the text to be processed represents time or score. In some embodiments, the task prediction module 220 may first identify the string to determine whether the string is of a preset type, and a string of a preset type may have multiple meanings. If so, the type of the string is then identified.

[0046] Determining the sentiment of the string in the to-be-processed text may be determining the sentiment of the sentence. For example, the sentiment of the sentence may be positive sentiment, negative sentiment, and blank sentiment, wherein positive sentiment may be joy, excitement, etc., negative sentiment may be anxiety, sadness, anger, despair, etc., and blank sentiment may refer to the absence of any sentiment.

[0047] In some embodiments, the task prediction module 220 can determine the prediction results of multiple tasks based on preset rules. In some embodiments, the preset rules may be different for multiple tasks. For example, for the task of determining the rhythm of characters in the text to be processed, the preset rhythm rules set may include: the rhythm of the Chinese character before the punctuation mark is #3, the rhythm of the punctuation mark indicating the end of the sentence is *, the rhythm of other Chinese characters is #1, the rhythm of some special punctuation marks (for example, "(book title marks) etc.) is [PAD], and the rhythm of some special punctuation marks (for example, "(quotation marks), : (colon) etc.) is #2, etc. For example, for the text to be processed "Really?", the task prediction module 220 can determine that its rhythm is [Really #1's #1's #3? *], indicating that the rhythm of "really", "of" is #1, the rhythm of "may" is #3, and the rhythm of "?" is *.

[0048] In some embodiments, the task prediction module 220 can process the text to be processed using a prediction model to determine the prediction results of multiple tasks. The prediction model is a machine learning model. For example, the type of prediction model may include but is not limited to a deep neural network model, a recurrent neural network model, a custom model, etc. In some embodiments, the prediction model may include multiple task layers, and the multiple task layers correspond to multiple tasks one by one. For more descriptions of the prediction model, see Figure 4 and its related description.

[0049] In some embodiments, the task prediction module 220 may obtain reference results of preset tasks among the multiple tasks. The task prediction module 220 may determine the initial prediction results of the multiple tasks based on the text to be processed. Further, the task prediction module 220 may replace at least a part of the initial prediction results of the preset tasks based on the reference results of the preset tasks to determine the prediction results of the multiple tasks.

[0050] The reference result may be the result of a preset preset task, wherein the preset task may be at least one of a plurality of tasks. In some embodiments, the preset task may be to determine the rhythm of characters in the text to be processed, and the reference result may be the rhythm of at least one character (i.e., the target character) in the text to be processed. In some embodiments, the preset task may be to determine the pronunciation of characters in the text to be processed, and the reference result may be the pronunciation of at least one character (i.e., the target character) in the text to be processed. In some embodiments, the preset task may be to determine the type of a string in the text to be processed, and the reference result may be the type of a plurality of strings (i.e., the target string) in the text to be processed. In some embodiments, the preset task may be to determine the emotion of a string in the text to be processed, and the reference result may be the emotion of at least one string (i.e., the target string) in the text to be processed.

[0051] In some embodiments, the task prediction module 220 can obtain reference results from the terminal 130, the storage device 140, or an external data source. For example, the task prediction module 220 can obtain a preset task table and a reference result table corresponding to the preset task from the terminal 130, the storage device 140, or an external data source. The preset task table is used to record the preset tasks, and a reference result table is used to record the reference results corresponding to a preset task. For example, the preset task table is [2,3], then the preset tasks include a character pronunciation determination task and a string emotion determination task, wherein the reference result table corresponding to the character pronunciation determination task is [[2,5][di1,di4]], then the reference result corresponding to the character pronunciation determination task is that the pronunciation of the second character is "di1", and the pronunciation of the fifth character is "di4".

[0052] In some embodiments, the initial prediction result is the result of multiple tasks determined directly based on the text to be processed, for example, the result of multiple tasks determined directly based on the text to be processed by preset rules, machine learning models, etc.

[0053] In some embodiments, the task prediction module 220 can replace at least a portion of the initial prediction results of the preset task based on the reference results of the preset task to determine the prediction results of multiple tasks. For example, for the character pronunciation determination task, the task prediction module 220 can replace the initial prediction result of the target character with the reference result of the target character to determine the prediction result. For example, for the text to be processed "take a taxi to the destination", the corresponding character pronunciation determination task has an initial prediction result of [da3, che1, qu4, mu4, di1, di4, *], and a reference result of [[2, 5][che1, di4]]. After replacement, the prediction result of the character pronunciation determination task determined by the task prediction module 220 is [da3, che1, qu4, mu4, di4, di4, *].

[0054] The task prediction module 220 can determine the prediction results of multiple tasks by combining the prediction model and the reference results of the preset tasks in various ways. For example, the prediction model (e.g., Figure 5 The prediction model shown in FIG. 1 can directly generate prediction results of multiple tasks based on the text to be processed and the reference results of the preset tasks. For example, the prediction model (such as Figure 6 The prediction model shown in the figure can first generate initial prediction results of multiple tasks based on the text to be processed, and then replace at least a part of the initial prediction results of the preset task with the reference results of the preset task to generate the prediction results.

[0055] In some embodiments, by using the reference results of the preset tasks, the preset results of the generated multiple tasks can be made more reliable.

[0056] Step 330 , based on the prediction results of the multiple tasks, determine the speech corresponding to the text to be processed. In some embodiments, step 330 may be performed by the result generation module 230 .

[0057] The speech corresponding to the text to be processed may be a speech synthesized based on the text to be processed and the prediction results of multiple tasks.

[0058] In some embodiments, after determining the prediction results of multiple tasks, the result generation module 230 can determine the speech corresponding to the to-be-processed text through the model based on the prediction results of multiple tasks. For example, the prediction results of each character corresponding to multiple tasks are input into an acoustic model (e.g., DeepVoice 3, Transformer TTS, or FastSpeech) to determine the acoustic features of each character, and the acoustic features are de-decoded into speech based on a vocoder (e.g., WaveFlow).

[0059] In some embodiments, the speech synthesis method 300 can determine the prediction results of at least two tasks including the rhythm of characters in the text to be processed, the pronunciation of characters in the text to be processed, the type of character strings in the text to be processed, and the emotion of the character strings in the text to be processed based on the text to be processed, and then synthesize speech based on the text to be processed and the prediction results of multiple tasks, so that the synthesized speech is closer to the actual pronunciation of humans, thereby improving the user experience.

[0060] It should be noted that the above description of process 300 is only for example and illustration, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to process 300 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0061] Figure 4It is a schematic structural diagram of the prediction model 400 shown in some embodiments of this specification. As Figure 4 shown, the prediction model 400 may include an embedding layer 410, a feature layer 420, and a plurality of task layers 430.

[0062] In some embodiments, the embedding layer 410 may encode the characters in the text to be processed to obtain the encodings of the characters in the text to be processed.

[0063] The encoding may be a number converted from the characters in the text to be processed. For example, converting the character "字" in the text to be processed to the number "100" is the encoding.

[0064] In some embodiments, the input of the embedding layer 410 may be the text to be processed, and the output of the embedding layer 410 may be a text encoding vector. Among them, the text encoding vector may be composed of the encodings of each character in the text to be processed. For example, the text encoding vector may be composed of the encodings of the characters in the text to be processed in the character order. In some embodiments, the dimension of the text encoding vector may be determined based on the number of characters in the text to be processed. For example, when the number of characters in the text to be processed is 3, the output dimension of the embedding layer 410 is a text encoding vector of 3. Exemplarily, the text to be processed input into the embedding layer 410 is "打车。", and the text encoding vector output by the embedding layer 410 is [502, 203, 101], where "502" is the encoding of "打", "203" is the encoding of "车", and "101" is the encoding of "。".

[0065] In some embodiments, the embedding layer 410 may encode the characters in the text to be processed through an encoding model to generate a text encoding vector. In some embodiments, the encoding model may be at least one or any combination of a Tokenizer model, a BERT (Bidirectional Encoder Representations from Transformers) model. In some embodiments, the encoding model may also be other models, such as at least one or any combination of a bag-of-words model, a CBOW (Continuous Bag Of Words Model) model, and a Skip-gram (Continuous Skip-gram Model) model.

[0066] In some embodiments, the embedding layer 410 may also encode the characters in the text to be processed in other ways to generate a text encoding vector. For example, the embedding layer 410 may first obtain the encodings of the characters in the text to be processed through a dictionary, and then combine the encodings of the characters in the text to be processed in the character order to generate a text encoding vector, where the dictionary may pre-store the encodings corresponding to multiple characters respectively.

[0067] In some embodiments, the feature layer 420 may extract feature vectors of characters in the text to be processed based on the encoding of the characters in the text to be processed.

[0068] The feature vector can be used to represent information related to the character. The information related to the character includes information about the character itself, and information about the relationship between the character and the text to be processed and / or other characters in the text to be processed. For example, the feature vector can represent the meaning of the character, contextual semantic information, and the position information of the character in the text to be processed. In some embodiments, for each character, the feature layer 420 can extract a corresponding feature vector.

[0069] In some embodiments, the number of feature layers may be greater than one.

[0070] In some embodiments, the input of the feature layer 420 may be the text encoding vector output by the embedding layer 410, and the output of the feature layer 420 may be a feature matrix, wherein the feature matrix may be composed of feature vectors of characters of the text to be processed. In some embodiments, the size of the feature matrix may be n*m, wherein n may represent the number of characters of the text to be processed, and m may represent the dimension of the feature vector of the character. For example, the text to be processed is "Take a taxi to the destination.", the input of the feature layer 420 may be [502, 203, 105, 980, 654, 789, 101], and the output of the feature layer 520 may be a 7*512 feature matrix, wherein 7 represents the number of characters of the text to be processed as 7, 512 represents the dimension of the feature vector of the character as 512, and the first to seventh rows of the matrix are the feature vectors of the first to seventh characters of the file to be processed, respectively.

[0071] In some embodiments, the embedding layer 410 and the feature layer 420 can share a BERT (Bidirectional Encoder Representations from Transformers) model, which can be used to first encode the characters in the text to be processed, generate a text encoding vector, and then extract the feature vectors of the characters in the text to be processed based on the generated text encoding vector, and output a feature matrix.

[0072] In some embodiments, the feature layer 420 can use a BERT model alone to extract feature vectors of characters in the text to be processed based on the generated text encoding vector and output a feature matrix. In some embodiments, the feature layer 420 can also extract feature vectors of characters in the text to be processed through other models (e.g., Transformer model, etc.) and output a feature matrix.

[0073] In some embodiments, the task layer 430 may determine the prediction result of the task based on the feature vector of the characters in the text to be processed. In some embodiments, the prediction model 400 may include multiple task layers 430, wherein the number of task layers 430 included in the prediction model 400 may be greater than or equal to the number of tasks that need to be performed on the text to be processed. Figure 4 As shown, the task layer 430 includes a task layer 430-1 for processing task 1, a task layer 430-2 for processing task 2, and a task layer 430-n for processing task n. Each task layer can be used to obtain a prediction result of a corresponding task. In some embodiments, the task layer 340 can be a linear layer.

[0074] In some embodiments, a task layer 430 can be used to determine the predicted result of the pronunciation of each character based on the feature vector of each character in the text to be processed. For more description of pronunciation, see Figure 3 And its related description, no further details are given here. In some embodiments, the input of the task layer 430 is the feature vector of the character, the task layer 430 can determine the probability of each character for each pronunciation, and the output of the task layer 430 can be an n-dimensional vector, wherein a numerical value in the vector represents the pronunciation with the maximum probability corresponding to a character, and n corresponds to the number of characters in the text to be processed. For example, the text to be processed is "Take a taxi to the destination.", and the output of the task layer 430 can be [da3, che1, qu4, mu4, di4, di4, *].

[0075] In some embodiments, a task layer 430 can be used to determine the prediction result of the prosody of each character based on the feature vector of each character in the text to be processed. For more description of prosody, see Figure 3 In some embodiments, the input of the task layer 430 is the feature vector of each character, the task layer 430 can calculate the probability that the rhythm of the character is each preset rhythm, and the output of the task layer 430 can be an n-dimensional vector, wherein a value of the vector can be the rhythm with the maximum probability corresponding to a character.

[0076] In some embodiments, a task layer 430 can be used to determine a prediction result of the type of a string based on a feature vector of characters included in the string in the text to be processed. For more description of strings and string types, see Figure 3The related descriptions thereof are not repeated here. For example, the input of the task layer 430 is the feature vector of the characters included in the string, and the output of the task layer 430 can be an x-dimensional vector, wherein a value of the vector can be a type with the maximum probability corresponding to a string, and x corresponds to the number of strings of the type to be predicted in the text to be processed, and the number of strings of the type to be predicted can be all the strings in the text to be processed, or it can be part of the strings. For example, the string of the type to be predicted can be a string of a specific type.

[0077] In some embodiments, a task layer 430 can be used to determine the predicted result of the emotion of the string based on the feature vectors of the characters included in the string in the text to be processed. For more description of the emotion of the string, see Figure 3 The related descriptions are not repeated here. In some embodiments, the input of the task layer 430 is the feature vector of the characters included in the string, and the output of the task layer 430 can be an x-dimensional vector, wherein a value of the vector can be the maximum probability emotion corresponding to a string, and x corresponds to the number of strings in the text to be processed. Similar to the predicted string type, the string at this time can be all the strings in the text to be processed, or it can be a part of the string.

[0078] In some embodiments, obtaining reference results of multiple tasks through a prediction model can improve the efficiency and accuracy of speech synthesis.

[0079] In some embodiments, the prediction model 400 can obtain reference results of preset tasks among multiple tasks, and process the text to be processed and the reference results of the preset tasks to determine the prediction results of the multiple tasks. For more descriptions of preset tasks and reference results, please refer to Figure 3 The related descriptions will not be repeated here.

[0080] For more information on how the prediction model determines the prediction results based on the reference results, see Figure 5 and Figure 6 For more information on training the prediction model, see Figure 7 and its related description.

[0081] Figure 5 is another schematic diagram of the prediction model 500 according to some embodiments of the present specification.

[0082] like Figure 5 As shown, the prediction model 500 may include an embedding layer 410 , a feature layer 420 , and multiple task layers 430 .

[0083] In some embodiments, the prediction model 500 can determine the task layer (for example, task layer 430-2) corresponding to the reference result based on the task number in the task list, and input the reference result into the task layer 430. Accordingly, the task layer 430 output can directly output the prediction result containing the reference result.

[0084] In some embodiments, the embedding layer 410 may include a dictionary 510 for obtaining encodings of characters in the reference results and inputting the encoded reference results into the corresponding task layer (eg, the task layer 430 - 2 ).

[0085] In some embodiments, the task layer corresponding to the preset task may first process other characters or other strings other than the target character or target string to obtain prediction results of other characters or other strings, and the task layer corresponding to the preset task then combines the reference result of the target character or target string to determine the prediction result of the text to be processed for the preset task. In some embodiments, the task layer corresponding to the preset task may process all characters or all strings in the text to be processed to obtain initial prediction results of all characters or all strings, and the task layer corresponding to the preset task then replaces the initial prediction result of the target character or target string with the reference result of the target character or target string to obtain the prediction result of the text to be processed for the preset task.

[0086] For example, for the task of pronouncing the characters in the text to be processed "Take a taxi to the destination." (this task is Task 2), there is a reference result [[2,5][che1,di4]] (that is, the pronunciation of the second character is "che1", and the pronunciation of the fifth character is "di4"). The prediction model 500 can first encode the pronunciation of the target characters in the reference result (that is, the second character and the fifth character) through a dictionary, and input the encoded reference result to the task layer 430-2 corresponding to Task 2. The task layer 430-2 can only predict the pronunciation of other characters (that is, characters other than the second and fifth characters in the text to be processed), and output the predicted pronunciation of other characters: [da3,,qu4,mu4,,di4,*], and then combine it with the reference result to generate an output prediction result [da3,che1,qu4,mu4,di4,di4,*].

[0087] Figure 6 is another schematic diagram of the prediction model 600 according to some embodiments of the present specification.

[0088] like Figure 6 As shown, the prediction model 600 may include an embedding layer 410 , a feature layer 420 , multiple task layers 430 , and a fusion layer 610 .

[0089] In some embodiments, the task layer 430 does not input the reference result and directly determines the initial prediction results of multiple tasks based on the input feature vectors. The fusion layer 610 can be used to determine the prediction results of multiple tasks based on the initial prediction results of multiple tasks output by the multiple task layers 430 and the reference results of the preset tasks. Among the prediction results of multiple tasks, the prediction result of the target character or target string in the preset task is the reference result, and the prediction results of other characters or character strings are the initial prediction results.

[0090] In some embodiments, there can be one fusion layer 610, and the reference results of all preset tasks are fused through this layer, so that the prediction result of this preset task is the reference result. There can be multiple fusion layers 610, and the multiple fusion layers correspond one-to-one with the multiple task layers. The initial prediction result output by each task layer is input into the corresponding fusion layer. The reference result of the preset task is input into the fusion layer corresponding to the preset task. As Figure 6 shown, the fusion layer includes the fusion layer 610-1 for fusing the reference result and the initial prediction result of task 1, the fusion layer 610-2 for fusing the reference result and the initial prediction result of task 2, and the fusion layer 610-n for fusing the reference result and the initial prediction result of task n. In some embodiments, the fusion layer can be a model such as a neural network (NN).

[0091] In some embodiments, for a certain task, when the initial prediction result of the task layer 430 for the target character is consistent with the reference result of the target character, the fusion layer 440 can directly output the initial prediction result; when the initial prediction result of the task layer 430 for the target character is inconsistent with the reference result of the target character, the task layer 430 can use the reference result of the target character to replace the initial prediction result of the target character to generate the prediction result.

[0092] For example, for the task of determining the pronunciation of the characters in the to-be-processed text "Take a taxi to the destination." (corresponding to task 2), the prediction result of the corresponding task layer 430-2 for the pronunciation of its characters is [da3, che1, qu4, mu4, di1, di4, *], and there is a reference result [[2, 5][che1, di4]] (that is, the pronunciation of the second character is "che1" and the pronunciation of the fifth character is "di4"). The initial prediction result of the fifth character "的" in the to-be-processed text is inconsistent with its reference result. The corresponding fusion layer 620-2 can use the reference result "di4" of this character to replace the initial prediction result "di1" of this character to generate the prediction result [da3, che1, qu4, mu4, di4, di4, *] for this task.

[0093] Figure 7 is a schematic diagram of the training prediction model shown in some embodiments of this specification.

[0094] In some embodiments, a basic model can be obtained through training, and then a prediction model can be obtained based on the basic model, that is, the parameters of the prediction model can be determined.

[0095] The base model is a model used to guide the training of the prediction model. The base model has a large number of layers or a large dimension of the intermediate layer. The base model has the same structural composition as the prediction model. The base model includes an embedding layer, a feature layer, and a multi-task layer. If the prediction model includes a fusion layer, the base model also includes a fusion layer.

[0096] In some embodiments, the number of feature layers in the prediction model is less than the number of feature layers in the base model.

[0097] The task prediction module 220 guides the prediction model to be trained through the basic model, so that the task prediction model 220 has performance comparable to that of the basic model, while greatly reducing the parameters of the prediction model and having good performance and generalization ability.

[0098] In some embodiments, a basic model may be obtained by training based on the first training set.

[0099] The first training set may be a collection of texts used to train the basic model. In some embodiments, the first training set may include multiple training samples and their corresponding labels, wherein the labels may include labels corresponding to multiple tasks, for example, the pronunciation of characters in the training samples, the rhythm of characters in the training samples, the type of strings in the training samples, and the emotions of strings in the training samples. In some embodiments, the labels of the training samples in the first training set may be real labels that are manually annotated. In some embodiments, the task prediction module 220 may obtain the first training set from the terminal 130, the storage device 140, or an external data source.

[0100] In some embodiments, the task prediction module 220 may input the first training set into the initial basic model, and update the parameters of the initial basic model through training until the updated initial basic model meets the preset conditions. The updated initial basic model may be designated as the basic model, wherein the preset conditions may be that the loss function of the updated initial basic model is less than a threshold, converges, or the number of training iterations reaches a threshold.

[0101] In some embodiments, the first training set may include an original labeled data set and an original unlabeled data set. In some embodiments, the task prediction module 220 may obtain a basic model by performing at least one round of training on at least one initial model based on the original labeled data set and the original unlabeled data set.

[0102] In some embodiments, the original labeled data set may be a set of training samples used to train the first initial model, wherein the training samples in the original labeled data set correspond to real labels. In some embodiments, the original unlabeled data set may be a set of unlabeled training samples. In some embodiments, the original unlabeled data set may be composed of at least one original unlabeled data subset, for example, the original unlabeled data set may include the first original unlabeled data subset, the second original unlabeled data subset, and so on.

[0103] In some embodiments, the initial model may be a pre-established large model (e.g., a model with a large number of layers or a large dimension of an intermediate layer). In some embodiments, the task prediction module 220 may obtain at least one initial model from the terminal 130, the storage device 140, or an external data source.

[0104] In some embodiments, the task prediction module 220 may obtain a basic model through multiple rounds of training based on the original labeled data set and at least one original unlabeled data subset as follows:

[0105] The first round of training process may include: training the first initial model based on the original labeled data set to obtain the first initial model after training; obtaining the labels (i.e., pseudo labels) of the training samples in the first original unlabeled data subset through the first initial model after training; training the second initial model based on the first training set to obtain the second initial model after training, wherein the first training set consists of the original labeled data set and the first original unlabeled data subset and their corresponding pseudo labels; judging whether the second initial model after training meets the preset conditions, if so, using the second initial model after training as the basic model, if not, performing the second round of training, wherein the preset conditions may be that the loss function is less than a threshold, converges, or the number of training rounds reaches a threshold, etc.

[0106] like Figure 7 As shown, the i-th (i>1) round of training process may include: obtaining pseudo labels of training samples in the i-th original unlabeled data subset 720 through the trained i-th initial model 710; training the (i+1)-th initial model 750 based on the i-th training set 760 to obtain the trained (i+1)-th initial model 770, wherein the i-th training set 760 is composed of the (i-1)-th training set 730 and the i-th original unlabeled data subset and its corresponding pseudo labels 740; judging whether the trained (i+1)-th initial model 770 meets the preset conditions, if so, using the trained (i+1)-th initial model 770 as the basic model, if not, performing the (i+1)-th round of training.

[0107] For example, the second round of training process may include: obtaining pseudo labels of training samples in the second original unlabeled data subset through the trained second initial model; training the third initial model based on the second training set to obtain the trained third initial model, wherein the second training set is composed of the first training set and the second original unlabeled data subset and their corresponding pseudo labels; judging whether the trained third initial model meets the preset conditions, if so, using the trained third initial model as the basic model, if not, performing the third round of training.

[0108] In some embodiments, the number of feature layers in the i-th initial model is greater than or equal to the number of feature layers in the (i-1)-th initial model.

[0109] In some embodiments, the task prediction module 220 can evaluate the pseudo-labels corresponding to the training samples in the i-th original unlabeled data subset, screen the training samples of the i-th original unlabeled data subset, and determine the i-th training set based on the screened i-th original unlabeled data subset. In some embodiments, the task prediction module 220 can evaluate and screen the training samples of the i-th original unlabeled data subset based on the authenticity of the pseudo-labels, and screen the training samples with authenticity to determine the i-th training set, wherein the authenticity can be used to characterize whether the pseudo-label is the actual corresponding label of the training sample. In some embodiments, the task prediction module 220 can determine the authenticity of the pseudo-label based on a machine learning model. In some embodiments, the task prediction module 220 can also determine the authenticity of the pseudo-label in other ways (e.g., manual determination).

[0110] In some embodiments, the task prediction module 220 may calculate information entropy and evaluate and screen the pseudo labels corresponding to the training samples of the i-th original unlabeled data subset based on the information entropy. The information entropy may represent the degree of uncertainty of the i-th initial model for the pseudo labels.

[0111] The following is an example of the i-th initial model after training predicting the pseudo-label corresponding to the i-th original unlabeled data subset. For each task, the i-th initial model after training can obtain the probability of each candidate pseudo-label corresponding to the training sample, and output the candidate pseudo-label with the largest probability as the pseudo-label of the training sample. For example, when the pseudo-label represents the rhythm of a sample character in the training sample, the i-th initial model can output the probability values ​​of the rhythm of the sample character as four types: #1, #2, #3, and *, and the probability value of #2 is the largest, then #2 is used as the pseudo-label of the sample character. In some embodiments, the closer the probability values ​​of different candidate pseudo-labels are, the greater the information entropy.

[0112] For example, the task prediction module 220 may calculate the information entropy of the pseudo label of a training sample in the i-th original unlabeled data subset based on the following formula:

[0113]

[0114] Among them, m is the total number of candidate pseudo-labels for the task, j is the jth candidate pseudo-label, and pj is the probability of the jth candidate pseudo-label determined by the i-th initial model of the training set.

[0115] In some embodiments, the task prediction module 220 may determine the i-th training set based on training samples of the i-th original unlabeled data subset whose information entropy of characters or strings is less than a preset threshold.

[0116] In some cases, the basic model can be trained by using the original labeled data set and the original unlabeled data set to train the basic model in a semi-supervised manner, which can reduce the cost of obtaining training samples. By evaluating the pseudo labels, the training samples of the training model are more reliable, improving the efficiency and accuracy of training the basic model.

[0117] In some embodiments, the task prediction module 220 can obtain a third training set, determine pseudo labels corresponding to multiple training samples of the third training set based on the basic model, and determine parameters of the prediction model based on the pseudo labels and true labels corresponding to the multiple training samples of the third training set.

[0118] In some embodiments, the third training set may include multiple training samples and their corresponding true labels. In some embodiments, the task prediction module may obtain the third training set from the terminal 130, the storage device 140, or an external data source. In some embodiments, the task prediction module 220 may obtain the third training set from the first training set. For example, the task prediction module 220 may obtain multiple training samples and their corresponding true labels from the original labeled data set as the third training set. For another example, the task prediction module 220 may obtain multiple training samples and their corresponding pseudo labels from the filtered original unlabeled data set as the third training set. In some embodiments, the third training set may also be independent of the first training set.

[0119] In some embodiments, the task prediction module 220 can input the third training set into the initial prediction model, the initial prediction model can output the prediction results of multiple training samples of the third training set, and update the parameters of the initial prediction model based on the prediction results, pseudo labels and true labels of the multiple training samples to obtain a prediction model.

[0120] In some embodiments, the task prediction module 220 may construct a first loss term based on the prediction results and pseudo labels of multiple training samples of the third training set, wherein the first loss term may be used to characterize the difference between the prediction results and pseudo labels of the multiple training samples.

[0121] In some embodiments, the task prediction module 220 may construct a second loss term based on the prediction results and true labels of multiple training samples of the third training set, wherein the first loss term may be used to characterize the difference between the prediction results and true labels of multiple training samples.

[0122] In some embodiments, the task prediction module 220 may construct the first loss term and / or the second loss term through a Focal Loss function to solve the problem of uneven sample categories in the third training set.

[0123] In some embodiments, the task prediction module 220 may fuse the loss function of the prediction model constructed based on the first loss term and the second loss term, and update the parameters of the prediction model based on the loss function. The fusion method may include summation, weighted summation, etc. In some embodiments, the task prediction module 220 may combine the RAAdam optimizer with the AMSGrad optimizer to optimize the loss function of the prediction model.

[0124] In some embodiments, the efficiency of training the initial prediction model can be improved by optimizing the loss function of the prediction model through the RAAdam optimizer and the AMSGrad optimizer.

[0125] In some embodiments, when the initial prediction model after parameter update meets preset conditions, the initial prediction model can be used as a prediction model for processing the text to be processed, wherein the preset conditions may be that the loss function of the initial prediction model after parameter update is less than a threshold, converges, or the number of training iterations reaches a threshold, etc.; when the initial prediction model after parameter update does not meet the preset conditions, the next round of training of the initial prediction model is required.

[0126] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0127] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0128] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0129] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0130] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0131] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0132] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A speech synthesis method, characterized in that: include: Get the text to be processed; Based on the text to be processed, prediction results of multiple tasks are determined, wherein the multiple tasks include at least two of the rhythm of characters in the text to be processed, the pronunciation of characters in the text to be processed, the type of character string in the text to be processed, and the emotion of the character string in the text to be processed. The step of determining the prediction results of multiple tasks based on the text to be processed includes: The text to be processed is processed by a prediction model to determine the prediction results of the multiple tasks, wherein: The prediction model is a machine learning model, and the prediction model includes multiple task layers, and the multiple task layers correspond to the multiple tasks one by one; Based on the prediction results of the multiple tasks, the speech corresponding to the text to be processed is determined.

2. The method according to claim 1, characterized in that The processing of the to-be-processed text by using a prediction model to determine the prediction results of the multiple tasks includes: Obtaining reference results of preset tasks among the multiple tasks; The text to be processed and the reference results of the preset tasks are processed by a prediction model to determine the prediction results of the multiple tasks.

3. The method according to claim 2, characterized in that The step of processing the text to be processed and the reference results of the preset tasks by using a prediction model to determine the prediction results of the multiple tasks includes: Based on the text to be processed, determining initial prediction results of the multiple tasks; The prediction results of the plurality of tasks are determined by replacing at least a portion of the initial prediction results of the preset tasks based on the reference results of the preset tasks.

4. The method according to claim 1, wherein the prediction model comprises: An embedding layer, a feature layer, and the plurality of task layers, wherein: The embedding layer encodes the characters in the text to be processed to obtain the encoding of the characters in the text to be processed; The feature layer extracts feature vectors of characters in the text to be processed based on the encoding of the characters in the text to be processed; The multiple task layers determine prediction results of the multiple tasks based on feature vectors of characters in the text to be processed. The method according to claim 4 , wherein the number of the characteristic layers is greater than 1.

6. According to the method of claim 1, the prediction model is obtained by: A basic model is obtained by training based on a first training set, where the first training set includes a plurality of training samples and their corresponding labels; Based on the basic model, the parameters of the prediction model are determined; wherein, The number of feature layers in the prediction model is smaller than the number of feature layers in the basic model.

7. A speech synthesis system, comprising: A text acquisition module is used to acquire the text to be processed; A task prediction module is used to determine the prediction results of multiple tasks based on the text to be processed, wherein the multiple tasks include at least two of the rhythm of the characters in the text to be processed, the pronunciation of the characters in the text to be processed, the type of the character string in the text to be processed, and the emotion of the character string in the text to be processed. The determining the prediction results of multiple tasks based on the text to be processed includes: The text to be processed is processed by a prediction model to determine the prediction results of the multiple tasks, wherein: The prediction model is a machine learning model, and the prediction model includes multiple task layers, and the multiple task layers correspond to the multiple tasks one by one; A result generation module is used to determine the speech corresponding to the text to be processed based on the prediction results of the multiple tasks.

8. A speech synthesis device, comprising a processor, wherein the processor is used to execute the speech synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the speech synthesis method according to any one of claims 1 to 6.

Images