A network training method, a data processing method and related devices

Abstract

Embodiments of the present application disclose a network training method, a data processing method and related equipment for a speech synthesis scene. The method comprises: obtaining a first text and a first speech corresponding to the first text; obtaining a first phoneme sequence of the first text; obtaining a correspondence between the first speech and the first phoneme sequence based on an attention mechanism; correcting the correspondence based on a dynamic programming method to obtain first duration information of each phoneme in the first phoneme sequence; training a first prediction network based on the first phoneme sequence and the first duration information to obtain a trained first prediction network, the trained first prediction network being used to predict duration information of each phoneme in a text to be processed. Since the dynamic programming can deduce unaligned phonemes through monotonicity and the like, the duration information of the phonemes is obtained through the attention mechanism and the dynamic programming method, thereby reducing the omission or misplacement of the phonemes and improving the listening experience of the synthesized speech in the speech synthesis scene.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of speech synthesis, and in particular to a network training method, a data processing method and related equipment. BACKGROUND

[0002] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. The research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, AI basic theory, etc.

[0003] At present, with the continuous development of multimedia communication technology, speech synthesis technology as one of the important ways of human-computer communication has been widely concerned by researchers due to its convenience and speed. In the end-to-end speech synthesis (text to speech, TTS) scene, in order to ensure that the synthesized speech has good listening, the model of TTS needs to correct the correspondence between phonemes and speech in the training process, so as to correct the speech output by the model according to the correspondence. The common way to correct the correspondence between phonemes and speech is forced alignment and the like.

[0004] However, in the forced alignment method, the alignment effect of phonemes and speech may be poor, that is, the omission of phonemes may occur. SUMMARY

[0005] Embodiments of the present application provide a network training method, a data processing method and related equipment, which obtain the duration information of phonemes through attention mechanism and dynamic programming method, thereby reducing the omission of phonemes and improving the listening of synthesized speech in the speech synthesis scene.

[0006] The first aspect of the embodiment of the present application provides a network training method. The method can be executed by a data processing device (for example, a terminal device or a server) or a component (for example, a processor, a chip or a chip system) of the data processing device. The method comprises the following steps: obtaining a first text and a first speech corresponding to the first text; obtaining a first phoneme sequence of the first text; obtaining a corresponding relationship between the first speech and the first phoneme sequence based on an attention mechanism, the corresponding relationship being used to represent a time length of each phoneme in the first phoneme sequence in the first speech (or the phoneme occupies a frame number of the first speech, or the phoneme occupies a time length in the first speech); correcting the corresponding relationship based on a dynamic programming method to obtain first time length information of each phoneme in the first phoneme sequence; and training a first prediction network based on the first phoneme sequence and the first time length information to obtain a trained first prediction network, the trained first prediction network being used to predict time length information of each phoneme in a to-be-processed text. The first speech can be a monolingual / dialect speech, or a speech including at least two languages / dialects, which is not limited herein.

[0007] In the embodiment, the first time length information of each phoneme in the first text can be obtained based on the attention mechanism and the dynamic programming method. Since the dynamic programming can deduce unaligned phonemes through monotonicity and the like, the probability of phonemes being estimated incorrectly (for example, phoneme transposition or phoneme swallowing) can be reduced. Furthermore, the trained first prediction network can realize time length prediction of phonemes in the to-be-processed text, which is convenient for application in a scenario requiring phoneme time length information, such as speech synthesis.

[0008] Optionally, in a possible implementation manner of the first aspect, the step of training the first prediction network based on the first phoneme sequence and the first time length information comprises the following steps: taking the first phoneme sequence as an input of the first prediction network; and training the first prediction network to obtain the trained first prediction network, with a value of a first loss function being less than a first threshold as a target, the first loss function being used to represent a difference between time length information output by the first prediction network and the first time length information.

[0009] In the possible implementation manner, by continuously reducing the difference between the time length information output by the first prediction network and the first time length information, the accuracy of the prediction network in predicting phoneme time length information can be improved.

[0010] Optionally, in a possible implementation manner of the first aspect, the first speech includes speech of at least two types of languages / dialects / minority languages, and the method further includes: obtaining second text and second speech corresponding to the second text, the second speech including speech of one type of language / dialect / minority language in the at least two types of languages / dialects / minority languages; obtaining a second phoneme sequence of the second text; obtaining second duration information of each phoneme in the second phoneme sequence; taking the second phoneme sequence as input of the second prediction network, and training the second prediction network to obtain the first prediction network, with a value of a second loss function being less than a second threshold as a target, the second loss function being used to represent a difference between duration information output by the second prediction network and the second duration information.

[0011] In the possible implementation manner, a large amount of data set of the first speech of one speaker is required, that is, a high requirement is imposed on the language / dialect of the speaker. To solve the problem, a large amount of single data set (that is, a data set corresponding to one language / dialect) can be obtained before the first prediction network is trained by using a small amount of mixed data set (the first text and the first speech), and the second prediction network is trained by using the single data set to obtain the first prediction network, and the first prediction network is further trained by using the mixed data set to obtain the trained first prediction network.

[0012] Optionally, in a possible implementation manner of the first aspect, the method further includes: obtaining first mel-spectrogram features of the first speech; obtaining first pronunciation features, the first pronunciation features being used to describe timbre features of the first speech; taking the first phoneme sequence, the first duration information, and the first pronunciation features as input of the first speech synthesis network, and training the first speech synthesis network to obtain the trained first speech synthesis network and the trained first pronunciation features, with a value of a third loss function being less than a third threshold as a target, the third loss function being used to represent a difference between second mel-spectrogram features output by the first speech synthesis network and the first mel-spectrogram features, the second mel-spectrogram features being obtained by expanding the first duration information.

[0013] In the possible implementation manner, the first speech synthesis network can be trained by using the first phoneme sequence, the first pronunciation features, and the first duration information, so that the trained first speech synthesis network can realize speech synthesis across languages or across dialects.

[0014] Optionally, in a possible implementation manner of the first aspect, the first speech synthesis network comprises an encoder and an autoregressive decoder; the first speech synthesis network is trained based on the first phoneme sequence and the first duration information, and the trained first speech synthesis network is obtained by taking the value of the third loss function being less than a third threshold as a target, and the training comprises: obtaining a first feature corresponding to the first phoneme sequence based on the encoder; extending the first feature based on the first duration information to obtain a second feature; obtaining a second mel-spectrogram feature based on the autoregressive decoder and the second feature; and training the encoder and the autoregressive decoder by taking the value of the third loss function being less than the third threshold as a target, to obtain the trained first speech synthesis network.

[0015] In this possible implementation manner, the dynamic programming can deduce the misaligned phonemes through monotonicity or the like, so as to reduce the probability of the phonemes being wrongly estimated (for example, phoneme transposition or phoneme swallowing). On the one hand, the first feature can be corrected by the first duration information, so that the speech output by the trained first speech synthesis network reduces the poor speech listening experience caused by the phonemes being wrongly estimated (for example, phoneme transposition or phoneme swallowing). On the other hand, the autoregressive model controlled according to the first duration information is established, so as to realize high naturalness and strong robustness of speech synthesis.

[0016] Optionally, in a possible implementation manner of the first aspect, the step of obtaining the second mel-spectrogram feature based on the autoregressive decoder and the second feature comprises: inputting the second feature into the autoregressive decoder to obtain the second mel-spectrogram feature.

[0017] In this possible implementation manner, on the one hand, the mel-spectrogram feature generated by introducing the encoder and the autoregressive decoder is more accurate. On the other hand, the autoregressive model controlled according to the first duration information is established, so as to realize high naturalness and strong robustness of speech synthesis.

[0018] Optionally, in a possible implementation manner of the first aspect, the step of obtaining the second mel-spectrogram feature based on the autoregressive decoder and the second feature comprises: performing convolution processing on the second feature to obtain a third feature; and inputting the third feature into the autoregressive decoder to obtain the second mel-spectrogram feature.

[0019] In this possible implementation manner, the convolution processing of the third feature is introduced, so that the fourth feature obtained is more perfect, or the fourth feature takes the features of the entire first phoneme sequence or first text into account.

[0020] Optionally, in a possible implementation manner of the first aspect, the first voice includes voices of at least two types of languages / dialects; the method further includes: obtaining third text and third voice corresponding to the third text, the third voice including voices of one type of language / dialect in the at least two types of languages / dialects; obtaining a third phoneme sequence of the third text; obtaining third duration information of each phoneme in the third phoneme sequence; obtaining second pronunciation features, the second pronunciation features being used to describe timbre features of the third voice; obtaining third mel-frequency spectrum features of the third voice; taking the third phoneme sequence, the third duration information, and the second pronunciation features as inputs of the second voice synthesis network, and taking a value of a fourth loss function being less than a fourth threshold as a target to train the second voice synthesis network to obtain the first voice synthesis network and the trained second pronunciation features, the fourth loss function being used to represent a difference between fourth mel-frequency spectrum features output by the second voice synthesis network and the third mel-frequency spectrum features, the fourth mel-frequency spectrum features being obtained after being expanded according to the third duration information.

[0021] In the possible implementation manner, a large amount of data set of the first voice of a speaker is required, that is, a high requirement is required for the language / dialect of the speaker. To solve the problem, a large amount of single data set (that is, a data set corresponding to one language / dialect) can be obtained before the first voice synthesis network is trained using a small amount of mixed data set (the first text and the first voice), and the first voice synthesis network is trained using the single data set to obtain the first voice synthesis network, and the first voice synthesis network is further trained using the mixed data set to obtain the trained first voice synthesis network.

[0022] Optionally, in a possible implementation manner of the first aspect, the dynamic programming method includes a monotonic alignment search (MAS) method or a Needleman-Wunsch algorithm.

[0023] In the possible implementation manner, by applying the MAS method or the Needleman-Wunsch algorithm in the scenario of correcting the correspondence between the voice and the phoneme, correct alignment between the phoneme and the voice can be obtained, and the voice with good hearing sense can be obtained.

[0024] The second aspect of the embodiments of the present application provides a data processing method, which can be executed by a data processing device (for example, a terminal device or a server) or a component (for example, a processor, a chip or a chip system) of the data processing device. The method comprises the following steps: obtaining a to-be-processed text; obtaining a phoneme sequence of the to-be-processed text based on the to-be-processed text; and predicting the duration information of each phoneme in the phoneme sequence based on a trained prediction network, wherein the trained prediction network is trained based on a first text and first duration information, the first duration information is obtained by correcting the correspondence between the first text and a first speech by a dynamic programming method, and the first speech is the speech of the first text. The correspondence is used to represent the duration of each phoneme in the phoneme sequence in the first speech (or it can be understood as the number of frames of the phoneme in the first speech, or the duration of the phoneme in the first speech). The first speech can refer to the speech of a single language / dialect, or can refer to the speech including at least two languages / dialects, and the specific implementation is not limited herein. Similarly, the to-be-processed text can be a text corresponding to a single language / dialect, or a text corresponding to at least two languages / dialects.

[0025] In the embodiments, the duration information of each phoneme in the phoneme sequence obtained by introducing the prediction network in the inference stage can avoid the problem of inaccurate duration prediction caused by phoneme omission or phoneme misplacement, because the duration information is obtained by correcting the correspondence between the first text and the first speech based on the dynamic programming method.

[0026] Optionally, in a possible implementation manner of the second aspect, the above-mentioned step further comprises: inputting the phoneme sequence and the duration information into a trained speech synthesis network to obtain a mel-spectrogram feature, wherein the trained speech synthesis network is used to generate the speech corresponding to the text; and converting the mel-spectrogram feature into the speech by a vocoder, and the speech is the speech of the to-be-processed text by the speaker.

[0027] In the possible implementation manner, the duration information can be applied to the speech synthesis scene, so that there is no phoneme omission or misplacement in the generated speech, and the listening experience of the synthesized speech is improved.

[0028] Optionally, in a possible implementation manner of the second aspect, the above-mentioned step of obtaining the speaker feature and inputting the phoneme sequence and the duration information into the trained speech synthesis network to obtain the mel-spectrogram feature comprises: inputting the phoneme sequence, the duration information and the speaker feature into the trained speech synthesis network to obtain the mel-spectrogram feature.

[0029] In the possible implementation manner, by introducing the speaker feature, the speech generated by the speech synthesis network can be more consistent with the timbre corresponding to the speaker feature, so that the timbre of the speech is consistent with the timbre of the speaker feature.

[0030] Optionally, in a possible implementation manner of the second aspect, the trained speech synthesis network comprises an encoder and an autoregressive decoder; and the inputting the phoneme sequence, the duration information and the pronunciation feature into the trained speech synthesis network comprises: obtaining a first feature corresponding to the phoneme sequence based on the encoder; extending the first feature based on the duration information to obtain a second feature; obtaining a third feature based on the second feature and the pronunciation feature; and obtaining the mel-spectrogram feature based on the autoregressive decoder and the third feature.

[0031] In this possible implementation manner, since the prediction network is trained by the dynamic programming method, the dynamic programming method can deduce unaligned phonemes through monotonicity and the like, so as to improve the duration effect of the prediction network, reduce the probability of phonemes being swallowed, and correct the output speech of the model through the duration information.

[0032] Optionally, in a possible implementation manner of the second aspect, the obtaining the mel-spectrogram feature based on the autoregressive decoder and the third feature comprises: inputting the third feature into the autoregressive decoder to obtain the mel-spectrogram feature.

[0033] In this possible implementation manner, on the one hand, the mel-spectrogram feature generated by introducing the encoder and the autoregressive decoder is more accurate. On the other hand, the autoregressive model controlled according to the first duration information is established, so as to realize high naturalness and strong robustness of speech synthesis.

[0034] Optionally, in a possible implementation manner of the second aspect, the obtaining the mel-spectrogram feature based on the autoregressive decoder and the third feature comprises: performing convolution processing on the third feature to obtain a fourth feature; and inputting the fourth feature into the autoregressive decoder to obtain the mel-spectrogram feature.

[0035] In this possible implementation manner, by introducing the convolution processing on the third feature, the fourth feature obtained is more perfect, or the fourth feature takes into account the features of the entire text to be processed.

[0036] Optionally, in a possible implementation manner of the second aspect, the text to be processed comprises texts of at least two languages / dialects, and the first feature is further used to describe a language / dialect to which the phoneme belongs.

[0037] In this possible implementation manner, the method can be applied to a multi-language / dialect scene, and the duration information of the multi-language / dialect text can be obtained based on the prediction network, so as to improve the correct prediction of the correspondence between the phonemes and the speech.

[0038] Optionally, in a possible implementation manner of the second aspect, the dynamic programming method comprises a monotonic alignment search (MAS) method or a Viterbi algorithm.

[0039] In the possible implementation manner, by applying the MAS method or the Nieddman-Ons algorithm in the scenario of correcting the correspondence between the speech and the phonemes, a correct phoneme-speech alignment effect can be obtained, and thus a speech with better listening experience can be obtained.

[0040] Optionally, in a possible implementation manner of the second aspect, the step of obtaining the text to be processed includes: receiving the text to be processed sent by the terminal device; and the method further includes: sending the speech to the terminal device.

[0041] In the possible implementation manner, the data processing device is a server, and by placing the network-related part on the server in the inference process, the storage and computing power of the terminal device can be saved.

[0042] The third aspect of the embodiments of the present application provides a data processing device, which can be a terminal device or a server. The data processing device includes: an obtaining unit, configured to obtain a first text and a first speech corresponding to the first text; the obtaining unit is further configured to obtain a first phoneme sequence of the first text; the obtaining unit is further configured to obtain a correspondence between the first speech and the first phoneme sequence based on an attention mechanism, the correspondence being used to represent the duration of each phoneme in the first phoneme sequence in the first speech (or the phoneme occupies the frame number of the first speech, or the duration of the phoneme in the first speech); a correction unit, configured to correct the correspondence based on a dynamic programming method to obtain first duration information of each phoneme in the first phoneme sequence; a training unit, configured to train a first prediction network based on the first phoneme sequence and the first duration information to obtain a trained first prediction network, the trained first prediction network being used to predict duration information of each phoneme in a text to be processed. The first speech can be a monolingual / dialect speech, or can be a speech including at least two languages / dialects, which is not limited here.

[0043] Optionally, in a possible implementation manner of the third aspect, the correction unit is specifically configured to take the first phoneme sequence as the input of the first prediction network, and train the first prediction network to obtain the trained first prediction network with the value of the first loss function being less than the first threshold, the first loss function being used to represent the difference between the duration information output by the first prediction network and the first duration information.

[0044] Optionally, in a possible implementation manner of the third aspect, the first voice includes voice of at least two types of languages / dialects; the obtaining unit is further configured to obtain second text and second voice corresponding to the second text, the second voice including voice of one type of language / dialect in the at least two types of languages / dialects; the obtaining unit is further configured to obtain a second phoneme sequence of the second text; the obtaining unit is further configured to obtain second duration information of each phoneme in the second phoneme sequence; and the training unit is further configured to take the second phoneme sequence as input of a second prediction network, and train the second prediction network to obtain the first prediction network, with a value of a second loss function being less than a second threshold, the second loss function being used to represent a difference between duration information output by the second prediction network and the second duration information.

[0045] Optionally, in a possible implementation manner of the third aspect, the obtaining unit is further configured to obtain first mel-spectrogram features of the first voice; the obtaining unit is further configured to obtain first pronunciation features, the first pronunciation features being used to describe timbre features of the first voice; and the training unit is further configured to take the first phoneme sequence, the first duration information, and the first pronunciation features as input of a first voice synthesis network, and train the first voice synthesis network to obtain a trained first voice synthesis network and trained first pronunciation features, with a value of a third loss function being less than a third threshold, the third loss function being used to represent a difference between second mel-spectrogram features output by the first voice synthesis network and the first mel-spectrogram features, the second mel-spectrogram features being obtained by expanding the first duration information.

[0046] Optionally, in a possible implementation manner of the third aspect, the first voice synthesis network includes an encoder and an autoregressive decoder; the training unit is specifically configured to obtain first features corresponding to the first phoneme sequence based on the encoder; the training unit is specifically configured to expand the first features to obtain second features based on the first duration information; the training unit is specifically configured to obtain the second mel-spectrogram features based on the autoregressive decoder and the second features; and the training unit is specifically configured to train the encoder and the autoregressive decoder, with a value of the third loss function being less than the third threshold, to obtain the trained first voice synthesis network.

[0047] Optionally, in a possible implementation manner of the third aspect, the training unit is specifically configured to input the second features into the autoregressive decoder to obtain the second mel-spectrogram features.

[0048] Optionally, in a possible implementation manner of the third aspect, the training unit is specifically configured to perform convolution processing on the second features to obtain third features; and the training unit is specifically configured to input the third features into the autoregressive decoder to obtain the second mel-spectrogram features.

[0049] Optionally, in a possible implementation manner of the third aspect, the first voice includes voice of at least two types of languages / dialects; the obtaining unit is further configured to obtain third text and third voice corresponding to the third text, the third voice including voice of one type of language / dialect in the at least two types of languages / dialects; the obtaining unit is further configured to obtain a third phoneme sequence of the third text; the obtaining unit is further configured to obtain third duration information of each phoneme in the third phoneme sequence; the obtaining unit is further configured to obtain second pronunciation features, the second pronunciation features being used to describe timbre features of the third voice; the obtaining unit is further configured to obtain third mel-frequency spectrum features of the third voice; the training unit is further configured to take the third phoneme sequence, the third duration information, and the second pronunciation features as inputs of the second voice synthesis network, and train the second voice synthesis network to obtain the first voice synthesis network and the trained second pronunciation features, with a value of a fourth loss function being less than a fourth threshold value as a target, the fourth loss function being used to represent a difference between fourth mel-frequency spectrum features output by the second voice synthesis network and the third mel-frequency spectrum features, the fourth mel-frequency spectrum features being obtained after being expanded according to the third duration information.

[0050] Optionally, in a possible implementation manner of the third aspect, the dynamic programming method includes a monotonic alignment search (MAS) method or a Needleman-Wunsch algorithm.

[0051] The fourth aspect of the embodiments of the present application provides a data processing device, which can be a terminal device or a server. The data processing device includes: an obtaining unit configured to obtain to-be-processed text; the obtaining unit is further configured to obtain a phoneme sequence based on the to-be-processed text; the obtaining unit is further configured to obtain pronunciation features, the pronunciation features being used to describe timbre features of a pronouncer; and a prediction unit configured to predict duration information of each phoneme in the phoneme sequence based on a trained prediction network, the trained prediction network being trained based on first text and first duration information, the first duration information being obtained by correcting a correspondence between the first text and first voice by a dynamic programming method, the first voice being voice of the first text, and the correspondence being used to represent duration (or understood as a number of frames of voice occupied by the phoneme in the first voice, or duration of voice occupied by the phoneme in the first voice) of each phoneme in the phoneme sequence in the first voice. The first voice can be voice of a single language / dialect, or voice including at least two types of languages / dialects, which is not limited herein. Similarly, the to-be-processed text can be text corresponding to a single language / dialect, or text corresponding to at least two types of languages / dialects.

[0052] Optionally, in a possible implementation manner of the fourth aspect, the data processing device further includes: a processing unit configured to input the phoneme sequence and the duration information into the trained voice synthesis network to obtain mel-frequency spectrum features; and a conversion unit configured to convert the mel-frequency spectrum features into voice by a vocoder, the voice being voice of the pronouncer for the to-be-processed text.

[0053] Optionally, in a possible implementation manner of the fourth aspect, the obtaining unit is further configured to obtain pronunciation features, the pronunciation features being used to describe timbre features of a pronouncer; and the processing unit is specifically configured to input the phoneme sequence, the duration information, and the pronunciation features into the trained speech synthesis network to obtain the mel spectrum features.

[0054] Optionally, in a possible implementation manner of the fourth aspect, the trained speech synthesis network comprises an encoder and an autoregressive decoder; the processing unit is specifically configured to obtain first features corresponding to the phoneme sequence based on the encoder; the processing unit is specifically configured to extend the first features to obtain second features based on the duration information; the processing unit is specifically configured to obtain third features based on the second features and the pronunciation features; and the processing unit is specifically configured to obtain the mel spectrum features based on the autoregressive decoder and the third features.

[0055] Optionally, in a possible implementation manner of the fourth aspect, the processing unit is specifically configured to input the third features into the autoregressive decoder to obtain the mel spectrum features.

[0056] Optionally, in a possible implementation manner of the fourth aspect, the processing unit is specifically configured to perform convolution processing on the third features to obtain fourth features; and the processing unit is specifically configured to input the fourth features into the autoregressive decoder to obtain the mel spectrum features.

[0057] Optionally, in a possible implementation manner of the fourth aspect, the text to be processed comprises texts in at least two types of languages / dialects, and the first features are further used to describe a language / dialect to which the phonemes belong.

[0058] Optionally, in a possible implementation manner of the fourth aspect, the dynamic programming method comprises a monotonic alignment search (MAS) method or a Needleman-Wunsch algorithm.

[0059] The fifth aspect of the present application provides a data processing device, which executes the method in the first aspect or any possible implementation manner of the first aspect, or executes the method in the second aspect or any possible implementation manner of the second aspect.

[0060] The sixth aspect of the present application provides a data processing device, comprising: a processor, the processor being coupled with a memory, the memory being used to store programs or instructions, when the programs or instructions are executed by the processor, the data processing device is caused to implement the method in the first aspect or any possible implementation manner of the first aspect, or implement the method in the second aspect or any possible implementation manner of the second aspect.

[0061] The seventh aspect of the present application provides a computer readable medium having stored thereon a computer program or instructions which, when executed on a computer, cause the computer to perform the method in the first aspect or any possible implementation of the first aspect, or cause the computer to perform the method in the second aspect or any possible implementation of the second aspect.

[0062] The eighth aspect of the present application provides a computer program product which, when executed on a computer, causes the computer to perform the method in the first aspect or any possible implementation of the first aspect, or causes the computer to perform the method in the second aspect or any possible implementation of the second aspect.

[0063] The technical effects brought by the third, fifth, sixth, seventh, eighth aspects or any possible implementation thereof can be referred to the technical effects brought by the first aspect or different possible implementations of the first aspect, which will not be repeated here.

[0064] The technical effects brought by the fourth, fifth, sixth, seventh, eighth aspects or any possible implementation thereof can be referred to the technical effects brought by the second aspect or different possible implementations of the second aspect, which will not be repeated here.

[0065] From the above technical solutions, it can be seen that the embodiments of the present application have the following advantages: the first duration information of each phoneme in the first text can be obtained based on the attention mechanism and the dynamic programming method. Since dynamic programming can infer unaligned phonemes through monotonicity and other methods, the probability of phoneme being estimated incorrectly (for example, phoneme misplacement or phoneme being swallowed) can be reduced. In turn, the correct alignment of phonemes in the first text and the first speech is achieved. This facilitates the application in scenarios requiring phoneme duration information, such as speech synthesis. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 A structural schematic diagram of a system architecture provided by the present application;

[0067] Figure 2 A structural schematic diagram of a convolutional neural network provided by the present application;

[0068] Figure 3 Another structural schematic diagram of a convolutional neural network provided by the present application;

[0069] Figure 4 A structural schematic diagram of a chip hardware provided by the present application;

[0070] Figure 5 A flowchart of a network training method provided by the present application;

[0071] Figure 6A structural diagram of an attention network provided in the present application;

[0072] Figure 7 A pre-revised speech text alignment diagram provided in the present application;

[0073] Figure 8 A revised speech text alignment diagram provided in the present application;

[0074] Figure 9 A flow diagram of another network training method provided in the present application;

[0075] Figure 10 A structural diagram of a speech synthesis network provided in the present application;

[0076] Figure 11 Another structural diagram of a speech synthesis network provided in the present application;

[0077] Figure 12 A training architecture diagram of a first network and a second network provided in the present application;

[0078] Figure 13 Another training architecture diagram of a first network and a second network provided in the present application;

[0079] Figure 14 A relationship structural diagram of a speech synthesis network and an attention network provided in the present application;

[0080] Figure 15 A flow diagram of a data processing method provided in the present application;

[0081] Figure 16 Another structural diagram of a speech synthesis network provided in the present application;

[0082] Figure 17 Another flow diagram of a data processing method in an embodiment of the present application;

[0083] Figures 18 to 21 Several structural diagrams of a data processing device in an embodiment of the present application. DETAILED DESCRIPTION

[0084] The embodiments of the present application provide a network training method, a data processing method and related devices, time length information of phonemes is obtained through an attention mechanism and a dynamic programming method, so as to reduce omission of phonemes and improve the listening experience of synthesized speech in a speech synthesis scenario.

[0085] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0086] For the convenience of understanding, the related terms and concepts mainly involved in the embodiments of the present application will be introduced first.

[0087] 1. Neural network

[0088] The neural network can be composed of neural units, and the neural unit can refer to an operation unit with X s and intercept 1 as input. The output of the operation unit can be:

[0089]

[0090] wherein s = 1, 2, … n, n is a natural number greater than 1, W s is the weight of X s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce a nonlinear characteristic into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. The neural network is a network formed by connecting many single neural units described above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0091] 2. Deep neural network

[0092] The deep neural network (DNN) is also called a multi-layer neural network, which can be understood as a neural network with many hidden layers. Here, “many” has no special measurement standard. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the number of layers in between is the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer. Of course, the deep neural network can also not include the hidden layer, which is not limited here.

[0093] The work of each layer in the deep neural network can be expressed by a mathematical expression The working of each layer in a deep neural network can be understood as a transformation from an input space (a set of input vectors) to an output space (i.e. the row space to the column space of a matrix) by five operations on the input space, which include: 1. dimensionality increase / decrease; 2. scaling up / down; 3. rotation; 4. translation; 5. "bending". The operations of 1, 2, and 3 are performed by , the operation of 4 is performed by , and the operation of 5 is performed by a(). The reason why "space" is used here is that the objects to be classified are not single things, but a class of things, and the space refers to the set of all individuals of this class. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in the layer neural network. The vector W determines the space transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a deep neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially learning the way to control the space transformation, more specifically, learning the weight matrix.

[0094] 3. Convolutional neural network

[0095] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network includes a feature extractor composed of convolutional layers and subsampling layers. The feature extractor can be regarded as a filter, and the convolution process can be regarded as making the same trainable filter convolve with an input image or a convolution feature plane. A convolutional layer refers to a layer of neurons in a convolutional neural network that performs convolution processing on an input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to part of the adjacent layer neurons. A convolutional layer usually includes several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units of the same feature plane share weights, and the shared weights are convolution kernels. Shared weights can be understood as the way of extracting image information being independent of the position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts, which means that the image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolutional layer, multiple convolution kernels can be used to extract different image information. Generally, the more the number of convolution kernels, the more image information the convolution operation reflects.

[0096] The convolution kernel can be initialized in the form of a matrix with random size, and the convolution kernel can obtain reasonable weights through learning in the training process of the convolution neural network. In addition, the direct benefit of sharing weights is to reduce the connections between layers of the convolution neural network, while reducing the risk of overfitting. The separation network, the identification network, the detection network, the depth estimation network and the like in the embodiments of the present application can all be CNNs.

[0097] 4. Recurrent neural network (RNN)

[0098] In a traditional neural network model, layers are fully connected, and nodes between layers are unconnected. However, such a general neural network cannot solve many problems. For example, predicting the next word of a sentence, because the words in a sentence are not independent, the previous words are generally needed. The recurrent neural network (RNN) refers to the current output of a sequence being related to the previous output. The specific form is that the network memorizes the previous information in the internal state of the network and applies it to the calculation of the current output.

[0099] 5. Loss function

[0100] In the process of training a deep neural network, because it is desired that the output of the deep neural network is as close as possible to the value that is truly desired to be predicted, the weight vector of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the target value that is truly desired to be predicted (of course, there is usually an initialization process before the first update, that is, the parameters of each layer of the deep neural network are pre-configured), for example, if the predicted value of the network is too high, the weight vector is adjusted to make it predict lower, and the adjustment is continuously made until the neural network can predict the target value that is truly desired to be predicted. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize the loss.

[0101] 6. Teacher forcing

[0102] Force teaching is a kind of network training method, which is essential for developing deep learning language models for machine translation, text summarization, image captioning and many other applications. It does not use the output of the previous state as the input of the next state each time, but directly uses the corresponding previous item of the standard answer (ground truth) of the training data as the input of the next state. Or understand that it provides the real output of the network as the input of the next time.

[0103] 7、From text to speech

[0104] From text to speech (TTS) is a program or software system that converts text into speech.

[0105] 8、Vocoder

[0106] Vocoder is a kind of sound signal processing module or software that can generate sound waveforms from acoustic features.

[0107] 9、Autoregressive model

[0108] Autoregressive model refers to predicting future features based on past information features.

[0109] 10、Force alignment

[0110] Force alignment refers to querying the phonetic time information of each word in the audio file according to the pronunciation dictionary.

[0111] 9、Fundamental frequency

[0112] When the sound body emits sound due to vibration, the sound can generally be decomposed into many simple sine waves, that is, all natural sounds are basically composed of many sine waves with different frequencies, and the sine wave with the lowest frequency is the fundamental frequency (that is, the fundamental frequency, which can be represented by F0), and other sine waves with higher frequencies are overtones.

[0113] 10、Prosody

[0114] In the field of speech synthesis, prosody refers to the characteristics of functions such as controlling intonation, pitch, stress, pause and rhythm. Prosody can reflect the emotional state or speaking form of the speaker, etc.

[0115] 11、Phoneme

[0116] Phone: is the smallest unit of speech divided according to the natural attributes of speech, analyzed according to the pronunciation action in the syllable, one action constitutes a phone. Phones are divided into two categories: vowels and consonants. For example, the Chinese syllable a (for example, the first tone: ah) has only one phone, ai (for example, the fourth tone: love) has two phones, dai (for example, the first tone: dull) has three phones, and so on.

[0117] 12. Word vector (Embedding)

[0118] Word vector can also be called "word embedding", "vectorization", "vector mapping", "embedding", etc. Formally, a word vector is a dense vector that represents an object, for example: a vector representing a sequence of phones in a text.

[0119] Currently, in the field of multi-language speech synthesis (multi-language TTS, ML-TTS) widely used in daily life, the construction of a multi-language data set (polyglot speech corpus) is realized by converting different vocal colors of other languages into target vocal colors through voice conversion (VC) to realize multi-language speech synthesis. However, the above method generally uses an existing single-language end-to-end speech synthesis system to learn cross-language synthesis capabilities in the above data set, but this method is limited by the VC conversion effect and cannot guarantee the conversion completeness and sound quality of the target vocal color.

[0120] In order to solve the above problems, the present application provides a data processing method, which can realize the reduction of phoneme omission of the trained model, thereby improving the listening experience of the multi-language speech output by the model.

[0121] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0122] First, the system architecture provided by the embodiments of the present application is introduced.

[0123] Referring to the accompanying Figure 1The embodiment of the present application provides a system architecture 10. As shown in the system architecture 10, a data collection device 16 is used to collect training data, and the training data in the embodiment of the present application includes training speech and training text corresponding to the training speech. The training data is stored in a database 13, and a training device 12 trains a target model / rule 101 based on the training data maintained in the database 13. How the training device 12 obtains the target model / rule 101 based on the training data will be described in detail below. The target model / rule 101 can be used to implement the data processing method provided by the embodiment of the present application, that is, the text is input into the target model / rule 101 after relevant preprocessing, and then the speech information of the text by a speaker can be obtained. The target model / rule 101 in the embodiment of the present application can be a speech synthesis network. It should be noted that in actual application, the training data maintained in the database 13 can not all be collected by the data collection device 16, and can also be received from other devices. In addition, it should be noted that the training device 12 can not completely train the target model / rule 101 based on the training data maintained in the database 13, and can also obtain training data from the cloud or other places to train the model. The above description should not be regarded as a limitation of the embodiment of the present application.

[0124] The target model / rule 101 trained by the training device 12 can be applied to different systems or devices, such as an execution device 11 as shown in the figure. Figure 1 The execution device 11 can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR), a vehicle-mounted terminal, and the like, and can also be a server or a cloud, and the like. In the following Figure 1 The execution device 11 is configured with an I / O interface 112 for data interaction with an external device. A user can input data to the I / O interface 112 through a client device 14. The input data in the embodiment of the present application can include a first text and a pronunciation feature used to describe the timbre feature of a speaker. In addition, the input data can be input by the user, can be uploaded by the user through other devices, and can also come from a database, and the specific implementation is not limited here.

[0125] The preprocessing module 113 is used for preprocessing according to the first text received by the I / O interface 112, for example, converting the first text into a phoneme, predicting the prosody of the first text, normalizing the non-standard text, and the like.

[0126] During the preprocessing of input data by the execution device 11, or during the calculation module 111 of the execution device 11 performing calculations and other related processes, the execution device 11 can call data, code, etc. in the data storage system 15 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 15.

[0127] Finally, I / O interface 112 returns the processing result, such as the speaker's voice information for the first text obtained above, to client device 14, thereby providing it to the user.

[0128] It is worth noting that the training device 12 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results or providing input for other subsequent processing.

[0129] In the appendix Figure 1 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 14 can automatically send input data to I / O interface 112. If user authorization is required for the client device 14 to automatically send input data, the user can set the corresponding permissions in the client device 14. The user can view the output results of the execution device 11 on the client device 14, which can be presented in various forms such as display, sound, or animation. The client device 14 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 13. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 14, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 13.

[0130] It is worth noting that, attached Figure 1 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in the attached diagram... Figure 1 In this case, the data storage system 15 is an external memory relative to the execution device 11. In other cases, the data storage system 15 may also be placed in the execution device 11.

[0131] like Figure 1As shown, the target model / rule 101 is trained according to the training device 12, which can be a neural network in the embodiments of the present application. Specifically, in the network provided by the embodiments of the present application, the speech synthesis network can be a recurrent neural network, a long short-term memory network, etc. The prediction network can be a convolutional neural network, a recurrent neural network, etc.

[0132] Optionally, the speech synthesis network and the prediction network in the embodiments of the present application can be two separate networks, or can be a multi-task neural network, in which one task is to output the duration of each phoneme of the text, and the other task is to output the speech information corresponding to the text.

[0133] Since CNN is a very common neural network, the structure of CNN will be described in detail below. Figure 2 As described in the foregoing basic concept introduction, the convolutional neural network is a deep neural network with convolutional structure, which is a deep learning architecture, i.e., a multi-level learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the input text.

[0134] As shown, the convolutional neural network (CNN) 100 can include an input layer 110, a convolutional layer / pooling layer 120, and a neural network layer 130, in which the pooling layer is optional. Figure 2

[0135] The convolutional layer / pooling layer 120:

[0136] Convolutional layer:

[0137] As shown, the convolutional layer / pooling layer 120 can include layers such as 121-126. In one implementation, 121 is a convolutional layer, 122 is a pooling layer, 123 is a convolutional layer, 124 is a pooling layer, 125 is a convolutional layer, and 126 is a pooling layer. In another implementation, 121 and 122 are convolutional layers, 123 is a pooling layer, 124 and 125 are convolutional layers, and 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or as the input of another convolutional layer for further convolution operation. Figure 2

[0138] ​​For example, the convolution layer 121 can include a plurality of convolution operators, also known as kernels, which function as filters for extracting specific information from the phoneme sequence. The convolution operator can be essentially a weight matrix, which is usually predefined. During the convolution operation on the phonemes, the weight matrix is usually processed along the phoneme direction one phoneme after another (or two phonemes after two phonemes, depending on the value of the stride), thereby completing the work of extracting specific features from the phonemes. It should be noted that the depth dimension of the weight matrix is the same as that of the phonemes, and the weight matrix extends to the entire depth of the phonemes during the convolution operation. Therefore, convolution with a single weight matrix produces a single-depth convolution output, but most cases do not use a single weight matrix, but rather a plurality of weight matrices of the same dimension. The output of each weight matrix is stacked to form the depth dimension of the convolution phonemes. Different weight matrices can be used to extract different features from the phonemes, such as a weight matrix for extracting language / dialect features of the phonemes, another weight matrix for extracting prosody of the phonemes, and the like. The plurality of weight matrices are of the same dimension, and the feature maps extracted by the plurality of weight matrices of the same dimension are also of the same dimension, and the extracted plurality of feature maps of the same dimension are combined to form the output of the convolution operation.

[0139] The weight values in these weight matrices need to be obtained through a large amount of training in actual applications. The weight values obtained through training form each weight matrix, which can extract information from the phoneme sequence, thereby helping the convolutional neural network 100 to make correct predictions.

[0140] When the convolutional neural network 100 has multiple convolution layers, the initial convolution layer (e.g., 121) often extracts more general features, which can also be referred to as low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolution layers (e.g., 126) become more and more complex, such as high-level semantic features, and the higher the semantic features, the more suitable they are for the problem to be solved.

[0141] Pooling layer:

[0142] Since it is often necessary to reduce the number of training parameters, a pooling layer is often periodically introduced after the convolution layer, i.e., as Figure 2Each layer of the example 121-126 shown in the middle 120 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In the text processing process (which can also be referred to as phoneme sequence processing), the only purpose of the pooling layer is to reduce the spatial size of the phonemes. The pooling layer can include an average pooling operator and / or a maximum pooling operator for sampling the phonemes to obtain phonemes of a smaller size. The neural network layer 130:

[0143] After the processing by the convolutional layer / pooling layer 120, the convolutional neural network 100 is still not sufficient to output the required output information. As described above, the convolutional layer / pooling layer 120 only extracts features and reduces the parameters brought by the phoneme sequence. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 100 needs to use the neural network layer 130 to generate an output of the number of required classes or a group of outputs. Therefore, the neural network layer 130 can include multiple hidden layers (such as 131, 132, and 13n shown in the figure) and an output layer 140, and the parameters included in the multiple hidden layers can be pre-trained according to the related training data of a specific task type, for example, the task type can include text-to-speech, phoneme duration prediction, etc. Figure 2

[0144] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140, which has a loss function similar to the classification cross-entropy, and is specifically used to calculate the prediction error. Once the forward propagation (such as the propagation from 110 to 140) of the entire convolutional neural network 100 is completed, the back propagation (such as the propagation from 140 to 110) will start to update the weight values and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the result output by the output layer of the convolutional neural network 100 and the ideal result. Figure 2 Figure 2

[0145] It should be noted that the convolutional neural network 100 shown in the figure is only an example of a convolutional neural network, and in specific applications, the convolutional neural network can also exist in the form of other network models, for example, multiple convolutional layers / pooling layers shown in the figure can be parallel, and the features extracted by each convolutional layer / pooling layer are input to the full neural network layer 130 for processing. Figure 2 Figure 3

[0146] The following describes a chip hardware structure provided by an embodiment of the present application.

[0147] Figure 4 ​​​​​A chip hardware structure is provided for the embodiments of the present application, and the chip includes a neural network processor 40. The chip can be arranged in the execution device 110 as shown in Figure 1 to complete the computing work of the computing module 111. The chip can also be arranged in the training device 120 as shown in Figure 1 to complete the training work of the training device 120 and output the target model / rule 101. The algorithms of each layer in the convolutional neural network as shown in Figure 2 can be implemented in the chip as shown in Figure 4 .

[0148] The neural network processor 40 can be a neural network processor (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU), or any other processor suitable for large-scale XOR operation processing. Taking the NPU as an example: the neural network processor 40 is mounted as a coprocessor on the host central processing unit (CPU), and the host CPU allocates tasks. The core part of the NPU is the operation circuit 403, and the controller 404 controls the operation circuit 403 to extract data in the memory (weight memory or input memory) and perform operations.

[0149] In some implementations, the operation circuit 403 includes a plurality of processing units (PEs) inside. In some implementations, the operation circuit 403 is a two-dimensional systolic array. The operation circuit 403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 403 is a general matrix processor.

[0150] For example, assuming there are an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of the matrix B from the weight memory 402 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 401 and performs matrix operations with the matrix B to obtain the partial results or final results of the matrix, which are saved in the accumulator 408.

[0151] The vector computation unit 407 can perform further processing on the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, magnitude comparison, etc. For example, the vector computation unit 407 can be used for network computation of non-convolution / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0152] In some implementations, the vector computation unit 407 can store the processed output vector to the unified buffer 406. For example, the vector computation unit 407 can apply a non-linear function to the output of the arithmetic circuit 403, e.g., a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 407 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 403, e.g., for use in a subsequent layer in a neural network.

[0153] The unified memory 406 is used to store input data as well as output data.

[0154] The weight data is transferred from external memory to the input memory 401 and / or the unified memory 406, from external memory to the weight memory 402, and from the unified memory 506 to external memory by the direct memory access controller (DMAC) 405.

[0155] The bus interface unit (BIU) 410 is used to interact with the main CPU, the DMAC, and the instruction fetch buffer 409 via a bus.

[0156] The instruction fetch buffer 409, connected to the controller 404, is used to store instructions used by the controller 404.

[0157] The controller 404 is used to invoke instructions stored in the instruction fetch buffer 409 to control the operation of the arithmetic accelerator.

[0158] Generally, the unified memory 406, the input memory 401, the weight memory 402, and the instruction memory 409 are on-chip memories, and the external memory is a memory external to the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM), or other readable and writable memories.

[0159] wherein, Figure 2 or Figure 3 The operations of each layer in the convolutional neural network shown in the formula (1) can be performed by the operation circuit 403 or the vector calculation unit 407.

[0160] The speech synthesis network, the training method of the prediction network, and the data processing method of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0161] The speech synthesis neural network and the prediction network in the embodiments of the present application can be two separate networks, or can be a multi-task neural network, in which one task is to output the duration of each phoneme of the text, and the other task is to output the speech information corresponding to the text.

[0162] Secondly, in combination with Figure 5 The training method of the speech synthesis network of the embodiments of the present application will be described in detail. Figure 5 The training method of the speech synthesis network can be performed by a speech synthesis network training device, which can be a cloud service device or a terminal device, such as a computer, a server, or other devices with sufficient computing power to perform the training method of the speech synthesis network, or a system composed of a cloud service device and a terminal device. For example, the training method can be performed by the training device 120 in the system 100, Figure 1 the neural network processor 40 in the system 200. Figure 4

[0163] Alternatively, the training method can be processed by a CPU, or by a CPU and a GPU together, or without a GPU, but using other processors suitable for neural network calculation, which is not limited in the present application.

[0164] Figure 5 The training method shown in the formula (1) includes steps 501 and 502. The steps 501 and 502 will be described in detail below.

[0165] Step 501: obtaining a first text and a first speech corresponding to the first text.

[0166] ​The manner of obtaining the first text and the first speech corresponding to the first text (which can also be understood as training data) in the embodiments of the present application can be to obtain the first text first, and then obtain the first text by recognizing the first speech. It can also be directly obtained as the first text and the first speech, and the specific implementation is not limited here. The first speech includes at least one type of speech / dialect, for example, the first speech includes at least one type of speech / dialect, or the first speech includes at least two types of speech / dialect. If the first speech includes one type of speech / dialect, the embodiments of the present application can be applied to a single language / dialect synthesis scene. If the first speech includes at least two types of speech / dialect, the embodiments of the present application can be applied to a multi-language / dialect synthesis scene.

[0167] The embodiments of the present application are described by taking speech / dialect as an example, and it can be understood that the first speech can also be at least one type of minority language and other forms of speech, and the specific implementation is not limited here.

[0168] Optionally, since the texts corresponding to the at least two dialect speeches can be the same text, the training data can also include a dialect identifier.

[0169] The classification of the language in the embodiments of the present application has multiple cases, which can refer to languages in different countries, and can also refer to languages classified by different language families or language systems, for example: Indo-European language system, Sino-Tibetan language system, non-Tai language system. It can also refer to the six working languages designated by the United Nations: Chinese, English, Russian, Arabic, French, and Spanish. In actual application, other cases of language classification can also be used, and the specific implementation is not limited here.

[0170] The classification of the dialect in the embodiments of the present application has multiple cases, which can refer to: northern dialect, Wu dialect, Xiang dialect, Hakka dialect, Min dialect, Cantonese dialect, Gan dialect, and Jin dialect. It can also refer to more specific local dialects, for example, Nanchang dialect, Guangzhou dialect, Changsha dialect, Cantonese, Minnan dialect, Chaoshan dialect, Hakka dialect, Northeastern dialect, and Beijing dialect. It can be understood that the dialect can also be, for example, American English and British English. In actual application, other cases of dialect classification can also be used, and the specific implementation is not limited here.

[0171] For example, if the first text is a multi-language / dialect text, the first text includes Chinese and English, and the first speech includes Chinese corresponding Chinese speech and English corresponding English speech. If the first text is a single language / dialect text, the first text includes Chinese or English, and the first speech includes Chinese corresponding Chinese speech or English corresponding English speech.

[0172] For example, the first text is "Gengzhi boy", the corresponding pronunciation of "Gengzhi" is Chinese pronunciation, and the corresponding pronunciation of "boy" is English pronunciation. Of course, the first text can also be a single language such as "Today the weather is good". Of course, "Today the weather is good" can also correspond to two types of dialects (for example, Northeastern Mandarin and Tianjin Mandarin).

[0173] Optionally, if the number of pronouncers (or users) is multiple, in order to make the subsequent predicted pronunciation information correct, the training data in the training data can also include a pronouncer identifier, or include a timbre feature of the first pronunciation, or include a voiceprint feature of the first pronunciation, etc.

[0174] The training data obtained in the embodiments of the present application can be obtained by directly recording the pronunciation of the pronunciation object (for example, the pronouncer), or can be obtained by inputting audio information and video information by the user, or can be obtained by receiving the sending of the collection device. In actual application, there are other ways to obtain training data, and the way to obtain training data is not limited here.

[0175] In step 502, a first phoneme sequence is obtained based on the first text.

[0176] This step can also be understood as preprocessing of the training data. For example, if the training data described above includes the first pronunciation, the first text can be obtained by recognizing the first pronunciation, and the first text is represented by the first phoneme sequence. For example, if the training data described above includes the first pronunciation and the first text corresponding to the first pronunciation, the preprocessing can include processing the first text to obtain the first phoneme sequence of the first text. The preprocessing can include at least one of text recognition of the first pronunciation, processing of the first text to the first phoneme sequence, prosody prediction of the first text, and normalization processing of the first text. The non-standard information in the first text is normalized (for example, the standardized text is recognized). For example: non-standard information is an expression (such as smiling, laughing), shorthand (such as 1.4cm), symbol (such as colon), number. The phoneme sequence proposed in the embodiments of the present application can be understood as a phoneme sequence containing prosody. It can be understood that the prosody prediction of the first text is not limited here.

[0177] For example, the "smiling" expression is normalized to obtain the text "hehe". The "laughing" expression is normalized to obtain the text "haha". The "1.4cm" is normalized to obtain the text "1.4cm". The "1:3" is normalized to obtain the text "1:3".

[0178] In a possible implementation, the first text is a text corresponding to a type of language / dialect, and the first phoneme sequence can be obtained through a grapheme-to-phoneme (G2P) model corresponding to the type of language / dialect. The prosody of the first phoneme sequence is obtained through prosody prediction, and the prosody is used to update the first phoneme sequence.

[0179] For example, continuing the above example, the first text is “this is a small test”, and the first phoneme sequence is “zhe4 shi4 yi2 ge4 xiao3 ce4 shi4”. 1 to 4 represent tones, for example: 1 represents flat tone, 2 represents rising tone, 3 represents falling tone, and 4 represents entering tone. The prosody of the first text is “this is a small test”. Among them, #1 and #2 are different prosodies. Correspondingly, the first phoneme sequence of the first text is “sos zhe4 SP1 shii4 SP1 i2 / ge4 SP2 xiao3 SP1 ce4 / shii4 SP1 eos”. Among them, sos represents the beginning of the first phoneme sequence, and eos represents the end of the first phoneme sequence. SP1 corresponds to #1, and SP2 corresponds to #2.

[0180] In another possible implementation, the first text is a text corresponding to two types of language / dialect, and the first phoneme sequence can be obtained through G2P models corresponding to the two types of language / dialect. The first text includes a first subtext and a second subtext, the first subtext is related to one of the at least two types of language, and the second subtext is related to another of the at least two types of language; or, the first subtext is related to one of the at least two types of dialect, and the second subtext is related to another of the at least two types of dialect. The above obtaining the first phoneme sequence of the first text can specifically include: obtaining the phoneme sequence of the first subtext, obtaining the phoneme sequence of the second subtext, and splicing the phoneme sequence of the first subtext and the phoneme sequence of the second subtext to obtain the first phoneme sequence of the first text.

[0181] For example, the first text is “this is a small test case”, the first subtext is “this is a small test”, and the first phoneme sequence is “zhe4 shi4 yi2 ge4 xiao3 ce4 shi4”. The second subtext is “case”, and the second phoneme sequence is “K EY1 S”. Among them, 1 to 4 represent tones, for example: 1 represents flat tone, 2 represents rising tone, 3 represents falling tone, and 4 represents entering tone.

[0182] For example, the prosody of the first text is "this #1 is #1 a #2 small #1 test #1 case #3". Wherein, #1, #2 and #3 are different prosodies. Correspondingly, the first phoneme sequence of the first text is "sos zh e4 SP1 sh ii4 SP1 i2 / g e4 SP2 x iao3 SP1 c e4 / sh ii4 SP1 K EY1 S pau eos". Wherein, sos represents the beginning of the first phoneme sequence, and eos represents the end of the first phoneme sequence. SP1 corresponds to #1, SP2 corresponds to #2, and pau is #3.

[0183] It can be understood that if the first subtext in the first text is segmented by the second subtext (for example, the first text is "hello, Xiao Li, nice to meet you", and the first subtext "hello, nice to meet you" is segmented by the second subtext "Xiao Li"), the position information of the first subtext and the second subtext in the text to be processed also needs to be obtained, and the first feature and the second feature can be spliced based on the position information to obtain the first feature.

[0184] In step 503, the correspondence between the first speech and the first phoneme sequence is obtained based on the attention mechanism.

[0185] The attention mechanism simulates the internal process of biological observation behavior, that is, a mechanism that aligns internal experience and external feeling to increase the observation accuracy of a part of the region. The attention mechanism can quickly extract important features of sparse data, which will be described below in conjunction with examples. The attention mechanism can be a self-attention mechanism or a location sensitive attention mechanism, and the specific mechanism is not limited herein.

[0186] This step can also be understood as obtaining the correspondence between the first speech and the first phoneme sequence by using the attention network. The correspondence is used to represent the duration of each phoneme in the phoneme sequence in the first speech (or understood as the number of frames of the phoneme in the first speech, or the duration of the phoneme in the first speech).

[0187] For example, the attention network can refer to Figure 6The attention network includes an encoder, an attention module, an autoregressive decoder, a duration extraction module (e.g., MAS), and a prediction network. The training process of the attention network is to take the first text as the input of the attention network, and to train the attention network to obtain a value of a first loss function less than a first threshold. The first loss function is used to represent the difference between the mel-spectrogram feature output by the attention network and the actual mel-spectrogram feature of the first speech. The task of the attention network is to determine the correspondence between the first speech and the first phoneme sequence in the process of predicting the mel-spectrogram feature of the first text.

[0188] The encoder is used to encode the first text into a text vector. The autoregressive decoder is used to obtain the speech feature (e.g., frequency spectrum feature or mel-spectrogram feature) corresponding to the text according to the first vector. In the training process of the autoregressive decoder, the calculation of each step takes the actual speech feature corresponding to the previous step as a condition. In addition, the attention module is used to pay attention to the appropriate position in the text vector, so that the generated speech is more consistent with the characteristics of the speaker. For example, the attention module uses the recurrent neural network (RNN) to calculate the location sensitive attention score of the output of the autoregressive decoder (i.e., the current speech frame output) and the output of the encoder as the query vector, so as to obtain the context information. In other words, at each time step of the autoregressive decoder, the attention RNN of the autoregressive decoder generates a vector, which is used to calculate the attention with the output of the encoder, that is, to obtain the attention score. According to the score, the autoregressive decoder can determine which part of the encoder output is related to the next frame prediction (this part is also the context information). By repeating the process, the correspondence between the first speech and the first phoneme sequence can be determined. For details, refer to the principle of the autoregressive model. The description of the duration extraction module (e.g., MAS) can refer to step 504.

[0189] Furthermore, for multilingual / dialect scenarios (i.e., the first speech includes at least two languages / dialects), the demand for a large dataset of a single speaker's first speech is significant, indicating a high requirement for the speaker's language / dialect. To address this issue, a single dataset (i.e., a dataset corresponding to one language / dialect) can be obtained before training the attention network using the mixed dataset (first text and first speech). This single dataset can then be used to train the attention network first, followed by further training using the mixed dataset to obtain a pre-trained attention network. This single dataset includes a second text and its corresponding second speech, which includes speech in one language / dialect.

[0190] For example, please refer to Figure 7 The correspondence between the first speech and the first phoneme sequence is obtained through an attention network. The horizontal axis represents the number of decoding steps in the autoregressive decoder, and the vertical axis corresponds to... Figure 6 The encoding part consists of multiple vertical bars. The horizontal axis can be understood as speech, and the vertical axis as phonemes.

[0191] In addition, pronunciation features (speaker embeddings) can be added to the text vectors obtained by the encoder described above. These pronunciation features can include voiceprint features or other features used to describe the speaker's pronunciation characteristics or timbre characteristics. The pronunciation features can be obtained through methods such as look-up table (LUP), speaker verification (SV), or neural networks; specific methods are not limited here.

[0192] Step 504: Based on the dynamic programming method, correct the correspondence to obtain the first duration information of each phoneme in the first phoneme sequence.

[0193] To achieve correct alignment between the first speech and the first phoneme sequence, embodiments of this application can obtain the first duration information (or corrected phoneme duration information) of each phoneme in the first phoneme sequence based on a dynamic programming method and an attention mechanism. The dynamic programming method is one that can correct the correspondence between speech and phonemes; for example, it includes a monotonic alignment search (MAS) algorithm or the Needleman–Wunsch algorithm. Since dynamic programming can infer misaligned phonemes through monotonicity and other methods, it can reduce the probability of phonemes being swallowed.

[0194] Step 503 obtains the correspondence between the first speech and phonemes, and then the correspondence is corrected according to the dynamic programming method to obtain the first duration information of each phoneme in the first phoneme sequence.

[0195] Optionally, Figure 6 The duration extraction module (e.g., MAS) in the attention network is specifically used to correct the correspondence between the first speech and the first phoneme sequence.

[0196] For example, the correspondence between each phoneme in the first phoneme sequence obtained by the attention network and the first speech is represented by a matrix P. This matrix P is an M-row, N-column matrix, where M is the dimension of the first text (e.g., ...). Figure 7 The vertical axis is 70), and N is the dimension of the first speech spectrogram (e.g., 70). Figure 7 If the x-coordinate in the matrix is ​​120, then this step can specifically include steps 1 to 3. Step 1: Initialize the Q matrix and the A matrix (which can also be understood as an all-zero matrix). Step 2: By... Figure 7 Starting from the first point at the bottom left corner of the diagonal, the Q matrix is ​​obtained iteratively based on monotonicity (see Formula 1 for details). Step 3: Extract the corresponding... Figure 7 Starting from the last point of the top right corner of the diagonal, update matrix A according to Formula 2, for example, using... Update matrix A, that is The element at position A is updated to 1, and this A can be interpreted as the obtained optimal path. That is, corresponding to... Figure 7 The x-axis value (75-85) corresponds to the y-axis value. Then, adjustments are made based on the y-axis value corresponding to the x-axis value (75-85). Figure 7 Get as Figure 8 The correspondence between the first speech and phonemes shown is then summed along the horizontal axis to obtain a vector with the same dimension as the vertical axis. This vector is rounded up / down to obtain the first duration information.

[0197] Formula 1: Q i,j =max(Q i-1,j-1 Q i,j-1 )+P i,j ;

[0198] Formula 2:

[0199] Among them, P i,j It can be understood as Figure 7 In the curve, point i can be understood as the x-coordinate and point j as the y-coordinate. Formula 1 can be understood as evaluating in an increasing direction, i.e., Q. i,j The value of Q is on the left. i-1,j-1 Or the bottom left Q i,j-1 The value of . yesFigure 7 The curve function of Formula One can be understood as a decreasing direction evaluation, The value of Formula One is the value on the right side or the value on the right lower side of Formula One.

[0200] It can be understood that the above formula is only an example, and other forms of formula can also be used in actual application, which is not limited here.

[0201] In step 505, the first prediction network is trained based on the first phoneme sequence and the first duration information, and a trained first prediction network is obtained.

[0202] The first phoneme sequence is taken as the input of the first prediction network, and the trained first prediction network is obtained by training the first prediction network with the value of the first loss function being less than the first threshold value. The first loss function is used to represent the difference between the duration information output by the first prediction network and the first duration information. The trained first prediction network is used to predict the duration information of the text to be processed. It can be understood that the specific form of the loss function is not limited for the loss function involved in the embodiments of the present application.

[0203] Optionally, for the scenario in which the first speech includes speech of at least two languages / dialects, the demand for the data set of the first speech of a speaker is large, that is, the requirement for the language / dialect of the speaker is high. In order to solve this problem, before the first prediction network is trained using the mixed data set (the first text and the first speech), a single data set (that is, a data set corresponding to one language / dialect) is obtained first, and the second prediction network is trained using the single data set to obtain the first prediction network, and then the first prediction network is further trained using the mixed data set to obtain the trained first prediction network. The single data set includes the second text and the second speech corresponding to the second text, and the second speech includes speech of one language / dialect. That is, if applied to the scenario of speech of two languages / dialects, the embodiments further include: obtaining the second text and the second speech corresponding to the second text. The second phoneme sequence of the second text is obtained. The second duration information of each phoneme in the second phoneme sequence is obtained. The second phoneme sequence is taken as the input of the second prediction network, and the first prediction network is obtained by training the second prediction network with the value of the second loss function being less than the second threshold value. The second loss function is used to represent the difference between the duration information output by the second prediction network and the second duration information.

[0204] In the embodiment of the application, the first duration information of each phoneme in the first text can be obtained based on the attention mechanism and the dynamic programming method. Since the dynamic programming can deduce the unaligned phonemes through monotonicity and the like, the probability of phoneme being wrongly estimated (for example, phoneme transposition or phoneme being swallowed) can be reduced. In turn, the correct alignment of the phonemes in the first text and the first speech is realized. This is convenient for application in scenarios requiring phoneme duration information, such as speech synthesis.

[0205] The following will be described taking the speech synthesis scenario as an example.

[0206] In this scenario, in addition to the steps 501 to 504 described above, the embodiment can further include steps 901 to 903. Figure 9

[0207] In step 901, the first mel-frequency spectrum feature of the first speech is obtained.

[0208] In this step, the first mel-frequency spectrum feature of the first speech is obtained based on the first speech in the training data. The first mel-frequency spectrum feature can also be understood as the actual mel-frequency spectrum feature of the first speech or the first text. The manner of obtaining the first mel-frequency spectrum feature is not limited here.

[0209] In step 902, the first pronunciation feature is obtained. This step is optional.

[0210] The first pronunciation feature can include a voiceprint feature and the like for describing the pronunciation feature of a pronouncer, or a timbre feature of the pronouncer. The manner of obtaining the first pronunciation feature is similar to the foregoing description, and can be LUP, SV or a neural network and the like, and the specific manner is not limited here.

[0211] Of course, the first pronunciation feature can also be initialized, and the trained first pronunciation feature is obtained through subsequent training.

[0212] In step 903, the first phoneme sequence and the first duration information are taken as the input of the first speech synthesis network, and the trained first speech synthesis network and the trained first pronunciation feature are obtained by training the first speech synthesis network with the value of the third loss function being less than the third threshold value as the target.

[0213] Optionally, if the embodiment includes step 902, this step can specifically include: taking the first phoneme sequence, the first duration information and the first pronunciation feature as the input of the first speech synthesis network, and obtaining the trained first speech synthesis network and the trained first pronunciation feature by training the first speech synthesis network with the value of the third loss function being less than the third threshold value as the target. The following will be described taking the embodiment including step 902 as an example, and it can be understood that step 902 can also not be included.

[0214] ​The third loss function described above is used to represent the difference between the second mel-spectrogram feature output by the first speech synthesis network and the first mel-spectrogram feature, and the second mel-spectrogram feature is obtained after the first duration information is expanded (details are described below in combination with Figure 10 The first speech synthesis network is described below. Figure 11 The first speech synthesis network is described below.

[0215] In the embodiments of the present application, there are multiple ways to train the first speech synthesis network, which are described below respectively.

[0216] First, the first speech synthesis network contains an attention network.

[0217] In this way, the attention network in step 503 can be understood as a second network in the first speech synthesis network.

[0218] Optionally, the architecture of the speech synthesis network (for example, the first speech synthesis network or the subsequent second speech synthesis network) can refer to Figure 10 The speech synthesis network can specifically include a first network and / or a second network. The first network includes a first encoder (Encoder A) and a first autoregressive decoder (Decoder A). The second network includes a second encoder (Encoder B), an attention module, and a second autoregressive decoder (Decoder B). The function of the first encoder / second encoder is to encode the first phoneme sequence into a first feature. The function of the first autoregressive decoder / second autoregressive decoder is to obtain the speech feature (for example, the frequency spectrum feature or the second mel-spectrogram feature) corresponding to the first text according to the first feature (or the second feature expanded by the expansion module). In the training process, the first autoregressive decoder / second autoregressive decoder calculates the real speech feature corresponding to the previous step as a condition for each step. The second network can be understood as the aforementioned attention network. The attention module in the second network is used to obtain the first duration information of the phonemes used in the expansion module of the first network.

[0219] Further, in order to ensure the coherence of the speech before and after, the first network can further include an expansion module. The function of the expansion module is to use the phoneme duration information generated based on the attention module (or understood as the aforementioned prediction network) for correction. That is, it can be understood that the text vector is up-sampled according to the duration of each phoneme in the first speech (which can also be understood as expanding the frame number of the vector) to obtain a vector corresponding to the frame number. The first autoregressive decoder is specifically used to decode the mel-spectrogram feature according to the expanded vector, and further obtain the speech of the speaker for the first text.

[0220] Optionally, the architecture of the speech synthesis network can be as follows Figure 11The first network includes a first encoder (Encoder A), an expansion module, and a first autoregressive decoder (Decoder A). The second network includes a second encoder (Encoder B), an attention module (Attention), a duration extraction module (MAS), and a second autoregressive decoder (Decoder B). The state expansion module includes an expansion submodule (State Exapansion) and a duration module (Duration). During the training process, the second network corrects the correspondence between the first phoneme sequence and the first speech through the MAS, and then obtains the corrected phoneme duration information (i.e., the first duration information) (for specific description, refer to the step shown in Figure 5 The first duration information is transmitted to the Duration in the state expansion module. On the one hand, the Duration sends the first duration information to the expansion submodule, and the expansion submodule uses the first duration information to expand the first feature output by the first encoder to obtain a second feature. On the other hand, the Duration sends the first duration information to the duration prediction module (Duration Predictor), and then uses the first duration information and the first phoneme sequence to train the duration prediction module. The duration prediction module can be understood as the prediction network in step 505 described above. On the other hand, the Duration sends the first duration information to the second autoregressive decoder, which can be understood as training the second network by the difference between the phoneme duration obtained through the Attention alignment information and the first duration information. For the prediction network, it can be understood as being trained according to the duration information output by the attention network.

[0221] It can be understood that after the attention module processes the text vector, the first pronunciation feature can also be added. The second autoregressive decoder is specifically configured to decode the speaker's speech corresponding to the text according to the expanded vector described above.

[0222] Further, the state expansion module can also include a convolution module (Fusion). The convolution module is configured to perform convolution processing on the second feature expanded by the expansion submodule and the first pronunciation feature to obtain a third feature, and then input the third feature into the first autoregressive decoder to obtain a second mel spectrum feature. Further, the obtained second mel spectrum feature is more perfect, or it can be understood that the second mel spectrum feature considers the features of the entire first phoneme sequence.

[0223] Optionally, the first encoder, the second encoder, the first autoregressive decoder, and the second autoregressive decoder described above can be a recurrent neural network, a long short-term memory (LSTM), etc.

[0224] Exemplarily, the second network can include transformer, tacotron2, etc.

[0225] Optionally, the decoder (the first autoregressive decoder and the second autoregressive decoder) described above can be a one-way decoder, or can be a two-way decoder (i.e., two directions in parallel), which is not limited here. Wherein, the two directions refer to the direction of the training text, which can also be understood as the direction of the vector corresponding to the first phoneme sequence, and can also be understood as the forward sequence or the reverse sequence of the first phoneme sequence. One direction is one side of the first phoneme sequence pointing to the other side of the training text, and the other direction is the other side of the first phoneme sequence pointing to one side of the training text. If the decoder is a two-way decoder, the decoders of the two directions (or forward and reverse sequences) are trained in parallel, and each is independently calculated in the training process, and there is no result dependency. Of course, if the speech synthesis network includes a prediction network, the prediction network can be referred to as a duration prediction module.

[0226] Exemplarily, if the first text is: "Gengzhi boy", the first direction or the forward sequence can be the direction from "Geng" to "y", and the second direction or the reverse sequence can be the direction from "y" to "Geng".

[0227] Further, similar to the training of the foregoing attention network, for a multi-lingual / dialect scenario, the demand for a data set of the first speech of a speaker is relatively large, i.e., the requirement for the language / dialect of the speaker is relatively high. In order to solve this problem, before using a mixed data set (the first text and the first speech) to train the first speech synthesis network, a single data set (i.e., a data set corresponding to one language / dialect) can be obtained first, and the second speech synthesis network is trained using the single data set to obtain the first speech synthesis network, and then the first speech synthesis network is further trained by using the mixed data set to obtain the trained first speech synthesis network. The single data set includes a third text and a third speech corresponding to the third text, and the third speech includes a speech of one language / dialect. Wherein, the third text can also be the second text described above, and the third speech can be the third speech described above.

[0228] If the first speech includes speech of at least two types of languages / dialects, the training process of the first speech synthesis network can be understood as follows: obtaining a third text and third speech corresponding to the third text, the third speech including speech of one type of language / dialect in the at least two types of languages / dialects; obtaining a third phoneme sequence of the third text; obtaining third duration information of each phoneme in the third phoneme sequence; obtaining second pronunciation features used to describe the timbre features of the third speech; obtaining third mel-spectrogram features of the third speech; taking the third phoneme sequence, the third duration information, and the second pronunciation features as inputs of the second speech synthesis network, and training the second speech synthesis network to obtain the first speech synthesis network and the trained second pronunciation features, with the value of a fourth loss function being less than a fourth threshold as a target, the fourth loss function being used to represent the difference between fourth mel-spectrogram features output by the second speech synthesis network and the third mel-spectrogram features, the fourth mel-spectrogram features being obtained after expansion of the third duration information.

[0229] For example, the training of the first network and the second network can refer to Figure 11 and Figure 12 . In this case, Figure 11 applied to a single language / dialect scenario, Figure 12 applied to a multi-language / dialect scenario. It can be seen that, for the training of a multi-language / dialect mixed data set, or for the training of a single language / dialect mixed data set, the corresponding relationship between the first speech and the phonemes obtained by the second network is provided to the first network after MAS processing, specifically to the expansion module in the first network, so as to align the corresponding relationship between the first speech and the phonemes and reduce the omission of phonemes in the speech output by the speech synthesis network.

[0230] Secondly, the attention network is not in the speech synthesis network.

[0231] In this way, the attention network in step 503 can be understood as a network other than the speech synthesis network.

[0232] Optionally, the architecture of the speech synthesis network can refer to Figure 13 The speech synthesis network can be specifically understood as the first network in the above description. The description of the first network can refer to the foregoing description, and is not limited herein.

[0233] It should be noted that the training process of the prediction network and the speech synthesis network can also use other training methods instead of the foregoing training methods, which are not limited herein.

[0234] In a possible implementation, the network training method in the embodiment of the present application includes steps 501 to 505, i.e., the training method of the prediction network. In another possible implementation, the network training method in the embodiment of the present application can include steps 501 to 505 and steps 901 and 903, i.e., the training method of the prediction network and the speech synthesis network, the speech synthesis network being used to predict the speech of the text to be processed. In another possible implementation, the network training method in the embodiment of the present application can include steps 501 to 505 and steps 901 to 903, i.e., the training method of the prediction network and the speech synthesis network, the speech synthesis network being used to predict the speech of the text to be processed by a speaker. Wherein, the prediction network and the speech synthesis network provided by the embodiment of the present application can be applied to a single language / dialect scene, and can also be applied to a single language / dialect scene, and the specific application scenarios are not limited here.

[0235] In the embodiment, the prediction network trained by the attention mechanism and the dynamic programming method can accurately predict the duration information of the phonemes, or be understood as obtaining the correct correspondence between the phonemes and the speech, thereby improving the speech listening feeling obtained by the speech synthesis network and reducing the poor speech listening feeling caused by phoneme omission.

[0236] Firstly, the application scenarios to which the data processing method provided by the embodiment of the present application is applied are described. The data processing method can be applied to a voice interaction scene, including a smart home, intelligent travel, smart government, etc. The specific application scenarios are not limited here. For example, the data processing method can be applied to a reading APP of speech synthesis technology, which can provide a higher natural mixed language reading function for users and provide an extreme experience. For another example, the data processing method can also be applied to mobile phones, sound boxes, etc. to broadcast fresh information for users anytime and anywhere. For another example, the data processing method can also be applied to a car-hailing software, a restaurant ordering, a queuing software, etc. to broadcast orders through speech synthesis and conveniently obtain notification information. It can be understood that the data processing method can also be applied to a children's story machine, a smart robot, a tablet device, etc. to make the interaction between the user and the device more natural and more friendly. It can be understood that the above-mentioned application scenarios are only examples, and other application scenarios can also be applied in actual applications, and the specific application scenarios are not limited here.

[0237] The data processing device is a terminal device or a cloud device for serving a user. The terminal device can include a head mount display (HMD), which can be a combination of a virtual reality (VR) box and a terminal, a VR all-in-one machine, a personal computer (PC), an augmented reality (AR) device, a mixed reality (MR) device, and the like. The terminal device can also include a cellular phone, a smart phone, a personal digital assistant (PDA), a tablet computer, a laptop computer, a PC, a vehicle terminal, a projector, a smart screen, a robot, and the like, without limitation.

[0238] The data processing method provided in the embodiments of the present application can be performed by a terminal device or a server alone, or by a terminal device and a server together. The following describes the two cases respectively.

[0239] Embodiment one: The terminal device or the server performs the data processing method alone.

[0240] Please refer to Figure 15 The data processing method provided in the embodiments of the present application can be performed by a terminal device or a server alone, or by a terminal device and a server together. The following describes the two cases respectively.

[0241] Step 1501: Obtain a text to be processed.

[0242] In the embodiments of the present application, the data processing device can obtain the text to be processed in various ways, such as by collecting user input text, by receiving a text to be processed sent by another device, by selecting a text to be processed from a database, and the like, without limitation.

[0243] The text to be processed in the embodiments of the present application includes a text of a single language / dialect, or a text of multiple languages / dialects. The description of the language / dialect can refer to the description in the foregoing embodiments, which will not be repeated here. Figure 5 The text to be processed in the embodiments of the present application includes a text of a single language / dialect, or a text of multiple languages / dialects. The description of the language / dialect can refer to the description in the foregoing embodiments, which will not be repeated here.

[0244] Step 1502: Obtain a phoneme sequence based on the text to be processed.

[0245] In this step, based on the text to be processed, the phoneme sequence and the aforementioned Figure 5 The manner of obtaining the first phoneme sequence based on the first text is similar, and will not be repeated here.

[0246] In step 1503, the pronunciation feature is obtained. This step is optional.

[0247] Optionally, the pronunciation feature can include a voiceprint feature or a timbre feature for describing the pronunciation feature of the pronouncer. The manner of obtaining the pronunciation feature is similar to the manner of obtaining the first pronunciation feature in the aforementioned step 503, which can be LUP, SV, or a neural network, and the specific manner is not limited here.

[0248] In step 1504, the duration information of each phoneme in the phoneme sequence is predicted based on the trained prediction network.

[0249] This step can specifically input the phoneme sequence into the trained prediction network to obtain the duration information of each phoneme in the phoneme sequence. The trained prediction network is trained based on the first text and the first duration information, the first duration information is obtained by correcting the correspondence between the first text and the first speech by the dynamic programming method, the first speech is the speech of the first text, and the correspondence is used to represent the duration of each phoneme in the phoneme sequence in the first speech.

[0250] The specific description of the trained prediction network can refer to the description of the first prediction network in the aforementioned Figure 5 embodiment, and will not be repeated here.

[0251] For example, the first duration information of each phoneme is (1, 2, 0), 1 represents the expansion multiple of the first phoneme, 2 represents the expansion multiple of the second phoneme, and 0 represents that the third phoneme is not expanded.

[0252] In step 1505, the phoneme sequence, the duration information, and the pronunciation feature are input into the trained speech synthesis network to obtain the mel spectrum feature. This step is optional.

[0253] This step can specifically input the phoneme sequence, the duration information, and the pronunciation feature into the trained speech synthesis network to obtain the mel spectrum feature.

[0254] Optionally, the trained speech synthesis network can be Figure 9 the first trained speech synthesis network in the aforementioned embodiment, and will not be repeated here.

[0255] Optionally, the trained speech synthesis network can be as follows: Figure 16As shown, the speech synthesis network includes an encoder, an expansion module, and an autoregressive decoder; this step specifically includes: obtaining a first feature corresponding to the phoneme sequence based on the encoder; obtaining a second feature by expanding the first feature based on the duration information (obtained by the prediction network) through the expansion module; obtaining a third feature based on the second feature and the pronunciation feature; and obtaining a mel-spectrogram feature based on the autoregressive decoder and the third feature. Among them, the above obtaining the mel-spectrogram feature based on the third feature and the autoregressive decoder can be inputting the third feature into the autoregressive decoder to obtain the mel-spectrogram feature, or can be obtaining a fourth feature by convolving the third feature through a convolution module (Fusion), and then inputting the fourth feature into the autoregressive decoder to obtain the mel-spectrogram feature. Figure 16 The speech synthesis network as shown can also be understood as the aforementioned Figure 5 The first network in the embodiment as shown does not include a prediction network.

[0256] It can be understood that the encoder, the expansion module, and the autoregressive decoder in this embodiment can refer to the description in the aforementioned Figure 6 、 Figure 11 or Figure 14 , which will not be described here again.

[0257] For example, the first feature is (a, b, c), and the duration information is (1, 2, 0), that is, it can be understood that a is expanded once, b is expanded twice, and c is expanded zero times, that is, the second feature after expansion is (a, b, b). It can be understood that a, b, and c here are only convenient for describing how to expand, and do not specifically limit the first feature.

[0258] Step 1506: converting the mel-spectrogram feature into speech through a vocoder. This step is optional.

[0259] After obtaining the mel-spectrogram feature, the mel-spectrogram feature can be input into the vocoder for conversion to obtain speech. The speech is the speech of the speaker (or understood as the pronunciation feature) for the text to be processed.

[0260] In one possible implementation, if the data processing device is a terminal device, the embodiment includes the above steps 1501 to 1506. In another possible implementation, if the data processing device is a server, the step 1501 specifically can be receiving the text to be processed sent by the terminal device. And after the step 1506, the method further includes: sending the speech to the terminal device.

[0261] In addition, if it is only applied to the scenario of obtaining the phoneme duration of the text, the embodiment can include steps 1501 to 1504. If it is applied to the scenario of speech synthesis, the embodiment can include steps 1501 to 1506.

[0262] In this embodiment, the duration information of the phonemes in the to-be-processed text is obtained, and the second feature is obtained by extending the first feature according to the duration information. The third feature is obtained based on the second feature and the pronunciation feature, and the speech of the speaker for the to-be-processed text is generated based on the third feature. Since the generated speech is obtained based on the duration information and the pronunciation feature, the speech synthesis of a single language / dialect or across languages / dialects can be realized, the timbre of the generated speech is consistent, and the poor speech listening feeling caused by the omission or misplacement of phonemes is avoided.

[0263] Embodiment two: the terminal device and the server jointly perform the data processing method.

[0264] Please refer to Figure 17 Another embodiment of the data processing method provided by the present application is provided, which can be executed by a data processing device or a component (for example, a processor, a chip, or a chip system) of the data processing device. The data processing device includes a terminal device and a server, and the embodiment includes steps 1701 to 1707.

[0265] In step 1701, the terminal device obtains the to-be-processed text.

[0266] In this step, the terminal device obtains the description of the to-be-processed text and the description in the foregoing Figure 15 The description in step 1501 in the embodiment shown in the foregoing is similar, and will not be repeated here.

[0267] In step 1702, the terminal device obtains the pronunciation feature.

[0268] In this step, the terminal device obtains the description of the to-be-processed text and the description in the foregoing Figure 15 The description in step 1503 in the embodiment shown in the foregoing is similar, and will not be repeated here.

[0269] In step 1703, the terminal device sends the to-be-processed text and the pronunciation feature to the server.

[0270] After the terminal device obtains the to-be-processed text and the pronunciation feature, the terminal device sends the to-be-processed text and the pronunciation feature to the server.

[0271] In step 1704, the server inputs the to-be-processed text and the pronunciation feature into the trained speech synthesis network to obtain the speech feature of the speaker for the to-be-processed text.

[0272] After the server receives the to-be-processed text and the pronunciation feature sent by the terminal device, the server inputs the to-be-processed text and the pronunciation feature into the trained speech synthesis network to obtain the speech feature (for example, the mel spectrum feature) of the speaker for the to-be-processed text.

[0273] Optionally, the description of the trained speech synthesis network can refer to the foregoing Figure 5The description of the speech synthesis network in the illustrated embodiment will not be repeated here. In addition, in the inference process, the speech synthesis network can be Figures 5 to 11 The description of the speech synthesis network in the illustrated embodiment will not be repeated here.

[0274] At step 1705, the server converts the speech features to language through the vocoder.

[0275] After the server obtains the mel-spectrogram features, the mel-spectrogram features are input into the vocoder to obtain the speech of the speaker for the text to be processed.

[0276] At step 1706, the server sends the speech to the terminal device.

[0277] After the server obtains the speech information, the server sends the speech to the terminal device.

[0278] At step 1707, the terminal device plays the speech. This step is optional.

[0279] Optionally, after the terminal device receives the speech sent by the server, the terminal device can play the speech to the user.

[0280] In this embodiment, the duration information of each phoneme in the phoneme sequence obtained by introducing the prediction network in the inference stage is obtained, and the speech is obtained based on the duration information and the text to be processed. The phoneme duration information obtained by the dynamic programming method can reduce the poor listening experience of the generated speech due to phoneme omission. Single language / dialect or cross-language / dialect speech synthesis can be implemented, and the tone consistency of the generated speech information is ensured.

[0281] The model training method and the data processing method in the embodiments of the application are described above, and the data processing device in the embodiments of the application is described below. Please refer to Figure 18 One embodiment of the data processing device (terminal device or server) in the embodiments of the application includes:

[0282] The obtaining unit 1801 is configured to obtain a first text and a first speech corresponding to the first text.

[0283] The obtaining unit 1801 is further configured to obtain a first phoneme sequence based on the first text.

[0284] The obtaining unit 1801 is further configured to obtain a corresponding relationship between the first speech and the first phoneme sequence based on the attention mechanism, the corresponding relationship being used to represent the duration of each phoneme in the first phoneme sequence in the first speech.

[0285] The correction unit 1802 is configured to correct the corresponding relationship based on a dynamic programming method to obtain first duration information of each phoneme in the first phoneme sequence.

[0286] The training unit 1803 is configured to train the first prediction network based on the first phoneme sequence and the first duration information, to obtain a trained first prediction network, and the trained first prediction network is configured to predict duration information of each phoneme in the to-be-processed text.

[0287] In the embodiment, the operations performed by the units in the data processing device are similar to those of the foregoing Figures 5 to 17 In the embodiment, the operations performed by the units in the data processing device are similar to those of the foregoing

[0288] In the embodiment, the first duration information of each phoneme in the first text can be obtained based on an attention mechanism and a dynamic programming method. Since the dynamic programming can deduce unaligned phonemes through monotonicity, etc., the probability of phonemes being estimated incorrectly (for example, phoneme transposition or phoneme swallowing) can be reduced. Furthermore, the first prediction network trained by the training unit 1803 can realize duration prediction of phonemes in the to-be-processed text. This is convenient for application in scenarios such as speech synthesis that need to use phoneme duration information.

[0289] In the embodiment, the operations performed by the units in the data processing device are similar to those of the foregoing Figure 19 In the embodiment, the operations performed by the units in the data processing device are similar to those of the foregoing

[0290] The obtaining unit 1901 is configured to obtain the to-be-processed text.

[0291] The obtaining unit 1901 is further configured to obtain a phoneme sequence based on the to-be-processed text.

[0292] The obtaining unit 1901 is further configured to obtain pronunciation features, the pronunciation features being used to describe timbre features of a pronouncer.

[0293] The prediction unit 1902 is configured to predict duration information of each phoneme in the phoneme sequence based on the trained prediction network, the trained prediction network being obtained by training the prediction network with the first text as input of the prediction network and with a value of a loss function being less than a threshold value as a target, the loss function being used to represent a difference between the duration information output by the prediction network and the first duration information, the first duration information being obtained by correcting a correspondence between the first text and a first speech through a dynamic programming method, and the first speech being a speech of the first text.

[0294] Optionally, the data processing device can further include a processing unit 1903 configured to input the phoneme sequence and the duration information into a trained speech synthesis network to obtain a mel spectrum feature.

[0295] Optionally, the data processing device can further include a conversion unit 1904 configured to convert the mel spectrum feature into a speech through a vocoder, and the speech being a speech of the to-be-processed text.

[0296] In the embodiment, the operations performed by the units in the data processing device are similar to those of the foregoing Figures 5 to 17Similar to the embodiments described above, details are not repeated here.

[0297] In this embodiment, the prediction unit 1902 introduces the duration information of each phoneme in the phoneme sequence obtained by the inference stage of the prediction network. Since the corresponding relationship between the first text and the first voice is corrected based on the dynamic programming method, the problem of inaccurate duration prediction caused by phoneme omission or phoneme misplacement can be avoided.

[0298] Referring to Figure 20 The data processing device provided in the embodiment of the present application includes a processor 2001, a memory 2002 and a communication interface 2003. The processor 2001, the memory 2002 and the communication interface 2003 are interconnected by lines. The memory 2002 stores program instructions and data.

[0299] The memory 2002 stores the aforementioned Figures 5 to 17 In the corresponding embodiments, the steps performed by the device correspond to the program instructions and data.

[0300] The processor 2001 is configured to perform the aforementioned Figures 5 to 17 The steps performed by the device in any of the embodiments described above.

[0301] The communication interface 2003 can be used for receiving and sending data, and is configured to perform the aforementioned Figures 5 to 17 The steps related to obtaining, sending and receiving in any of the embodiments described above.

[0302] In an implementation manner, the data processing device can include more or less components, and the present application is only exemplary and is not limited. Figure 20 More or less components, the present application is only exemplary and is not limited.

[0303] Referring to Figure 21 The data processing device provided in the embodiment of the present application includes a processor 2001, a memory 2002 and a communication interface 2003. The processor 2001, the memory 2002 and the communication interface 2003 are interconnected by lines. The memory 2002 stores program instructions and data.

[0304] Figure 21 The figure shows a block diagram of part of the structure of the mobile phone related to the data processing device provided in the embodiment of the present application. Referring to Figure 21The mobile phone includes radio frequency (RF) circuit 2110, memory 2120, input unit 2130, display unit 2140, sensor 2150, audio circuit 2160, wireless fidelity (WiFi) module 2170, processor 2180, and power supply 2190, etc. Those skilled in the art can understand that Figure 21 The mobile phone structure shown in the figure is not intended to limit the mobile phone, which can include more or less components than the figure, or combine certain components, or different component arrangement.

[0305] The following will be described in detail Figure 21 The various components of the mobile phone will be described in detail:

[0306] RF circuit 2110 can be used for receiving and sending signals in the process of information or call, in particular, receiving the downlink information of the base station and processing by processor 2180; in addition, sending the uplink data to the base station. Usually, RF circuit 2110 includes but is not limited to antenna, at least one amplifier, transceiver, coupler, low noise amplifier (LNA), duplexer, etc. In addition, RF circuit 2110 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to global system for mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short message service (SMS), etc.

[0307] The memory 2120 can be used to store software programs and modules, and the processor 2180 can execute various function applications and data processing of the mobile phone by running the software programs and modules stored in the memory 2120. The memory 2120 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), and the like. In addition, the memory 2120 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0308] The input unit 2130 can be used to receive inputted digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 2130 can include a touch panel 2131 and other input devices 2132. The touch panel 2131, also known as a touch screen, can collect a user's touch operation (such as the user's operation on or near the touch panel 2131 using a finger, a stylus, or any suitable object or accessory) and drive the corresponding connection device according to the pre-set program. Optionally, the touch panel 2131 can include two parts of a touch detection device and a touch controller. The touch detection device detects the user's touch position and detects the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch coordinates, and sends it to the processor 2180, and can also receive the command from the processor 2180 and execute it. In addition, the touch panel 2131 can be implemented in various types such as a resistive type, a capacitive type, an infrared type, and a surface acoustic wave type. In addition to the touch panel 2131, the input unit 2130 can also include other input devices 2132. Specifically, the other input devices 2132 can include one or more of a physical keyboard, a function key (such as a volume control key, an on-off key, etc.), a trackball, a mouse, a joystick, and the like.

[0309] The display unit 2140 can be used to display information input by a user or information provided to the user as well as various menus of the mobile phone. The display unit 2140 can include a display panel 2141, which can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Further, a touch panel 2131 can be overlaid on the display panel 2141, which, when detecting a touch operation thereon or in the vicinity thereof, transmits to the processor 2180 to determine the type of touch event, and then the processor 2180 provides corresponding visual output on the display panel 2141 according to the type of touch event. Although in the above embodiment, the touch panel 2131 and the display panel 2141 are implemented as two independent components to realize the input and output functions of the mobile phone, in some embodiments, the touch panel 2131 and the display panel 2141 can be integrated to realize the input and output functions of the mobile phone. Figure 21

[0310] The mobile phone can further include at least one sensor 2150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor can include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 2141 according to the brightness of ambient light, and the proximity sensor can turn off the display panel 2141 and / or the backlight when the mobile phone is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and when at rest, it can detect the magnitude and direction of gravity, which can be used for applications that identify the posture of the mobile phone (such as landscape / portrait screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), and the like. As for other sensors that the mobile phone can be configured, such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, they will not be described here.

[0311] The audio circuit 2160, the speaker 2161, and the microphone 2162 can provide an audio interface between the user and the mobile phone. The audio circuit 2160 can convert received audio data into an electrical signal, which is transmitted to the speaker 2161 to be converted into a sound signal for output. On the other hand, the microphone 2162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 2160 to be converted into audio data. The audio data is output to the processor 2180 for processing, and then transmitted to another mobile phone via the RF circuit 2110, or output to the memory 2120 for further processing.

[0312] ​WiFi is a short-range wireless transmission technology. The WiFi module 2170 can help users send and receive emails, browse web pages, and access streaming media, etc. It provides users with wireless broadband Internet access. Figure 21 The WiFi module 2170 is shown, but it is understood that it is not a necessary component of the mobile phone.

[0313] The processor 2180 is the control center of the mobile phone. It connects all parts of the mobile phone through various interfaces and lines, and performs various functions and processes data of the mobile phone by running or executing software programs and / or modules stored in the memory 2120 and calling data stored in the memory 2120, thereby monitoring the mobile phone as a whole. Optionally, the processor 2180 can include one or more processing units; preferably, the processor 2180 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It is understood that the above-mentioned modem processor can also not be integrated into the processor 2180.

[0314] The mobile phone also includes a power supply 2190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 2180 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management, etc. through the power management system.

[0315] Although not shown, the mobile phone can also include a camera, a Bluetooth module, etc., which will not be described here.

[0316] In the embodiments of the present application, the processor 2180 included in the data processing device can perform the functions in the above-described embodiments, which will not be described here. Figures 5 to 17

[0317] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other means. For example, the above-described device embodiments are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0318] ​The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0319] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized by software, hardware, firmware or any combination thereof, in whole or in part.

[0320] When the integrated unit is implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website site, computer, server or data center to another website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk (SSD)) and the like.

[0321] The terms "first", "second", etc. in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, which is only a way of distinguishing the objects with the same properties in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the process, method, system, product or equipment containing a series of units does not necessarily limit to those units, but can include other units not clearly listed or inherent to these processes, methods, products or equipment.

Claims

1. A network training method, characterized by, The method comprises: obtaining a first text and a first voice corresponding to the first text; based on the first text, obtaining a first phoneme sequence; obtaining the correspondence between the first voice and the first phoneme sequence based on the attention mechanism, wherein the correspondence is used to represent the duration of each phoneme in the first phoneme sequence in the first voice; based on the dynamic programming method, determining the unaligned phonemes in the correspondence, correcting the correspondence, and obtaining the first duration information of each phoneme in the first phoneme sequence; training the first prediction network based on the first phoneme sequence and the first duration information to obtain a trained first prediction network, wherein the trained first prediction network is used to predict the duration information of each phoneme in the to-be-processed text. The first voice includes at least two types of language / dialect voices, and before training the first prediction network based on the first phoneme sequence and the first duration information, the method further comprises: obtaining a second text and a second voice corresponding to the second text, wherein the second voice includes a voice of one type of language / dialect in the at least two types of language / dialect; obtaining a second phoneme sequence of the second text; obtaining second duration information of each phoneme in the second phoneme sequence; training a second prediction network based on the second phoneme sequence and the second duration information to obtain the first prediction network.

2. The method of claim 1, wherein, The training of the first prediction network based on the first phoneme sequence and the first duration information comprises: training the first prediction network by taking the first phoneme sequence as the input of the first prediction network and taking the value of the first loss function being less than the first threshold value as the target to obtain the trained first prediction network, wherein the first loss function is used to represent the difference between the duration information output by the first prediction network and the first duration information.

3. The method according to claim 1 or 2, characterized in that, The training of the second prediction network based on the second phoneme sequence and the second duration information to obtain the first prediction network comprises: training the second prediction network by taking the second phoneme sequence as the input of the second prediction network and taking the value of the second loss function being less than the second threshold value as the target to obtain the first prediction network, wherein the second loss function is used to represent the difference between the duration information output by the second prediction network and the second duration information.

4. The method of any one of claims 1 to 2, characterized in that, The method further comprises: obtaining a first mel-spectrum feature of the first voice; obtaining a first pronunciation feature, wherein the first pronunciation feature is used to describe the timbre feature of the first voice; training the first voice synthesis network by taking the first phoneme sequence, the first duration information, and the first pronunciation feature as the input of the first voice synthesis network and taking the value of the third loss function being less than the third threshold value as the target to obtain the trained first voice synthesis network and the trained first pronunciation feature, wherein the third loss function is used to represent the difference between the second mel-spectrum feature output by the first voice synthesis network and the first mel-spectrum feature, and the second mel-spectrum feature is obtained after the first duration information is expanded.

5. The method of claim 4, wherein, The first voice synthesis network comprises an encoder and an autoregressive decoder. The first phoneme sequence and the first duration information are taken as inputs of a first speech synthesis network, and the first speech synthesis network is trained to obtain a trained first speech synthesis network, with a value of a third loss function being less than a third threshold as a target. The first phoneme sequence is taken as an input of an encoder to obtain first features corresponding to the first phoneme sequence. The first features are extended based on the first duration information to obtain second features. The second features are taken as inputs of an autoregressive decoder to obtain the second mel-spectrogram features. The encoder and the autoregressive decoder are trained with the value of the third loss function being less than the third threshold as a target to obtain the trained first speech synthesis network.

6. The method of claim 5, wherein, The second features are taken as inputs of the autoregressive decoder to obtain the second mel-spectrogram features. The second features are taken as inputs of the autoregressive decoder to obtain the second mel-spectrogram features.

7. The method of claim 5, wherein, The second features are taken as inputs of the autoregressive decoder to obtain the second mel-spectrogram features. The first speech includes speech of at least two types of languages / dialects. The method further includes:

8. The method of claim 4, wherein, obtaining a third text and a third speech corresponding to the third text, the third speech including speech of one type of language / dialect in the at least two types of languages / dialects; obtaining a third phoneme sequence of the third text; obtaining third duration information of each phoneme in the third phoneme sequence; obtaining a second pronunciation feature, the second pronunciation feature being used to describe a timbre feature of the third speech; obtaining a third mel-spectrogram feature of the third speech; The third phoneme sequence, the third duration information, and the second pronunciation feature are taken as inputs of a second speech synthesis network, and the second speech synthesis network is trained to obtain the first speech synthesis network and a trained second pronunciation feature, with a value of a fourth loss function being less than a fourth threshold as a target, the fourth loss function being used to represent a difference between a fourth mel-spectrogram feature output by the second speech synthesis network and the third mel-spectrogram feature, the fourth mel-spectrogram feature being obtained after the third duration information is extended. The dynamic programming method includes a monotonic alignment search (MAS) method or a Viterbi algorithm. The method includes:

9. The method of any one of claims 1 to 2, wherein, obtaining a text to be processed; 10. A data processing method, characterized by, based on the text to be processed, obtaining a phoneme sequence of the text to be processed; based on a trained prediction network, predicting duration information of each phoneme in the phoneme sequence, the trained prediction network being trained based on a first text and first duration information, the first duration information being obtained by determining unaligned phonemes in a correspondence between the first text and a first speech through a dynamic programming method and correcting the correspondence, the first speech being a speech of the first text, the first speech including speech of at least two types of languages / dialects, the correspondence being used to represent durations of the phonemes in the phoneme sequence in the first speech; ​ ​ The prediction network is trained based on a second phoneme sequence of a second text and second duration information of each phoneme in the second phoneme sequence, the second text corresponding to a second speech, and the second speech including speech of one of the at least two types of languages / dialects.

11. The method of claim 10, wherein, The method further includes: inputting the phoneme sequence and the duration information into a trained speech synthesis network to obtain a mel-spectrogram feature, the trained speech synthesis network being configured to generate speech corresponding to the text; convert the mel-spectrogram feature into speech by a vocoder, the speech being speech of the text to be processed.

12. The method of claim 11, wherein, The method further includes: obtaining a pronunciation feature, the pronunciation feature being configured to describe a timbre feature of a pronouncer; The inputting the phoneme sequence and the duration information into a trained speech synthesis network to obtain a mel-spectrogram feature includes: inputting the phoneme sequence, the duration information, and the pronunciation feature into the trained speech synthesis network to obtain the mel-spectrogram feature.

13. The method of claim 12, wherein, The trained speech synthesis network includes an encoder and an autoregressive decoder; The inputting the phoneme sequence, the duration information, and the pronunciation feature into a trained speech synthesis network includes: obtaining a first feature corresponding to the phoneme sequence based on the encoder; extending the first feature based on the duration information to obtain a second feature; obtaining a third feature based on the second feature and the pronunciation feature; obtaining the mel-spectrogram feature based on the autoregressive decoder and the third feature.

14. The method of claim 13, wherein, The obtaining the mel-spectrogram feature based on the autoregressive decoder and the third feature includes: inputting the third feature into the autoregressive decoder to obtain the mel-spectrogram feature.

15. The method of claim 13, wherein, The obtaining the mel-spectrogram feature based on the autoregressive decoder and the third feature includes: performing convolution processing on the third feature to obtain a fourth feature; inputting the fourth feature into the autoregressive decoder to obtain the mel-spectrogram feature.

16. The method according to any one of claims 13 to 15, characterized in that, The text to be processed includes text of at least two types of languages / dialects, and the first feature is further configured to describe a language / dialect to which the phoneme belongs.

17. The method according to any one of claims 10 to 15, characterized in that, The dynamic programming method includes a monotonic alignment search (MAS) method or a Viterbi algorithm.

18. A data processing device, characterized by The data processing device includes: an obtaining unit configured to obtain a first text and a first speech corresponding to the first text, the first speech including speech of at least two types of languages / dialects; The obtaining unit is further configured to obtain a first phoneme sequence based on the first text. The obtaining unit is further configured to obtain a correspondence between the first speech and the first phoneme sequence based on an attention mechanism, the correspondence being configured to represent a duration of each phoneme in the first phoneme sequence in the first speech. A correction unit is configured to determine unaligned phonemes in the correspondence based on a dynamic programming method, correct the correspondence, and obtain first duration information of each phoneme in the first phoneme sequence. The obtaining unit is further configured to obtain a second text and a second speech corresponding to the second text, the second speech including speech of one of the at least two types of languages / dialects. The acquisition unit is further configured to acquire a second phoneme sequence of the second text. The acquisition unit is further configured to acquire second duration information of each phoneme in the second phoneme sequence. The training unit is configured to train a second prediction network based on the second phoneme sequence and the second duration information, to obtain the first prediction network. The training unit is further configured to train the first prediction network based on the first phoneme sequence and the first duration information, to obtain the trained first prediction network, which is used to predict duration information of each phoneme in a text to be processed.

19. The apparatus of claim 18, wherein, The correction unit is specifically configured to take the first phoneme sequence as an input of the first prediction network, and train the first prediction network to obtain the trained first prediction network, with a value of a first loss function being less than a first threshold, where the first loss function is used to represent a difference between duration information output by the first prediction network and the first duration information.

20. A data processing device, characterized by The data processing device comprises: An acquisition unit configured to acquire a text to be processed. The acquisition unit is further configured to obtain a phoneme sequence based on the text to be processed. A prediction unit configured to predict duration information of each phoneme in the phoneme sequence based on a trained prediction network, where the trained prediction network is trained based on a first text and first duration information, the first duration information is obtained by correcting a correspondence relationship based on a dynamic programming method, the first duration information is used to represent a duration of each phoneme in a first voice corresponding to the first text, the first voice comprises voice of at least two types of languages / dialects, and the correspondence relationship is used to represent the duration of each phoneme in the phoneme sequence in the first voice. The prediction network is trained based on a second phoneme sequence of a second text and second duration information of each phoneme in the second phoneme sequence, the second text corresponds to a second voice, and the second voice comprises voice of one type of language / dialect in the at least two types of languages / dialects.

21. The apparatus of claim 20, wherein, The data processing device further comprises: A processing unit configured to input the phoneme sequence and the duration information into a trained voice synthesis network to obtain a mel-spectrogram feature. A conversion unit configured to convert the mel-spectrogram feature into voice by a vocoder, where the voice is voice of the speaker for the text to be processed.

22. A data processing device, characterized by A processor coupled with a memory, the memory being configured to store programs or instructions, when the programs or instructions are executed by the processor, the data processing device is caused to perform the method in any one of claims 1 to 9, or the data processing device is caused to perform the method in any one of claims 10 to 17. The computer-readable storage medium stores instructions, when the instructions are executed on a computer, the computer is caused to perform the method in any one of claims 1 to 9, or the computer is caused to perform the method in any one of claims 10 to 17.

23. A computer-readable storage medium, characterized in that, ​ 24. A computer program product, characterised in that, The computer program product, when executed on a computer, causes the computer to perform the method of any one of claims 1 to 9, or causes the computer to perform the method of any one of claims 10 to 17.

Images