Prosody annotation model, prosody prediction model training method and related equipment

Abstract

The method for training a prosody labeling model and the related device provided in the embodiments of the present application input a text sequence into a text encoder to output a text encoding vector; input a phoneme sequence into the phoneme encoder to output a phoneme encoding vector; input a mel spectrum into a spectrum encoder to output a spectrum encoding vector; perform fusion processing on the text encoding vector, the phoneme encoding vector and the spectrum encoding vector; input the fused feature vector into a self-attention network to output a predicted prosody label sequence; in the foregoing manner, multi-modal data is extracted from audio data as input, richer prosody information is provided for a prosody labeling task, and the prosody labeling model obtained through training has high labeling accuracy; the method for training a prosody prediction model and the related device provided in the embodiments of the present application use the trained prosody labeling model to perform sample enhancement, which is conducive to simplifying the structure of the prosody prediction model and input data, and the prosody prediction model obtained through training has good prosody prediction effect.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, specifically to a prosody annotation model, a prosody prediction model training method, and related equipment. Background Technology

[0002] TTS (text-to-speech) is a text-to-speech synthesis algorithm. Currently, mainstream speech synthesis tasks are based on neural networks (Neural TTS). Compared to traditional concatenative and parametric speech synthesis models, Neural TTS produces smoother, more natural-sounding speech.

[0003] The prosody of language refers to the characteristics of "stress, intonation, and rhythm" in human speech, which manifests acoustically as pauses, pitch, duration, and intensity, and is also known as suprasegmental features. High-quality prosodic prediction is crucial for synthesizing highly natural-sounding speech. Improper prosodic processing in speech synthesis systems can result in monotonous and dull-sounding synthesized speech. Prosodic annotation is a critical step in current online TTS (Text-to-Speech) service models. However, obtaining high-quality prosodic annotation data in existing technologies requires a significant investment of manpower with relevant linguistic knowledge, resulting in high costs and often low-quality external prosodic annotation data used for prosodic prediction. This negatively impacts the performance of the prosodic prediction stage and subsequent speech synthesis. Summary of the Invention

[0004] In view of the above problems, this application provides a training method and related equipment for a prosody annotation model and a prosody prediction model to solve the above technical problems.

[0005] In a first aspect, embodiments of this application provide a training method for a prosodic annotation model, wherein the prosodic annotation model includes a text encoder, a phoneme encoder, a spectrum encoder, and a self-attention network, and the training method includes:

[0006] Obtain a first training sample, which includes a text sequence, a phoneme sequence, and a Mel spectrum obtained from speech data, as well as a real prosodic label sequence corresponding to the text sequence;

[0007] The text sequence is input into the text encoder, and the corresponding text encoding vector is output.

[0008] The phoneme sequence is input into the phoneme encoder, and the corresponding phoneme encoding vector is output.

[0009] The Mel spectrum is input to the spectrum encoder, and the corresponding spectrum encoding vector is output.

[0010] The text encoding vector, the phoneme encoding vector, and the spectrum encoding vector are fused together to obtain the corresponding fused feature vector.

[0011] The fused feature vector is input into the self-attention network, which outputs the predicted prosodic label sequence corresponding to the text sequence.

[0012] The loss value is calculated based on the predicted prosodic label sequence and the real prosodic label sequence. The parameters of the prosodic labeling model are adjusted based on the loss value until the prosodic labeling model reaches the training convergence condition.

[0013] As one implementation method, the step of fusing the text encoding vector, the phoneme encoding vector, and the spectrum encoding vector to obtain a corresponding fused feature vector includes:

[0014] The text encoding vector, the phoneme encoding vector, and the spectrum encoding vector are concatenated to obtain a concatenated vector.

[0015] The concatenated vector is input into a fully connected layer, and the fused feature vector corresponding to the text sequence is output.

[0016] In one implementation, the self-attention network includes a plurality of converter modules and an output layer connected in sequence, and each converter module includes a multi-head attention mechanism and a feedforward network;

[0017] The step of inputting the fused feature vector into the self-attention network and outputting the predicted prosodic label sequence corresponding to the text sequence includes:

[0018] The first input intermediate vector is obtained by using the multi-head self-attention mechanism and layer normalization to process the representation vector output by the previous layer converter module. The first input intermediate vector is then input into the feedforward network of the current converter module to obtain the output second input intermediate vector. The first input intermediate vector and the second input intermediate vector are then processed by layer normalization to obtain the current representation vector. The current representation vector is then output to the converter module of the next layer.

[0019] The output layer outputs a probability feature vector of the text sequence based on the representation vector output by the converter module of the last layer. The probability feature vector includes the probability that each word in the text sequence belongs to each preset prosodic category. The preset prosodic category with the highest probability is used as the predicted prosodic of the word to output the predicted prosodic label sequence.

[0020] Secondly, embodiments of this application provide a method for training a prosody prediction model, comprising:

[0021] The text sequence is fed into the prosodic annotation model, and the first predicted prosodic label sequence of the text sequence is output to obtain a second training sample based on the text sequence and the corresponding first predicted prosodic label sequence; wherein, the prosodic annotation model is obtained according to the training method of the prosodic annotation model described above;

[0022] The text sequence is input into the prosody prediction model, and the second predicted prosody label sequence corresponding to the text sequence is output.

[0023] The loss value is calculated based on the first predicted prosodic label sequence and the second prosodic label sequence. The parameters of the prosodic prediction model are adjusted based on the loss value until the prosodic prediction model reaches the training convergence condition.

[0024] In one implementation, the prosody prediction model includes an input layer, an LSTM network, and a CRF layer;

[0025] The step of inputting the text sequence into the prosodic prediction model and outputting the second predicted prosodic label sequence corresponding to the text sequence includes:

[0026] The text sequence is input into the input layer, and the first word-based representation vector corresponding to the text sequence is output.

[0027] The input representation vector is fed into the LSTM network for feature extraction, and the second representation vector corresponding to the text sequence is output.

[0028] The second representation vector is input into the CRF layer, and the second predicted prosodic label sequence corresponding to the text sequence is output.

[0029] Thirdly, embodiments of this application provide a speech synthesis method, including:

[0030] Obtain the phoneme sequence corresponding to the text sequence to be synthesized;

[0031] Obtain the corresponding text encoding vector based on the text sequence;

[0032] Obtain the corresponding phoneme encoding vector based on the phoneme sequence;

[0033] The text sequence is input into the trained prosodic prediction model, and the predicted prosodic label sequence corresponding to the text sequence is output. The prosodic prediction model is obtained according to the training method of the prosodic prediction model described above.

[0034] Speech synthesis data is obtained based on the text encoding vector, the phoneme encoding vector, and the predicted prosodic label sequence.

[0035] Fourthly, embodiments of this application provide a training device for a prosodic annotation model, the prosodic annotation model including a text encoder, a phoneme encoder, a spectrum encoder, and a self-attention network, the training device including:

[0036] The first training sample construction module is used to obtain the first training sample, which includes a text sequence, a phoneme sequence, and a Mel spectrum obtained from the speech data, as well as a real prosodic label sequence corresponding to the text sequence.

[0037] The text encoding module is used to input the text sequence into the text encoder and output the corresponding text encoding vector;

[0038] A phoneme encoding module is used to input the phoneme sequence into the phoneme encoder and output the corresponding phoneme encoding vector;

[0039] The spectrum encoding module is used to input the Mel spectrum into the spectrum encoder and output the corresponding spectrum encoding vector.

[0040] The fusion module is used to fuse the text encoding vector, the phoneme encoding vector, and the spectrum encoding vector to obtain a corresponding fused feature vector.

[0041] The first prediction module is used to input the fused feature vector into the self-attention network and output the predicted prosodic label sequence corresponding to the text sequence;

[0042] The first parameter adjustment module is used to calculate the loss value based on the predicted prosodic label sequence and the real prosodic label sequence, and adjust the parameters of the prosodic labeling model based on the loss value until the prosodic labeling model reaches the training convergence condition.

[0043] Fifthly, embodiments of this application provide a training apparatus for a prosody prediction model, comprising:

[0044] The second training sample construction module is used to feed the text sequence into the prosodic annotation model and output the first predicted prosodic label sequence of the text sequence, so as to obtain the second training sample based on the text sequence and the corresponding first predicted prosodic label sequence; wherein, the prosodic annotation model is obtained according to the training method of the prosodic annotation model;

[0045] The second prediction module is used to input the text sequence into the prosody prediction model and output the second predicted prosody label sequence corresponding to the text sequence;

[0046] The second parameter adjustment module is used to calculate a loss value based on the first predicted prosodic label sequence and the second prosodic label sequence, and adjust the parameters of the prosodic prediction model based on the loss value until the prosodic prediction model reaches the training convergence condition.

[0047] Sixthly, embodiments of this application provide an electronic device, including a processor and a memory coupled to the processor, the memory storing program instructions executable by the processor; when the processor executes the program instructions stored in the memory, it implements the above-described prosody annotation model training method or the above-described prosody prediction model training method.

[0048] In a seventh aspect, embodiments of this application provide a storage medium storing program instructions, which, when executed by a processor, implement the above-described prosody annotation model training method or the above-described prosody prediction model training method.

[0049] The prosodic annotation model training method and related equipment provided in this application embodiment input the text sequence into the text encoder and output the corresponding text encoding vector; input the phoneme sequence into the phoneme encoder and output the corresponding phoneme encoding vector; input the Mel spectrum into the spectrum encoder and output the corresponding spectrum encoding vector; fuse the text encoding vector, the phoneme encoding vector, and the spectrum encoding vector to obtain the corresponding fused feature vector; input the fused feature vector into the self-attention network and output the predicted prosodic label sequence corresponding to the text sequence; through the above method, multimodal data is extracted from the audio data as input, providing richer prosodic information for the prosodic annotation task, and the trained prosodic annotation model has high annotation accuracy; the prosodic prediction model training method and related equipment provided in this application embodiment use the trained prosodic annotation model for sample augmentation to obtain a large number of text sequence-prosodic label sequences as training samples, which helps to simplify the structure and input data of the prosodic prediction model, and the trained prosodic prediction model has good prosodic prediction effect. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart illustrating the training method of a prosody annotation model provided in an embodiment of this application is shown.

[0052] Figure 2 A flowchart illustrating the training method of a prosody prediction model provided in an embodiment of this application is shown.

[0053] Figure 3 A schematic flowchart of a speech synthesis method provided in an embodiment of this application is shown.

[0054] Figure 4 A schematic diagram of the structure of a training device for a prosody annotation model provided in an embodiment of this application is shown.

[0055] Figure 5 A schematic diagram of the structure of a training device for a prosody prediction model provided in an embodiment of this application is shown.

[0056] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown.

[0057] Figure 7 A schematic diagram of the structure of a storage medium provided in an embodiment of this application is shown. Detailed Implementation

[0058] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0059] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0060] In the embodiments of this application, "at least one" refers to one or more; "multiple" refers to two or more. In the description of this application, the terms "first," "second," "third," etc., are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying order.

[0061] References such as “one embodiment” or “some embodiments” as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the terms “comprising,” “including,” “having,” and variations thereof, as used in this specification, mean “including, but not limited to,” unless otherwise specifically emphasized.

[0062] It should be noted that in the embodiments of this application, "and / or" describes the relationship between associated objects, indicating that there can be three kinds of relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0063] It should be noted that in the embodiments of this application, "connection" can be understood as electrical connection. The connection between two electrical components can be a direct or indirect connection between the two electrical components. For example, the connection between A and B can be a direct connection between A and B, or an indirect connection between A and B through one or more other electrical components.

[0064] One embodiment of this application provides a training method for a prosodic annotation model and a training method for a prosodic prediction model. The execution entity of the prosodic annotation model training method or the prosodic prediction model training method includes, but is not limited to, at least one of the following electronic devices: a server, a terminal, or any other electronic device that can be configured to execute the prosodic annotation model training method or the prosodic prediction model training method provided in this application embodiment. In other words, the prosodic annotation model training method or the prosodic prediction model training method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0065] Please see Figure 1 The diagram shown is a flowchart illustrating a training method for a prosodic annotation model provided in one embodiment of this application. It should be noted that if substantially the same result is obtained, the method of this application does not necessarily require further elaboration. Figure 1 The illustrated process sequence is limited. The prosodic annotation model includes a text encoder, a phoneme encoder, a spectrum encoder, and a self-attention network. In this embodiment, the training method of the prosodic annotation model includes the following steps:

[0066] S11, Obtain a first training sample, the first training sample including a text sequence, phoneme sequence and melodic spectrum obtained from speech data, and a real prosodic label sequence corresponding to the text sequence;

[0067] Among them, speech recognition is performed on speech data to extract text content, and then a text sequence is generated according to the text content. As an implementation, the text sequence can be the Chinese word segmentation sequence of the text content. Word segmentation processing is performed on the text content. For example, deep learning word segmenters can be used for word segmentation processing, or word segmentation tools such as jieba segmentation, pkuseg segmentation, or thulac segmentation can be used to perform word-level encoding on the text content to obtain an initial word vector representation as the text sequence. For example, if the text content is "There are 112 211 universities in the country", the word segmentation result is [country / together / have / one hundred and twelve / schools / 211 / universities], and the text sequence is [C 全国 ,C 一共 ,C 有 ,C 一百一十二 ,C 所 ,C 二一一 ,C 高校 ,C x represents the word vector of x.

[0068] Among them, a phoneme is the smallest unit in speech and can be analyzed based on the pronunciation actions in the syllables of a word. One action constitutes one phoneme. For example, in Chinese, there are 32 phonemes, and phonemes can be divided into initials and finals. For example, for the text content recognized by speech, its corresponding phoneme sequence can be composed of the initials and finals of each character in turn. Taking the text content "Hello, Ping An of China" as an example, its corresponding phoneme sequence can be "n, i, h, ao, zh, ong, g, uo, p, ing, an". It should be understood that its corresponding phoneme sequence can also be in other forms, and the application does not specifically limit the form of the phoneme sequence. Another example, if the text content is "There are 112 211 universities in the country", its corresponding phoneme sequence can be "quan2 g uo2 y i2 g ong4 y ou3 y i4 b ai3 y i1 sh i2 er4 s uo3 er4 y ao1 y ao1g oa1 x iao4", or its corresponding phoneme sequence can be "[q uan g uo y i g ong y ou y i b aiy i sh i er s uo er y ao y ao g oa x iao], [0 2 0 2 0 2 0 4 0 3 0 4 0 3 0 10 24 0 3 4 0 1 0 1 0 1 0 4]".

[0069] The process involves performing a short-time Fourier transform (STFT) on the speech data to obtain a spectrogram. Then, a linear transformation is applied to the coordinates of the obtained spectrogram to obtain the linear transformation result. Finally, the linear transformation result is multiplied by a Mel filter to obtain the Mel spectrum. Alternatively, the librosa tool in Python can be used to perform the STFT on the speech data.

[0070] S12, input the text sequence into the text encoder and output the corresponding text encoding vector;

[0071] In this process, the text encoder maps each word segmentation vector in the text sequence to a corresponding feature vector, and these feature vectors form the text encoding vector. As one implementation method, BERT (Bidirectional Encoder Representation from Transformers) can be used to encode the text sequence, obtaining the text encoding vector output by BERT.

[0072] S13, input the phoneme sequence into the phoneme encoder and output the corresponding phoneme encoding vector;

[0073] The phoneme encoder maps each character in the phoneme sequence to a corresponding character feature vector, and the character feature vectors form the phoneme encoding vector.

[0074] S14, input the Mel spectrum to the spectrum encoder and output the corresponding spectrum encoding vector;

[0075] The spectrum encoder is used to extract the high-dimensional latent features of the Mel spectrum and output the corresponding high-dimensional latent feature vectors. The high-dimensional latent feature vectors form the spectrum encoding vector.

[0076] S15, the text encoding vector, the phoneme encoding vector and the spectrum encoding vector are fused to obtain the corresponding fused feature vector;

[0077] As one implementation method, step S15 specifically includes the following steps:

[0078] S151, the text encoding vector, the phoneme encoding vector and the spectrum encoding vector are concatenated to obtain a concatenated vector;

[0079] The text encoding vector, the phoneme encoding vector, and the spectrum encoding vector can be summed directly, and the three vectors are superimposed according to their order to obtain the concatenated vector.

[0080] S152, input the concatenation vector into the fully connected layer, and output the fused feature vector corresponding to the text sequence;

[0081] In one implementation, two fully connected layers can be set up, namely a first fully connected layer and a second fully connected layer. The number of nodes in the first fully connected layer is a first number, and the number of nodes in the second fully connected layer is a second number. The first number is greater than the second number. In step S152, the concatenation vector is first input to the first fully connected layer. At each node of the first fully connected layer, the feature vectors in the concatenation vector are fused to obtain a first number of first-level cross features. The first number of first-level cross features form a high-dimensional fused feature vector. Then, the high-dimensional fused feature vector is input to the second fully connected layer. At each node of the second fully connected layer, the first number of first-level cross features are fused to obtain a second number of different second-level cross features. The second number of second-level cross features form a low-dimensional fused feature vector. This low-dimensional fused feature vector is the fused feature vector corresponding to the text sequence.

[0082] S16, input the fused feature vector into the self-attention network, and output the predicted prosodic label sequence corresponding to the text sequence;

[0083] As one implementation method, please refer to Figure 3 As shown, the self-attention network includes multiple transformer blocks and an output layer connected in sequence. Each transformer block includes a multi-head attention mechanism and a feedforward network.

[0084] In step S16, the representation vector output by the previous layer's converter module is processed by a multi-head self-attention mechanism and layer normalization using the current converter module to obtain a first input intermediate vector. The first input intermediate vector is then input into the feedforward network of the current converter module to obtain a second input intermediate vector. Layer normalization is then performed on the first and second input intermediate vectors to obtain the current representation vector. The current representation vector is then output to the next layer's converter module. The output layer outputs a probability feature vector of the text sequence based on the representation vector output by the last layer's converter module. The probability feature vector includes the probability that each word in the text sequence belongs to each preset prosodic category. The preset prosodic category with the highest probability is used as the predicted prosodic of the word segment to output the predicted prosodic label sequence.

[0085] S17, calculate the loss value based on the predicted prosodic label sequence and the real prosodic label sequence, and adjust the parameters of the prosodic labeling model based on the loss value until the prosodic labeling model reaches the training convergence condition.

[0086] As one implementation method, the loss function of the output layer is the cross-entropy loss function:

[0087]

[0088] Where M is the number of preset prosody categories; y ic The sign function is 0 or 1; it takes the value 1 if the true class of sample i is equal to c, and 0 otherwise. ic Let be the predicted probability that observed sample i belongs to category c.

[0089] The prosodic annotation model that meets the training convergence condition is used as the trained prosodic annotation model.

[0090] In the training process of the prosodic annotation model (Teacher Model), this application introduces text encoding vector, phoneme encoding vector and spectrum encoding vector for fusion processing to obtain the corresponding fused feature vector; then the fused feature vector is input into the self-attention network for annotation to obtain the predicted prosodic label sequence. This application uses multimodal (text, phoneme and spectrum) to provide richer prosodic information for the prosodic prediction task and improves the accuracy of prosodic annotation.

[0091] Please see Figure 2 The diagram shown is a flowchart illustrating a training method for a prosody prediction model provided in one embodiment of this application. It should be noted that if substantially the same result is obtained, the method of this application does not necessarily require further elaboration. Figure 2 The illustrated process sequence is limited. In this embodiment, the training method for the prosody prediction model includes the following steps:

[0092] S21, feed the text sequence into the prosody labeling model and output the first predicted prosody label sequence of the text sequence, so as to obtain the second training sample based on the text sequence and the corresponding first predicted prosody label sequence;

[0093] The prosodic annotation model is obtained according to the training method of the prosodic annotation model described above. In this embodiment, the trained prosodic annotation model (Teacher Model) is used to generate training samples for the prosodic prediction model (Student Model), and the first predicted prosodic label sequence is directly used as the real prosodic label sequence.

[0094] S22, input the text sequence into the prosody prediction model, and output the second predicted prosody label sequence corresponding to the text sequence;

[0095] As one implementation method, the LSTM-CRF model, CRF model, or MEMM model can be used. The following description uses the LSTM+CRF model as an example. The prosody prediction model (Student Model) includes an input layer, an LSTM network, and a CRF layer. In step S22, firstly, the text sequence is input into the input layer, and the first word-based representation vector corresponding to the text sequence is output. Then, the input representation vector is input into the LSTM network for feature extraction, and the second representation vector corresponding to the text sequence is output. Finally, the second representation vector is input into the CRF layer, and the second predicted prosodic label sequence corresponding to the text sequence is output.

[0096] S23, calculate the loss value based on the first predicted prosodic label sequence and the second prosodic label sequence, and adjust the parameters of the prosodic prediction model based on the loss value until the prosodic prediction model reaches the training convergence condition.

[0097] In this embodiment, the prosody prediction model (Student Model) takes only a text sequence as input. Furthermore, because the prosody annotation model (Teacher Model) provides a large number of training samples, the prediction performance of the trained prosody prediction model (Student Model) is comparable to that of the prosody annotation model (Teacher Model). In actual speech synthesis, this embodiment utilizes the trained prosody prediction model (Student Model) to generate prosodic label sequences.

[0098] Please see Figure 3 The diagram shown is a flowchart illustrating a speech synthesis method according to an embodiment of this application. It should be noted that if substantially the same result is achieved, the method of this application does not necessarily require further elaboration. Figure 3 The illustrated process sequence is limited. In this embodiment, the speech synthesis method includes the following steps:

[0099] S31, Obtain the phoneme sequence corresponding to the text sequence to be synthesized;

[0100] The specific method for obtaining the corresponding phoneme sequence based on the text sequence is described in step S11, and will not be elaborated here.

[0101] S32, Obtain the corresponding text encoding vector based on the text sequence;

[0102] The text sequence can be input into the text encoder, which outputs a corresponding text encoding vector. The text encoder maps each word segmentation vector in the text sequence to a corresponding feature vector, and the feature vectors form the text encoding vector. As one implementation, BERT (Bidirectional Encoder Representation from Transformers) can be used to encode the text sequence to obtain the text encoding vector output by BERT.

[0103] S33, Obtain the corresponding phoneme encoding vector according to the phoneme sequence;

[0104] The phoneme sequence can be input into the phoneme encoder, which outputs the corresponding phoneme encoding vector. The phoneme encoder maps each character in the phoneme sequence to a corresponding character feature vector, and the character feature vectors form the phoneme encoding vector.

[0105] S34, Input the text sequence into the trained prosody prediction model and output the predicted prosody label sequence corresponding to the text sequence;

[0106] The prosody prediction model is obtained according to the training method of the prosody prediction model described above.

[0107] S35, obtain speech synthesis data based on the text encoding vector, the phoneme encoding vector, and the predicted prosodic label sequence.

[0108] The speech synthesis method in this application uses a trained prosodic prediction model to obtain a predicted prosodic label sequence. Since the prosodic prediction model has a simple model structure and fast prediction speed, it is beneficial to improve the speech synthesis speed. Furthermore, since the prosodic prediction model has high prediction accuracy, it is beneficial to improve the speech synthesis accuracy.

[0109] like Figure 4As shown, one embodiment of this application provides a training device for a prosodic annotation model. The prosodic annotation model includes a text encoder, a phoneme encoder, a spectrum encoder, and a self-attention network. The training device 40 for the prosodic annotation model includes: a first training sample construction module 41, a text encoding module 42, a phoneme encoding module 43, a spectrum encoding module 44, a fusion module 45, a first prediction module 46, and a first parameter adjustment module 47. The first training sample construction module 41 is used to acquire first training samples, which include a text sequence, a phoneme sequence, and a Mel spectrum obtained from speech data, as well as a real prosodic label sequence corresponding to the text sequence. The text encoding module 42 is used to input the text sequence into the text encoder and output the corresponding text encoding vector. The phoneme encoding module 42 is used to input the text sequence into the text encoder and output the corresponding text encoding vector. The encoding module 43 is used to input the phoneme sequence into the phoneme encoder and output the corresponding phoneme encoding vector; the spectrum encoding module 44 is used to input the Mel spectrum into the spectrum encoder and output the corresponding spectrum encoding vector; the fusion module 45 is used to fuse the text encoding vector, the phoneme encoding vector, and the spectrum encoding vector to obtain the corresponding fused feature vector; the first prediction module 46 is used to input the fused feature vector into the self-attention network and output the predicted prosodic label sequence corresponding to the text sequence; the first parameter adjustment module 47 is used to calculate the loss value based on the predicted prosodic label sequence and the real prosodic label sequence, and adjust the parameters of the prosodic labeling model based on the loss value until the prosodic labeling model reaches the training convergence condition.

[0110] In one implementation, the fusion module 45 is further configured to: concatenate the text encoding vector, the phoneme encoding vector, and the spectrum encoding vector to obtain a concatenated vector; input the concatenated vector into a fully connected layer and output the fusion feature vector corresponding to the text sequence.

[0111] In one implementation, the self-attention network includes multiple converter modules and an output layer connected in sequence. Each converter module includes a multi-head attention mechanism and a feedforward network. The first prediction module 46 is further configured to: use the current converter module to process the representation vector output by the converter module of the previous layer through the multi-head self-attention mechanism and layer normalization to obtain a first input intermediate vector; input the first input intermediate vector into the feedforward network of the current converter module to obtain a second input intermediate vector; perform layer normalization on the first input intermediate vector and the second input intermediate vector to obtain a current representation vector; and output the current representation vector to the converter module of the next layer. The output layer outputs a probability feature vector of the text sequence based on the representation vector output by the converter module of the last layer. The probability feature vector includes the probability that each word in the text sequence belongs to each preset prosodic category. The preset prosodic category with the highest probability is used as the predicted prosodic of the word segment to output the predicted prosodic label sequence.

[0112] like Figure 5 As shown, one embodiment of this application provides a training device for a prosodic prediction model. The prosodic prediction model training device 50 includes: a second training sample construction module 51, a second prediction module 52, and a second parameter adjustment module 53. The second training sample construction module 51 is used to input a text sequence into the prosodic annotation model and output a first predicted prosodic label sequence of the text sequence to obtain second training samples based on the text sequence and the corresponding first predicted prosodic label sequence. The prosodic annotation model is obtained according to the above-described training method for the prosodic annotation model. The second prediction module 52 is used to input the text sequence into the prosodic prediction model and output a second predicted prosodic label sequence corresponding to the text sequence. The second parameter adjustment module 53 is used to calculate a loss value based on the first predicted prosodic label sequence and the second prosodic label sequence, and adjust the parameters of the prosodic prediction model based on the loss value until the prosodic prediction model reaches the training convergence condition.

[0113] In one implementation, the prosody prediction model includes an input layer, an LSTM network, and a CRF layer; the second prediction module 52 is further configured to: input the text sequence into the input layer and output a word-based first representation vector corresponding to the text sequence; input the input representation vector into the LSTM network for feature extraction and output a second representation vector corresponding to the text sequence; input the second representation vector into the CRF layer and output the second predicted prosody label sequence corresponding to the text sequence.

[0114] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Figure 6 As shown, the electronic device 60 includes a processor 61 and a memory 62 coupled to the processor 61.

[0115] The memory 62 stores program instructions for implementing the training method of the prosody annotation model or the training method of the prosody prediction model of any of the above embodiments.

[0116] The processor 61 is used to execute program instructions stored in the memory 62 to train a prosody annotation model or a prosody prediction model.

[0117] The processor 61 can also be referred to as a CPU (Central Processing Unit). The processor 61 may be an integrated circuit chip with signal processing capabilities. The processor 61 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0118] See Figure 7 , Figure 7 This is a schematic diagram of the structure of a storage medium according to an embodiment of this application. The storage medium 70 of this embodiment stores program instructions 71 capable of implementing all the above methods. These program instructions 71 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0120] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0121] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although this application has disclosed preferred embodiments as above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A method for training a prosody annotation model, characterized in that, The prosody labeling model comprises a text encoder, a phoneme encoder, a spectrum encoder and a self-attention network, and the training method comprises: obtaining a first training sample, the first training sample comprising a text sequence, a phoneme sequence and a mel spectrum obtained according to voice data, and a real prosody label sequence corresponding to the text sequence; inputting the text sequence into the text encoder to output a corresponding text encoding vector; inputting the phoneme sequence into the phoneme encoder to output a corresponding phoneme encoding vector; inputting the mel spectrum into the spectrum encoder to output a corresponding spectrum encoding vector; fusing the text encoding vector, the phoneme encoding vector and the spectrum encoding vector to obtain a corresponding fusion feature vector; inputting the fusion feature vector into the self-attention network to output a predicted prosody label sequence corresponding to the text sequence; calculating a loss value according to the predicted prosody label sequence and the real prosody label sequence, adjusting parameters of the prosody labeling model according to the loss value until the prosody labeling model reaches a training convergence condition; the fusion processing of the text encoding vector, the phoneme encoding vector and the spectrum encoding vector to obtain a corresponding fusion feature vector comprises: splicing the text encoding vector, the phoneme encoding vector and the spectrum encoding vector to obtain a splicing vector; summing the text encoding vector, the phoneme encoding vector and the spectrum encoding vector, and vector superimposing the three vectors in the arrangement order to obtain the formed splicing vector; inputting the splicing vector into a full connection layer to output the fusion feature vector corresponding to the text sequence; the full connection layer comprises a first full connection layer and a second full connection layer, the number of nodes of the first full connection layer is a first number, the number of nodes of the second full connection layer is a second number, and the first number is greater than the second number; the self-attention network comprises a plurality of transformer modules connected in sequence and an output layer, each transformer module comprises a multi-head attention mechanism and a feedforward network; the inputting of the fusion feature vector into the self-attention network to output the predicted prosody label sequence corresponding to the text sequence comprises: performing multi-head self-attention mechanism processing and layer normalization processing on the representation vector output by the transformer module in the previous layer by using the current transformer module to obtain a first input intermediate vector, inputting the first input intermediate vector into the feedforward network of the current transformer module to obtain a second input intermediate vector, performing layer normalization processing on the first input intermediate vector and the second input intermediate vector to obtain a current representation vector, and outputting the current representation vector to the transformer module in the next layer; outputting a probability feature vector of the text sequence according to the representation vector output by the transformer module in the last layer by using the output layer, wherein the probability feature vector comprises a probability of each word in the text sequence belonging to each preset prosody category; and taking the preset prosody category with the maximum probability as the predicted prosody of the word to output the predicted prosody label sequence. 2.A method for training a prosody prediction model, characterized in that, ​ inputting the text sequence into the prosody labeling model, and outputting a first predicted prosody label sequence of the text sequence, so as to obtain a second training sample according to the text sequence and the corresponding first predicted prosody label sequence; wherein the prosody labeling model is obtained according to the training method of the prosody labeling model in claim 1; inputting the text sequence into the prosody prediction model, and outputting a second predicted prosody label sequence corresponding to the text sequence; calculating a loss value according to the first predicted prosody label sequence and the second predicted prosody label sequence, and adjusting parameters of the prosody prediction model according to the loss value until the prosody prediction model reaches a training convergence condition. 3.The method of Claim 2, wherein, The prosody prediction model comprises an input layer, an LSTM network and a CRF layer. The inputting the text sequence into the prosody prediction model and outputting the second predicted prosody label sequence corresponding to the text sequence comprises: inputting the text sequence into the input layer, and outputting a first word-based representation vector corresponding to the text sequence; inputting the input representation vector into the LSTM network for feature extraction, and outputting a second representation vector corresponding to the text sequence; inputting the second representation vector into the CRF layer, and outputting the second predicted prosody label sequence corresponding to the text sequence.

4. A speech synthesis method characterized by, comprises: obtaining a phoneme sequence corresponding to a text sequence to be synthesized; obtaining a text encoding vector corresponding to the text sequence; obtaining a phoneme encoding vector corresponding to the phoneme sequence; inputting the text sequence into the trained prosody prediction model, and outputting a predicted prosody label sequence corresponding to the text sequence, wherein the prosody prediction model is obtained according to the training method of the prosody prediction model in claim 2 or 3; obtaining speech synthesis data according to the text encoding vector, the phoneme encoding vector and the predicted prosody label sequence.

5. An apparatus for training a prosody labeling model, the apparatus configured to implement the method for training a prosody labeling model according to claim 1, wherein The prosody labeling model comprises a text encoder, a phoneme encoder, a spectrum encoder and a self-attention network, and the training device comprises: a first training sample construction module configured to obtain a first training sample, the first training sample comprising a text sequence, a phoneme sequence and a mel spectrum obtained according to speech data, and a real prosody label sequence corresponding to the text sequence; a text encoding module configured to input the text sequence into the text encoder, and output a corresponding text encoding vector; a phoneme encoding module configured to input the phoneme sequence into the phoneme encoder, and output a corresponding phoneme encoding vector; a spectrum encoding module configured to input the mel spectrum into the spectrum encoder, and output a corresponding spectrum encoding vector; a fusion module configured to fuse the text encoding vector, the phoneme encoding vector and the spectrum encoding vector to obtain a corresponding fusion feature vector; a first prediction module configured to input the fusion feature vector into the self-attention network, and output a predicted prosody label sequence corresponding to the text sequence; The first parameter adjustment module is configured to calculate a loss value according to the predicted prosody label sequence and the real prosody label sequence, and adjust parameters of the prosody labeling model according to the loss value until the prosody labeling model reaches a training convergence condition. 6.A device for training a prosody prediction model, characterized by, The method comprises: The second training sample construction module is configured to input a text sequence into the prosody labeling model to output a first predicted prosody label sequence of the text sequence, and obtain a second training sample according to the text sequence and the corresponding first predicted prosody label sequence; wherein the prosody labeling model is obtained according to the training method of the prosody labeling model in claim 1; The second prediction module is configured to input the text sequence into the prosody prediction model to output a second predicted prosody label sequence corresponding to the text sequence; The second parameter adjustment module is configured to calculate a loss value according to the first predicted prosody label sequence and the second predicted prosody label sequence, and adjust parameters of the prosody prediction model according to the loss value until the prosody prediction model reaches a training convergence condition.

7. An electronic device, comprising: The processor executes the program instructions stored in the memory to implement the training method of the prosody labeling model in claim 1 or the training method of the prosody prediction model in claim 2 or 3.

8. A storage medium, characterized by The storage medium stores program instructions, and the program instructions are executed by the processor to implement the training method of the prosody labeling model in claim 1 or the training method of the prosody prediction model in claim 2 or 3.

Images